Training a driving policy

Abstract

A computer-implemented method for training a driving policy, the method comprising: first training 401, using a first machine learning technique, a first driving policy on a first dataset of driving s

Description

FIELD AND BACKGROUND

[0001] The present techniques relate to the field of driving policy training. More particularly, but not exclusively, the present techniques relate to techniques for training a driving policy, and its subsequent deployment.

[0002] In recent times, the use of autonomous vehicles (also known as self-driving vehicles, SDVs) or robots has become more widespread. Autonomous vehicles or robots are vehicles or robots capable of sensing their own environment and operating without human involvement. The degree of autonomy may vary from fully autonomous requiring no human involvement (sometimes known as Level 5 - Full Driving Automation), to lower levels of autonomy requiring some degree of human involvement (Levels 1 to 4). As used herein, an autonomous vehicle or robot refers to a vehicle or robot having at least some capability of operating without human involvement.

[0003] A significant consideration associated with the use of autonomous vehicles or robots is safety. In the example of an SDV such as a self-driving car, the self-driving car must operate safely in the real-world when driving, and must be able to respond to interactions with other road users and pedestrians, for example, in a safe and predictable manner. Given the enormous variation in possible road scenarios and interactions (which themselves depend on the behaviour of other road users), ensuring that a self-driving car is behaving and reacting to changes in its environment safely is a significant challenge.

[0004] In order to operate, some autonomous vehicles or robots utilise a learned driving policy (also known as a motion planner) in order to make decisions, such as how to control movement, behaviour, and / or howto react to changes in its environment. In this case, a driving policy may be devised that mimics expert or ideal driving behaviour, such that when used to control the autonomous vehicle or robot, the autonomous vehicle or robot is able to behave in a certain manner without human involvement.

[0005] Driving policies or motion planners may be trained using machine learning techniques. However, the present inventors have identified that in order to achieve a desired level of performance of the driving policy, for example with respect to safety, training of the driving policy may take an excessively long time and require excessively large amounts of computing power with associated energy costs.

[0006] At least certain embodiments of the present disclosure address one or more of these problems as set out above. More particularly, the present inventors have devised an improved technique for training a driving policy that is more efficient than prior approaches. Further, the performance of the training technique may be improved without manual human intervention. SUMMARY

[0007] Particular aspects and embodiments are set out in the appended claims.

[0008] Viewed from a first aspect, there is provided a computer-implemented method for training a driving policy, the method comprising: first training, using a first machine learning technique, a first driving policy on a first dataset of driving scenes; simulating, for each driving scene in a second dataset of driving scenes, driving behaviour in the driving scene based on the trained driving policy; determining, for each driving scene in the second dataset of driving scenes, one or more performance metrics of the trained first driving policy associated with the simulated driving behaviour; selecting, based on the one or more performance metrics, a subset of the driving scenes in the second dataset; assembling a third dataset of driving scenes based on a similarity between a plurality of driving scenes and the subset of the driving scenes in the second dataset; second training, using a second machine learning technique, a second driving policy on a fourth dataset, wherein the fourth dataset comprises the third dataset.

[0009] Thus, the present approach can be considered as, for example, a method for training a driving policy that selectively augments the training dataset of driving scenes with additional driving scenes which have been selected based on the performance of the trained driving policy during closed-loop simulation of driving behaviour using the trained driving policy. As a result, particularly useful or significant driving scenes may be identified and added to the training dataset so as to create a dataset of driving scenes that are particularly well-suited to a desired performance. Accordingly, training efficiency of the driving policy is increased.

[0010] Indeed, by augmenting the training dataset with driving scenes similar to driving scenes where driving behaviour has been simulated and the performance of the driving policy dictates (e.g. driving scenes with poor measured performance in the simulation), the training dataset can be used to improve performance of the driving policy on scenarios similar to these driving scenes. As a result, the time required to train the driving policy to a pre-determined level of performance can be reduced. This also reduces the overall amount of computing power required to train the driving policy.

[0011] Put another way, the training dataset may be augmented during the overall training process to include driving scenes that are similar but different to particularly significant driving scenes (where performance of the trained driving policy was particularly good, or bad, for example).

[0012] As discussed herein, the present training method advantageously uses closed-loop simulation. In comparative examples that utilise an open-loop training paradigm to train a driving policy, prediction and planning performance may diverge when the driving policy is evaluated using a closed-loop simulation or in the real-world, due to errors that may compound. In these comparative examples, where driving scenes are up-sampled after an initial training, data logs can increase in size rapidly resulting in prohibitively large amounts of data. Thus, in these comparative examples, it may be unfeasible and ineffective to use the available data logs for training the driving policy. Hence, in these comparative examples, the closed-loop performance of a trained driving policy is not considered and instead only perception level inputs and outputs are considered. This can result in reduced training efficiency due to the requirement for perception data rather than being able to evaluate the driving policy using a software-based closed loop simulation as per the present approach.

[0013] In some examples, the second training uses the first driving policy as an input. Thus, the first driving policy may be trained further on the fourth dataset, which includes the third dataset. In other words, the first driving policy, after it has been trained by the first training on the first dataset, is used as an input for the second training step. In examples, the first driving policy is thus then trained on the fourth dataset. As a result, the first driving policy may be refined or fine-tuned using the augmented fourth dataset, which includes the specifically selected scenes.

[0014] Accordingly, because the training is performed over the fourth dataset which includes the specifically selected scenes, the training can be better controlled and optimised for desired performance. Further, the training time can be reduced because the driving policy is trained on specifically selected driving scenes based on its performance, and so an end condition such as a pre-determined minimum level of performance, can be reached earlier than in examples where the driving policy is not trained on the fourth dataset. Hence, training efficiency is increased.

[0015] In some examples, the second training does not use the first driving policy as an input. In other words, the fourth dataset is used to train a new driving policy (the second driving policy), and the first driving policy is used only to find the fourth dataset. Thus, in this embodiment, different driving polices may be used depending on performance requirements. Flexibility in the training approach is also increased.

[0016] In some examples, the first machine learning technique and the second machine learning technique are the same or different. Thus, in some examples, the machine learning technique used to train the first driving policy and the machine learning technique used to train the second driving policy may be the same. Advantageously, this may reduce complexity in the training process. Alternatively, in some examples, the first and second machine learning techniques may be different, thereby resulting in a second driving policy trained in a different manner from the first driving policy. This may be advantageous in scenarios where flexibility in the training process is desirable depending on performance constraints.

[0017] In some examples, the first dataset, second dataset, third dataset, and fourth dataset comprise real-world driving scenes. Thus, the driving policies are advantageously trained on real-world driving scenes so as to more accurately mimic the behaviour of real-world driving. The real-world driving scenes may be selected as satisfying one or more predetermined criteria, for example criteria defined as representing expert, ideal, or safe driving behaviour. In some examples, there may be provided a real-world driving scene corpus, from which the driving scenes of each of the first, second, third, and fourth dataset are assembled. It will be appreciated that the second dataset may overlap with the first dataset, or alternatively the second dataset may have no driving scenes in common with the first dataset.

[0018] Further, the real-world driving scenes may be driving scenes captured by vehicles, for example SDVs, using on-board sensing apparatus. In some examples, the driving scenes may be video data depicting a driving scenario. In some examples, the real-world driving scenes comprise a range of different driving scenarios such as roads, highways, motorways, traffic lights, intersections, cross-roads, junctions, stop signs, multi-lane roads, parking lots, etc. As such, the first and second dataset may comprise a random selection of driving scenes depicting various driving scenarios. This advantageously provides a varied training corpus and simulation corpus for the first driving policy, to ensure that training is performed across a range of scenarios that an autonomous vehicle may encounter.

[0019] In other examples, the real-world driving scenes may comprise scenes of robot movement, such as picking items, placing items, gripping items, etc. This advantageously provides a varied training corpus and simulation corpus for the first driving policy, to ensure that training is performed across a range of scenarios that an autonomous robot may encounter.

[0020] In some examples, the fourth dataset comprises the first dataset and the third dataset. Thus, the first dataset is augmented with the driving scenes of the third dataset. This augmentation results in a dataset with specifically selected driving scenes, based on the simulated driving performance of the driving policy. As a result, training of the second driving policy can be specifically tuned in dependence on the selection and the performance metrics.

[0021] In some examples, the third dataset comprises one or more driving scenes which are not present in either the first dataset or the second dataset. That is to say, scenes similar to the subset of scenes in the second dataset are selected to form the third dataset, but these selected scenes are different from scenes present in the first or second dataset. In other words, rather than modify the training dataset by up-sampling the selected scenes to increase their relative weight in the overall dataset, the training dataset is augmented with similar but different scenes. This increase in information and different scenes in the dataset results in more efficient training of the driving policies, and a driving policy that performs better across a range of driving scenarios. In other words, the training dataset may be augmented during the overall training process to include driving scenes that are similar but different to particularly significant driving scenes (where performance of the driving policy was particularly good, or bad, for example).

[0022] In some examples, the method further comprises iterating the simulating, determining, selecting, assembling and second training steps one or more times until a pre-determined condition is satisfied. Thus, the training process may be repeated until a condition, for example an end condition has been satisfied. In some examples, the pre-determined condition may be a total number of iterations. In other examples, the training process may be repeated until the performance of the second driving policy satisfies one or more criteria when the second driving policy is used to simulate driving behaviour in a driving scene.

[0023] In some examples, the first and / or second machine learning technique is an imitation learning technique or a reinforcement learning technique. The present inventors have identified that imitation or reinforcement learning are particularly well-suited to training a driving policy and thus increase efficiency of the training.

[0024] In some examples, selecting the subset of the simulated driving scenes is based on the one or more performance metrics satisfying pre-determined criteria. In some examples, the criteria may relate to so-classified good or bad driving (or safe and unsafe driving, judged in comparison to pre-determined driving rules). Hence, the pre-determined criteria may be criteria related to safe driving. Thus, the predetermined criteria may be used to select driving scenes of the second dataset where the trained driving policy performs particularly well or particularly badly (for example in terms of a safety metric).

[0025] For example, where the criteria relate to good driving (such as safe driving), driving scenes of the second dataset may be selected where the driving policy performed particularly well. As a result, driving scenes similar to those where the driving policy performed well are selected and added to the third dataset, for training of the second driving policy. This can advantageously result in the driving policy becoming specialised or performing particularly well in certain driving scenarios. This can be beneficial in certain deployments, for example if the driving policy is to be used most often in certain driving scenarios, having a driving policy particularly performant in those certain driving scenarios can be advantageous.

[0026] Alternatively, in examples where the criteria relate to bad driving (such as unsafe driving), driving scenes of the second dataset may be selected where the driving policy performed badly (for example in an unsafe manner). As a result, driving scenes similar to those where the driving policy performed badly are selected and added to the third dataset, for training of the second driving policy. This increases efficiency of the training process, as the training dataset is augmented to include similar but different scenes where the driving policy is known to perform badly. The driving policy is then trained on these datasets and the time taken to satisfy the training end condition (e.g. a minimum safety performance 5 level) can be reduced, as the training time is better focussed on scenarios the driving policy does not perform well on. Thus, a desired level of performance of the driving policy may be achieved and in a more efficient manner than would otherwise have been possible if the driving policy had been trained on only the first dataset.

[0027] In some examples, the one or more performance metrics comprise metrics associated with one or more of acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and a difference between the simulated driving behaviour and stored driving data. Thus, the performance of the driving policy during the simulation of the driving scenes of the second dataset can be evaluated against metrics associated with real-world driving safety and compliance with ideal driving behaviour. This results in a selection of driving scenes of the second dataset based on real-world driving safety metrics of the driving policy, resulting in a trained driving policy that performs more safely in real-world driving scenarios.

[0028] In some examples, simulating driving behaviour, for each driving scene in the second dataset of driving scenes, comprises: controlling driving behaviour of a simulated agent in each driving scene based on the trained driving policy. In other words, for each driving scene in the second dataset, a simulated agent (for example a simulated vehicle), is included in the real-world driving scene and its driving behaviour in the real-world driving scene is controlled based on the trained driving policy. This results in a simulated agent that behaves according to the trained driving policy, and performance metrics associated with its simulated driving behaviour may then be determined. For example, simulation software may be used to simulate the driving behaviour and selected metrics may be determined based on collecting information associated with the behaviour and movement of the simulated agent, such as position, speed, acceleration, deceleration, jerk, curvature, etc.

[0029] In an example where the simulated agent is a simulated vehicle and the driving scene is a scene of a road recorded by a vehicle, the vehicle is replaced with the simulated vehicle and its behaviour on the road is controlled by the trained driving policy. This allows an assessment of the driving behaviour of the trained driving policy to be performed.

[0030] In some examples, simulating driving behaviour, for each driving scene in the second dataset of driving scenes, further comprises altering behaviour of actors in the driving scenes based on the simulated driving behaviour of the simulated agent. Thus, the effect of the simulated driving behaviour of the simulated agent on other actors in the scene is used, increasing the amount of information taken into account when assessing the performance of the trained driving policy.

[0031] In the example where the simulated agent is a simulated vehicle and the driving scene is a scene of a road recorded by a vehicle, the simulated actors may be other vehicles and / or road users 6 (pedestrians, cyclists, etc.) on the road whose behaviour is dependent on the behaviour of the simulated vehicle. Information associated with the behaviour of the other simulated vehicles may thus be determined to aid with the assessment of the performance of the trained driving policy.

[0032] In some examples, a hybrid approach of simulating the agent and altering the behaviour of the simulated actors based on the simulated driving behaviour of the simulated agent. In some examples, a subset of other vehicles present in the driving scene may be simulated, for example the selection of the subset may be based on a position in the driving scene of the other vehicles. This approach may represent a suitable balance between gathering useful information for use in the performance assessment while not overly increasing the complexity of the simulation.

[0033] In some examples, simulating driving behaviour comprises simulating driving behaviour in the driving scene for a plurality of successive time intervals, and wherein determining the one or more performance metrics comprises determining one or more performance metrics associated with simulated driving behaviour during each of the plurality of successive time intervals.

[0034] In some examples, determining the plurality of driving scenes of the third dataset of driving scenes is based on using a similarity model. The present inventors have identified that a similarity model, for example a trained neural network similarity model (such as an artificial neural network implemented as hardware or software, or indeed emulated in software and thus considered hardware) may advantageously be used to automatically assemble driving scenes similar to those of the subset of driving scenes of the second dataset.

[0035] The similarity model may be a neural network configured to transform driving scenes into multidimensional feature vectors. For example, the similarity model may be a trained artificial neural network, trained to transform driving scenes into multi-dimensional feature vectors. The assembling may then be based on the proximity of the features vectors in a feature vector space. For example, the n closest driving scenes in the feature vector space to the feature vector of each driving scene in the subset of driving scenes in the second dataset may be considered similar driving scenes and assembled into the third dataset.

[0036] Assembling the third dataset of driving scenes may be based on transforming the plurality of driving scenes and the subset of simulated driving scenes into multi-dimensional feature vectors and determining a similarity between the transformed multi-dimensional feature vectors. In some examples, the plurality of driving scenes are the driving scenes forming the third dataset.

[0037] The driving scenes assembled to form the third dataset may be different from the driving scenes forming the first and second datasets. In examples, the first dataset and second dataset and the driving scenes selected to form the third dataset may each be selected from a broader training corpus of driving scenes.

[0038] In some examples, the first and / or second driving policy is a neural network, for example a trained artificial neural network (implemented as hardware or software, or indeed emulated in software and thus considered hardware), such as an artificial neural network configured to take as input information relating to a driving environment and output information relating to control of driving behaviour in the environment.

[0039] In some examples, the first dataset of driving scenes is created based on random sampling of driving scenes. As a result, the first driving policy is trained on a random range of driving scenarios without over-sampling certain driving scenarios. This results in more uniform initial performance (before a second training step takes place) compared to a training dataset with driving scene bias.

[0040] In some examples, the method further comprises using the first and / or second trained driving policy in a motion planner of an autonomous vehicle. In other words, trained driving policy / policies are deployed in a motion planner of an autonomous vehicle to control motion of the autonomous vehicle. The motion planner may be neural motion planner.

[0041] In some examples, the method comprises controlling an autonomous vehicle or robot based on the first and / or second trained driving policy. Hence, the driving policy (the first or second) may then be deployed so as to actually control an autonomous vehicle or robot.

[0042] In some examples, the method further comprises collecting real-world driving data based on controlling the autonomous vehicle or robot. Thus, the autonomous vehicle or robot controlled using the trained driving policy is itself used to collect real-would driving data that may be used for training of driving policies.

[0043] In a second aspect, there is provided a computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method described herein.

[0044] In a third aspect, there is provided an apparatus comprising a processor configured to carry out the method described herein.

[0045] In a fourth aspect, there is provided a computer-readable medium comprising a driving policy trained according to the method described herein.

[0046] In a fifth aspect, there is provided an apparatus comprising memory comprising a driving policy trained according to the method described herein.

[0047] In some examples, the apparatus is a robot or an autonomous vehicle, for example an autonomous car or autonomous airborne vehicle

[0048] In a sixth aspect, there is provided a method for controlling an autonomous vehicle, comprising controlling the autonomous vehicle based on a driving policy trained according to the method described herein.

[0049] Other aspects will also become apparent upon review of the present disclosure, in particular upon review of the Brief Description of the Drawings, Detailed Description and Claims sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Examples of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:

[0051] Figure la: schematically illustrates an autonomous vehicle or robot according to the present techniques.

[0052] Figure lb: schematically illustrates a vehicle according to the present techniques.

[0053] Figure 2: schematically illustrates a self-driving vehicle in a driving scenario.

[0054] Figure 3: schematically illustrates a representation of a driving scene according to the present techniques.

[0055] Figure 4: schematically illustrates a method according to the present techniques.

[0056] Figure 5a: schematically illustrates an example technique for determining feature vectors.

[0057] Figure 5b: schematically illustrates an example feature embedding space representation in two dimensions.

[0058] Figure 6: schematically illustrates a method according to the present techniques.

[0059] Figure 7: schematically illustrates a computing device that may implement the techniques described herein.

[0060] While the disclosure is susceptible to various modifications and alternative forms, specific example approaches are shown by way of example in the drawings and are herein described in detail. It should be understood however that the drawings and detailed description attached hereto are not intended to limit the disclosure to the particular form disclosed but rather the disclosure is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the claimed invention.

[0061] It will be recognised that the features of the above-described examples of the disclosure can conveniently and interchangeably be used in any suitable combination. DETAILED DESCRIPTION

[0062] Figure la schematically illustrates an example autonomous vehicle (also known as an SDV) or robot 1. Autonomous vehicle or robot 1 comprises perception circuitry 2. Perception circuitry 2 is configured to perceive an environment of the autonomous vehicle or robot 1, for example using one or more image, video, and / or sensing apparatuses. For example, perception circuitry 2 may comprise one or more camera, radar, LiDAR, SONAR, and / or LASER systems for capturing data of an environment of the autonomous vehicle or robot 1. Perception circuitry 2 may also comprise one or more sensors for sensing a state of the autonomous vehicle or robot 1. Such sensors may include position (e.g. GPS), speed, acceleration, braking, steering and / or movement sensors. Vehicle state refers to properties associated with a current state of the vehicle, such as driving properties, for example a current speed, position, acceleration, deceleration, and / or steering state.

[0063] Perception circuitry 2 may determine or otherwise collect data relating to the environment of the autonomous vehicle or robot 1 and / or the vehicle state and provide this data as an input to a driving policy 3 of the autonomous vehicle or robot 1 (as indicated by the line between perception circuitry 2 and driving policy 3 in figure la). For example, an autonomous vehicle (such as autonomous vehicle 1) may collect, determine, or sense data relating to its environment using on-board camera, radar, and / or LiDAR devices, and additionally collect, determine, or sense data relating to its state (e.g. properties relating to the current state of the vehicle such as driving state), using on-board sensors. Autonomous vehicle 1 may also obtain sensor data from one or more other autonomous vehicles using one or more vehicle-to-vehicle communication protocols. The sensor data may then be used as an input to a trained driving policy, such as driving policy 3.

[0064] Driving policy 3 may correspond to one or more of the driving policies described herein, for example a driving policy trained according to the techniques described herein.

[0065] Driving policy 3 may be a trained artificial neural network, configured or trained to take as input data relating to the environment of the autonomous vehicle or robot and vehicle state (such as that determined and provided by perception circuitry 2) and output data relating to, or indicative of, control of the autonomous vehicle or robot 1 in the environment. For example, the data output may comprise instructions or data indicative of instructions for controlling movement or behaviour of the autonomous vehicle or robot 1 in the environment and / or a trajectory or data indicative of a trajectory that the autonomous vehicle or robot 1 is to follow in a subsequent time period. Further detail of the training of the driving policy 3 is provided below.

[0066] It will be appreciated that the driving policy 3, or driving policies described herein, may be a hardware or software-based trained artificial neural network, or indeed emulated in software and thus considered hardware.

[0067] The data output from the driving policy 3 is provided to control circuitry 4 (as indicated by the line between driving policy 3 and control circuitry 4 in figure la). Control circuitry 4 is configured to control the autonomous vehicle or robot 1 based on the output of the driving policy 3. For example, the control circuitry 4 is configured to control movement and / or behaviour of the autonomous vehicle or robot 1 in its environment. For example, one or more systems of the autonomous vehicle or robot 1 may be controlled, such as a propulsion system (like an engine or electric motor), a braking system, and / or a turning system, etc., to follow a trajectory predicted by the driving policy 3 for a subsequent time period or based on data output by the driving policy 3 indicative of a predicted trajectory. As an example, one or more propulsion systems of the autonomous vehicle 1, such as an engine system or motor system, may be controlled to provide power to wheels or actuators to move the autonomous vehicle or robot 1 based on the output of the driving policy 3.

[0068] It will be appreciated that, in examples involving an autonomous robot, one or more actuators, motors, grippers, end effectors, etc. may be controlled based on the output of the driving policy 3, such as based on a predicted trajectory or data indicative of a predicted trajectory. It will be appreciated that, in examples involving an autonomous vehicle 1, the driving policy 3 may output data used by control circuity for controlling various systems of the autonomous vehicle. For example, an engine system, motor system, braking system, and / or steering system, etc.

[0069] Examples involving autonomous vehicles will now be described in further detail with reference to figure lb and figure 2. It will be appreciated that the term "autonomous vehicle" may refer to a selfdriving vehicle, such as a car, a van, a lorry or other vehicle, or an unmanned delivery robot, an aerial vehicle, for example an unmanned aerial vehicle or drone, or a robot with at least some autonomous capability. In examples involving unmanned aerial vehicles, it will be appreciated that the driving scenes described herein may refer to three dimensional scenes of UAV driving, for example airborne flight, landing and / or take-off. Indeed, it will be appreciated that the present techniques provide an advantageous training technique applicable to any type of computer-controlled vehicle or device that uses a driving policy (also known as a motion planner) to control its movements.

[0070] Figure lb shows a schematic illustration of a vehicle to which the teachings of the disclosure can be applied. Vehicle 100 is depicted as comprising a first image sensor 110, a second image sensor 120, further image sensors 130 and a control device 140. It will be understood that, in some examples, the second image sensor 120 and further image sensors 130 are optional and that a single image sensor may be provided. One or more of sensors 110, 120, and 130 may correspond to the perception circuitry of figure la.

[0071] In some examples, one or more of the first image sensor 110, the second image sensor 120, and the further image sensors 130 is a camera. The camera uses visible and / or invisible light. Thereby performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths. In other examples, one or more of the image sensors may be a LIDAR system, a RADAR system, a SONAR system and / or a LASER scanning system. Thereby, higher spatial accuracy may be achieved and performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths.

[0072] In the present example, control device 140 is a general-purpose computer, for example of the form depicted in figure 7. Thereby costs may be reduced, for example, by using non-specialised hardware. In other examples, the control device 140 may comprise a specialist computing component such as an ASIC or FPGA. Thereby power efficiency, speed and / or latency may be enhanced.

[0073] Control device 140 may correspond to control circuitry 4 of figure la. Further, control device 140 may comprise one or more processors for evaluating the driving policy 3 of figure la, and data storage for storing the trained driving policy 3 and the program instructions for evaluating the driving policy 3. It will be appreciated that control device 140 may be loaded with a trained driving policy, for example as described herein, and control device 140 may be configured to control the vehicle 100 based on the output of the driving policy.

[0074] Figure 2 schematically illustrates a driving scenario 5. Driving scenario 5 includes an autonomous vehicle or SDV 6 driving along a roadway 7. SDV 6 may correspond to the autonomous vehicle 1 of figure la, or the vehicle 100 of figure lb. As such, SDV 6 comprises perception circuitry for sensing data related to an environment of the SDV 6 and sensing data related to a state of the SDV 6. In this example, SDV 6 includes on-board camera, radar and LiDAR systems and on-board vehicle sensor systems for sensing the vehicle environment, like those described in relation to figures la and lb. The data captured by these systems may be processed by one or more on-board processors, such as control device 140 of figure lb.

[0075] For example, this captured data may be processed into an input format for the trained driving policy. It will be appreciated that this step will vary depending on the implementation and in some cases may not be performed.

[0076] Data relating to the perceived environment and sensed vehicle state may then then input to a trained driving policy to determine how to control the SDV 6, for example the trained driving policy 3 of figure la.

[0077] As shown in figure 2, also present in the driving scenario 5 and in the opposite and oncoming lane of the roadway 7, is another vehicle 8. Vehicle 8 may be an SDV or a human-operated vehicle (or indeed any other road user such as a pedestrian, cyclist, motorcyclist etc.). SDV 6 may perceive vehicle 8 using its on-board camera, radar and / or LiDAR systems. An on-board computing or processing circuitry processes the data captured by the camera, radar, and / or LiDAR systems to perform analysis of the data so as to determine the presence of the vehicle 8. Various image, video, and data analysis techniques may be used that could be envisaged by the skilled person, such as machine learning techniques to identify objects in the data.

[0078] On-board processing circuitry of the SDV 6 may also determine properties associated with the vehicle 8 and successive inputs from data recorded by the camera, radar, and / or LiDAR systems, for example to determine a speed, a changing position, and / or an acceleration of the vehicle 8.

[0079] Further, the driving scenario 5 comprises a building 9 and tree 10. It will be appreciated that driving scenarios may comprise an enormous range of objects such as structures, foliage, and street furniture. The perception circuitry of the SDV 6 perceives these objects of the scenario and the on-board processing circuitry may identify and classify the various objects.

[0080] In order for SDV 6 to navigate the roadway 7 safely, SDV 6 utilises a trained driving policy, also known as a motion planner, to make decisions as to how to control its movement and driving behaviour, for example a trajectory 11 to follow in a subsequent time period. In the example of figure 2, the output from the driving policy may be data indicative of a trajectory 11 that causes the SDV 6 to remain at a constant speed and in the centre of its lane on the roadway as the other vehicle 8 passes by in the opposite direction.

[0081] For the driving policy or motion planner (e.g. a neural network policy) to be able to control the driving behaviour of the SDV 6 across a range of driving scenarios in a safe and predictable manner, the driving policy is trained using one or more machine learning techniques. The training data for this training process will now be described in more detail with respect to figure 3.

[0082] Figure 3 shows a representation of an example driving scene 12 according to the present techniques. Driving scene 12 may be an example of a driving scene used to train the driving policy described herein. In other words, driving scene 12 is an example of training data used to train the driving policy described herein.

[0083] As shown, driving scene 12 depicts a driving scenario involving a road-way 14 having a three-way intersection (also known as a three-way junction). It will be appreciated that driving scene 12 is an example of training data, and that a large variety and number of driving scenes 12 may be used to train the driving policy. Each driving scene 12 may depict a driving scenario, for example, motorway scenes, intersections, junctions, parking lots, traffic light scenes, road sign scenes, multi-lane roadways, single lane roadways etc. Thus, a corpus of driving scenes (the training data) will comprise a great number of driving scenarios and examples of the same driving scenarios.

[0084] In figure 3, driving scene 12 is depicted at three successive times, tl, t2, and t3. Driving scenes may correspond to data describing the scene at successive time points. This may include data relating to the environment (such as data relating to objects in the environment) and data relating to the vehicle state. Thus figure 3 shows three successive snapshots of the driving scene 12. For example, driving scenes may correspond to data recorded by one or more camera, radar, and / or LiDAR systems over time captured by a recording vehicle (for example a vehicle equipped with environment perceiving circuitry) and also data relating to the vehicle state of the recording vehicle over time for the duration of the recorded driving scene. In one or more driving scenes, the vehicle may be driven using a trained driving policy. In one or more other driving scenes, the vehicle may be driven by a human to represent safe or 'good' driving behaviour. This data may then be reconstructed into data representing the scene that the SDV or recording vehicle experienced and the vehicle state for the duration of the recorded driving scene to form the driving scenes (i.e. training data) described herein.

[0085] In other words, to collect driving scene training data, a vehicle equipped with environment perceiving circuitry and vehicle state sensing circuity may be driven (by a trained driving policy or human) in a manner that corresponds to safe or 'good' driving behaviour for a period of time. During that period of time, the on-board environment perceiving circuity and vehicle state sensing circuitry records data relating to the perceived environment of the recording vehicle and data relating to the state of the recording vehicle. The data relating to the perceived environment may include, for successive time points during the recording period, one or more of image, radar, and / or LiDAR data associated with the environment (including perceived other road vehicles, pedestrians, objects in the environment, etc.). The data relating to the state of the recording vehicle may include, for successive time points during the recording period, one or more of a speed, position, acceleration, deceleration, and / or steering value of the recording vehicle. In some examples, the driving scene data comprises sensor data (recorded by a vehicle), vehicle GPS data (of the vehicle), and vehicle controlling signals (of the vehicle), each associated with timestamps indicative of when the data was generated.

[0086] Using this recorded data, the driving scene (i.e. the scene the recording vehicle was driving through when recording data), can be reconstructed in software to create a representation of the driving scene that can be used for training a driving policy. In other words, using the data captured by a recording vehicle 13, the driving scene, how the recording vehicle 13 traversed the driving scene, including information relating to its driving behaviour, and how other actors (like vehicles, pedestrians, other road users etc.) behaved can all be reconstructed in software. This reconstructed driving scene can then be used for training the driving policies described herein.

[0087] An example of a driving scene training dataset is the Woven by Toyota Prediction Dataset, titled One Thousand and One Hours: Self-driving Motion Prediction Dataset, authored by Houston, J. and Zuidhof, G. and Bergamini, L. and Ye, Y. and Jain, A. and Omari, S. and Iglovikov, V. and Ondruska, P., published in 2020, and available at https: / / woven.toyota / en / prediction-dataset, the contents of which is hereby incorporated by reference. A further example of a driving scene dataset is "nuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles.", Holger, Caesar, et al., arXiv preprint arXiv:2106.11810 (2021), and available at https: / / www.nuscenes.org / nuplan.

[0088] Turning back to figure 3, this depicts a representation of a driving scene 12, i.e. the reconstructed driving scene, captured from the perspective of a recording vehicle 13. The driving scene 12 at three successive time points is depicted, where the relative change in positions of the recording vehicle 13 and other road actors 15 is shown. Other road actors 15 are other vehicles on the roadway 14. It will be appreciated that other actors may include pedestrians, cyclists, and / or other road users.

[0089] As discussed above, the driving policy may be trained on a variety of different driving scenes, including for example driving scene 12 shown in figure 3. The training process will now be discussed in more detail with respect to figure 4.

[0090] Figure 4 shows a method 400 for training a driving policy, for example the driving policy / policies described herein such as driving policy 3 of figure la. Method 400 comprises steps 401, 402, 403, 404, 405, and 406. It will be appreciated that the steps of method 400 may be performed in an alternative order and that steps may be omitted.

[0091] Step 401 comprises first training, using a first machine learning technique, a first driving policy on a first dataset of driving scenes. The first driving policy may be a neural network driving policy model. The first dataset of driving scenes may be a subset of a wider corpus of driving scenes.

[0092] The first machine learning technique may be imitation learning. The imitation learning policy may be a machine learning model designed to solve the task, trained with an imitation learning loss function in a supervised setting optionally using a model optimiser. An example of an imitation learning policy and training method is a neural network, trained with an LI loss function to imitate the behaviour of the vehicle in the driving scenes, for example as discussed in 'SafetyNet: Safe planning for real-world selfdriving vehicles using machine-learned policies', version 1, submitted 28 September 2021, and available 15 at arXiv reference 2109.13602, the contents of which is hereby incorporated by reference. In some examples, additional loss functions could be added to further improve performance, for example the jerk and curvature regularization. The model optimiser may be a gradient-based optimiser, for example a stochastic gradient descent optimiser or adaptive moment estimation optimiser.

[0093] Step 402 comprises simulating, for each driving scene in a second dataset of driving scenes, driving behaviour in the driving scene based on the trained driving policy. In other words, the performance of the trained driving policy is evaluated using closed-loop simulation.

[0094] The closed-loop simulation uses software configured to replay a recorded driving scene and alter actions performed by actors present in the driving scene using the output of a decision-making model (such as the trained driving policies described herein). As discussed above in relation to figure 3, each driving scene includes data relating to the environment and vehicle state for a time period, and so the simulation may use this recorded data to recreate the driving scene in software. In particular, in the simulation, the recording vehicle in the driving scene (for example that of figure 3) may be replaced with a simulated vehicle controllable using the trained driving policy. In other words, the simulated vehicle drives in the reconstructed driving scene according to the outcomes of the trained driving policy, and replaces the actions of the recording vehicle. A comparison between the decisions and driving behaviour of the simulated vehicle may therefore be made against the decisions and driving behaviour of the recording vehicle (which represents the ground truth for driving behaviour). This comparison may be used to determine performance metrics associated with the driving of the simulated vehicle and thus the trained driving policy.

[0095] Indeed, the simulation may be capable of reviewing the simulated driving scene, comparing the simulated driving scene with the recorded driving scene and producing one or more performance metrics. This review and metric generation may be performed by the same or different software that performed the simulation. Examples of simulators include a simulator using intermediate-level representation, representing the scene as vectors and updating the self-driving vehicle location according to the given actions (jerk, acceleration, etc.) and a vehicle kinematic model. This simulator may be extended to simulate the behaviour of other actors in the driving scene, such as other vehicles acting in response to the simulated vehicle. An example of simulation software is CARLA, an open-source simulator available at carla.org.

[0096] Step 403 comprises determining, for each driving scene in the second dataset of driving scenes, one or more performance metrics of the trained first driving policy associated with the simulated driving behaviour.

[0097] During step 403, one or more performance metrics associated with the simulated driving behaviour are determined. These performance metrics may include metrics associated with the simulated driving behaviour and / or metrics associated with a difference between the simulated driving behaviour and the recorded driving behaviour (the ground truth driving behaviour).

[0098] Metrics associated with the simulated driving behaviour may include metrics relating to harsh acceleration, harsh braking, driving off-route, collisions or near collisions with other agents or actors in the simulation (vehicles, cycles, pedestrians) or static obstacles (trees, poles, street furniture, etc.) from the front, side rear, running red lights, the type of actor collided with (car, lorry, van, cycle, pedestrian), running stop signs, speed limit violation, average speed, and / or recorded maximum speed. Metrics associated with a difference between the simulated driving behaviour and the recorded driving behaviour may include metrics relating to a distance between the simulated vehicle and the recorded vehicle in the driving scene, a lateral distance from a recorded trajectory, and / or a difference in speed / acceleration / curvature.

[0099] Step 404 comprises selecting, based on the one or more performance metrics, a subset of the driving scenes in the second dataset.

[00100] The selection in step 404 may be based on the one or more determined performance metrics satisfying pre-determined criteria. For example, the one or more performance metrics may indicate that the driving policy has controlled the simulated vehicle to behave in an unsafe or unpredictable manner during simulation of a subset of the driving scenes. In this case, the driving policy has performed poorly on a subset of the scenes being simulated, and as such it would be advantageous to select this subset of scenes where performance of the driving policy relates to unsafe or unpredictable driving behaviour.

[00101] In some examples, the pre-determined criteria may be that a given performance metric satisfies or fails to satisfy a pre-determined threshold associated with that performance metric. The threshold for a given performance metric may depend on the specific metric and in particular the severity of the metric. These thresholds may be determined based on the desired performance of the driving policy and will thus depend on implementation.

[00102] As an example, the threshold for a performance metric associated with a collision, which is one of the worst failure states, could be 1 on a scale of 0 to 1 (in other words, the scene would be selected if any collision occurs during simulation of the scene). As a further example, a threshold for a performance metric associated with jerk or harsh braking could be defined according to human comfort levels (in other words, the scene would be selected if at least one acceleration / braking event would be considered uncomfortable for humans). As a yet further example, a threshold for a performance metric indicating a simulated and recorded driving behaviour mismatch might be a numerical value indicating a distance such 17 as 10 meters (in other words, select the scene if the simulated behaviour leads to a position of the simulated vehicle which is at least 10 meters away from the recorded position of the recorded vehicle in the driving scene).

[00103] It will be appreciated that, rather than selecting scenes in the second dataset where the driving policy is considered to have performed poorly (i.e. unsafely), the selection may instead result in scenes where the driving policy is considered to have performed well (i.e. safely). In this case, the pre-determined criteria and thresholds may be set accordingly. For example, the pre-determined criteria may be the inverse of that set for the selection of unsafe or poor driving, or the pre-determined criteria may be different in a different manner.

[00104] Step 405 comprises assembling a third dataset of driving scenes based on a similarity between a plurality of driving scenes and the subset of the driving scenes in the second dataset.

[00105] Once the subset of driving scenes has been selected in step 404, for example selected as they are scenes where the driving policy has performed poorly, a third dataset of driving scenes is assembled based on a determined similarity with the subset of driving scenes of the second dataset. In other words, scenes similar but different to the selected scenes of the second dataset are assembled. This results in a very advantageous augmentation of the training dataset, where scenes in a broader training corpus are assembled to add new scenes into the training process that are particularly helpful to the training process.

[00106] The assembling of the third dataset of step 405 may be based on using a trained similarity model. For example, a trained artificial neural network (a similarity model) may be trained and configured to transform driving scenes into multi-dimensional feature vectors. Hence, driving scenes of a driving scene training corpus and also driving scenes of the subset of the second dataset (i.e. the scenes selected for the similarity determination) may be transformed into multi-dimensional feature vector representations and these representations may be compared to determine a similarity.

[00107] The similarity model may be a neural network configured or trained to convert a driving scene into a feature vector such that similar driving scenes are encoded to numerically close feature vectors and driving scenes that are dissimilar are encoded to numerically far apart feature vectors. The input for the similarity model may be trajectories in the driving scene, for example of the recording vehicle and / or other vehicles or objects in the driving scene. Thus, in some examples, the similarity model determines scenes that have similar trajectories to the trajectories in the driving scenes of the subset of the second dataset. The training data for the similarity model may be the driving scenes as described herein, for example as described in relation to figure 3.

[00108] In some examples, the similarity model comprises an autoencoder that takes a driving scene as an input, compresses the driving scene to a small-sized vector and then reconstructs the original driving scene. For example, assuming that a driving scene, for example the trajectories of the recording vehicle and / or other vehicles in the scenes, is represented as a set of 3D vectors, an autoencoder could be defined and trained to encode a scene into an embedding space and decode the scene into a reconstructed scene, and act so as to minimise a reconstruction loss. This is shown in figure 5a. In other words, an autoencoder may be trained to reconstruct trajectories in a driving scene, and create searchable embeddings. This approach may also support data introspection.

[00109] Another example is a combination of two PerceiverlO architectures to embed inputs of a variable input size into a single embedding. The resulting neural embeddings are multi-dimensional vectors.

[00110] For example, once transformed, the multi-dimensional vectors of the subset of the second dataset can be compared to multi-dimensional feature vectors of other driving scenes to determine similar scenes to assemble into the third dataset. Similar scenes may be determined based on a distance in the multi-dimensional feature space between the multi-dimensional feature vectors satisfying one or more criteria, such as the distance between the multi-dimensional feature vectors being below a predetermined distance to classify the feature vectors, and thus scenes, as being similar. The criteria, such as the distance threshold to classify two feature vectors as similar, may depend on implementation. The number of scenes selected may depend on a predetermined number of required similar scenes. Further, in some examples, the x most similar driving scenes are selected (based on the distance between the multi-dimensional feature vectors), where x is a predetermined number. In other words, the x closest scenes are selected rather than every scene that is within the pre-determined distance to classify the feature vectors as being similar.

[00111] As a result, a third dataset of driving scenes is compiled having driving scenes similar but different to the scenes where the driving policy has performed poorly, i.e. unsafely (or well depending on implementation). This similarity determination is further illustrated by figure 5b.

[00112] Figure 5b shows an example feature embedding space representation in two dimensions. The triangle corresponds to a feature embedding of a driving scene of the subset of the second dataset. In other words, the triangle corresponds to one of the scenes of the second dataset selected based on the one or more performance metrics, such as a scene where the trained driving policy performed unsafely. The stars correspond to feature embeddings of driving scenes of a driving scene corpus.

[00113] As shown in figure 5b, the feature embedding of the driving scene of the subset of the second dataset (the triangle) is compared to two feature embeddings of driving scenes of a driving scene corpus (the two stars, although it will be appreciated that additional feature embeddings not shown in figure 5b 19 may also be present), and the distances x and y between the triangle and the stars may be determined. The distance may be measured using Euclidean distance, cosine or dot product, depending on implementation. If this distance x or y in feature embedding space is below a predetermined value, the driving scene corresponding to the star in question may be considered similar to the driving scene corresponding to the triangle and assembled as part of the third dataset. For example, if the predetermined value (the similarity threshold) is s and y <s <x, the star (i.e. feature vector and thus driving scene) corresponding to the distance y may be considered similar and the star (i.e. feature vector and thus driving scene) corresponding to the distance x may not be considered similar. In another example, if the predetermined value is s and y <x <s, both of the stars (i.e. feature vectors and thus driving scenes) in figure 5b may be considered similar. In this example, both of the driving scenes associated with the features embeddings (the stars) may be added to the third dataset. However, in other examples, only the driving scene associated with the nearest feature embedding of the two similar feature embeddings may be added to the third dataset, as discussed above. The number of driving scenes added to the third dataset may vary depending on configuration and implementation. This process may be repeated for each of the driving scenes of the subset of the second dataset to assemble a third dataset having a plurality of driving scenes.

[00114] Step 406 comprises second training, using a second machine learning technique, a second driving policy on a fourth dataset, wherein the fourth dataset comprises the third dataset.

[00115] The second machine learning technique may be the same or different from the first machine learning technique. For example, the second machine learning technique may be imitation learning. The imitation learning policy may be a machine learning model designed to solve the task, trained with an imitation learning loss function in a supervised setting optionally using a model optimiser, such as that described in relation to the first machine learning technique.

[00116] The second driving policy may be the first driving policy. In other words, the first driving policy is re-trained or refined using the fourth dataset of driving scenes. In some cases, the second training uses the first driving policy as an input. Alternatively, the second training does not use the first driving policy as input. In other words, a new driving policy is trained using the fourth dataset. For example, a first driving policy may be trained, its performance evaluated and used to assemble the third dataset, but then a different driving policy is trained using the assembled third dataset.

[00117] Thus, as described herein, there is provided a driving policy training method that is particularly advantageous.

[00118] A further example method will now be described with reference to figure 6.

[00119] As shown at step 1, a large set of driving logs (also known herein as driving scenes) is stored in a cloud database and is sampled randomly to select the training data for the training of the driving policy using imitation learning and also for the training of the scene similarity model.

[00120] At step 2, the closed-loop simulation is run and performance metrics are collected. As described above, a small set of significant driving logs is determined (for example where the driving policy has performed badly), and these scenes are input into the scene similarity model in step 3.

[00121] More particularly, in step 3, scenes similar to the small set of significant driving logs determined in step 2 are found using the similarity model, to assemble a new set of significant driving logs. As shown by the arrow from the cloud database, the new driving logs may be retrieved from the cloud database.

[00122] In step 4, the new set of significant driving logs are then used to re-train or fine-tune a driving policy as described herein. As denoted by the dotted return line to the closed-loop simulation step of step 2, the loop may be closed by using the new driving policy and re-evaluating the closed-loop simulation metrics.

[00123] The trained driving policy of step 4 is then deployed in an SDV in step 5. Vehicle sensor data, such as perception data and vehicle state data, is then collected by the SDV while the SDV is controlled using the trained driving policy. This recorded data may be uploaded to the cloud database of driving logs (as illustrated by the dotted return line to the cloud database).

[00124] An example computing apparatus will now be described with reference to figure 7. Figure 7 schematically illustrates an example of a computing device 700 which can be used to implement teachings described above, for example method 400 or the method of figure 5a, and figure 6. In some examples, the computing device 700 may correspond to a computing device provided at the autonomous vehicle or robot (or SDV) described herein, which may be loaded with the trained driving policy as described herein. In addition, computing device 700 can, in some examples, correspond to control device 140 of figure lb.

[00125] The computing device 700 has processing circuitry 710 for performing data processing in response to program instructions and data storage 720 for storing data and instructions to be processed by the processing circuitry 710. It will be appreciated that in some examples (such as those corresponding to method 400) computing device 700 may be a distributed computing device, for example using a cloud computing infrastructure comprising high-powered compute nodes. It will be appreciated that in some examples (such as those corresponding to method 400) the data storage 720 may also be provided using a cloud storage server. In some examples, the processing circuitry 710 includes one or more caches for caching recent data or instructions. It will be appreciated that Figure 7 is merely an example of possible hardware that may be provided in the computing device and other components may also be provided. For example, the device may include a dedicated sensor interface 730 for communicating with sensors. As another example, for some devices for which user interaction is expected, the device may be provided with one or more user input / output device(s) 740 to receive input from a user or to output information to a user. The computing device 700 may additionally or alternatively have a communications interface 750 for communicating with external devices. For example, communications interface 760 could use any of a range of different communication protocols, such as Ethernet, WiFi®, Bluetooth®, ZigBee®, etc.

[00126] The methods discussed above may be performed under control of a computer program executing on a computing device. Hence a computer program may comprise instructions for controlling a computing device to perform any of the methods discussed above. The program can be stored on a computer-readable medium. A computer readable medium may include non-transitory type media such as physical storage media including storage discs and solid state devices. A computer readable medium may additionally or alternatively include transient media such as carrier signals and transmission media. A computer-readable storage medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[00127] In the present application, the words "configured to..." are used to mean that an element of an apparatus has a configuration able to carry out the defined operation. In this context, a "configuration" means an arrangement or manner of interconnection of hardware or software. For example, the apparatus may have dedicated hardware which provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not imply that the apparatus element needs to be changed in any way in order to provide the defined operation.

[00128] The various embodiments described herein are presented only to assist in understanding and teaching the claimed features. These embodiments are provided as a representative sample of embodiments only, and are not exhaustive and / or exclusive. It is to be understood that advantages, embodiments, examples, functions, features, structures, and / or other aspects described herein are not to be considered limitations on the disclosure scope defined by the claims or limitations on equivalents to the claims, and that other embodiments may be utilised and modifications may be made without departing from the scope of the invention as defined by the claims.

Claims

1. A computer-implemented method for training a driving policy, the method comprising: first training, using a first machine learning technique, a first driving policy on a first dataset ofdriving scenes;simulating, for each driving scene in a second dataset of driving scenes, driving behaviour in the driving scene based on the trained driving policy;determining, for each driving scene in the second dataset of driving scenes, one or more performance metrics of the trained first driving policy associated with the simulated driving behaviour;selecting, based on the one or more performance metrics, a subset of the driving scenes in the second dataset;assembling a third dataset of driving scenes based on a similarity between a plurality of driving scenes and the subset of the driving scenes in the second dataset;second training, using a second machine learning technique, a second driving policy on a fourth dataset, wherein the fourth dataset comprises the third dataset.

2. The computer-implemented method of claim 1, wherein the second training uses the first driving policy as an input.

3. The computer-implemented method of claim 1, wherein the second training does not use the first driving policy as an input.

4. The computer-implemented method of any preceding claim, wherein the first machine learning technique and the second machine learning technique are the same or different.

5. The computer-implemented method of any preceding claim, wherein the first dataset, second dataset, third dataset, and fourth dataset comprise real-world driving scenes.

6. The computer-implemented method of any preceding claim, wherein the fourth dataset comprises the first dataset and the third dataset.

7. The computer-implemented method of any preceding claim, wherein the third dataset comprises one or more driving scenes which are not present in either the first dataset or the second dataset.

8. The computer-implemented method of any preceding claim, further comprising iterating the simulating, determining, selecting, assembling and second training steps one or more times until a predetermined condition is satisfied.

9. The computer-implemented method of any preceding claim, wherein the first and / or second machine learning technique is an imitation learning technique or a reinforcement learning technique.

10. The computer-implemented method of any preceding claim, wherein selecting the subset of the simulated driving scenes is based on the one or more performance metrics satisfying pre-determined criteria.

11. The computer-implemented method of any preceding claim, wherein the one or more performance metrics comprise metrics associated with one or more of acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and a difference between the simulated driving behaviour and stored driving data.

12. The computer-implemented method of any preceding claim, wherein simulating driving behaviour, for each driving scene in the second dataset of driving scenes, comprises: controlling driving behaviour of a simulated agent in each driving scene based on the trained driving policy.

13. The computer-implemented method of claim 12, wherein simulating driving behaviour, for each driving scene in the second dataset of driving scenes, further comprises altering behaviour of actors in the driving scenes based on the simulated driving behaviour of the simulated agent.

14. The computer-implemented method of any preceding claim, wherein assembling the third dataset of driving scenes is based on using a similarity model.

15. The computer-implemented method of claim 14, wherein the similarity model is a neural network configured to transform driving scenes into multi-dimensional feature vectors.

16. The computer-implemented method 15, wherein assembling the third dataset of driving scenes is based on transforming the plurality of driving scenes and the subset of simulated driving scenes into multi-dimensional feature vectors and determining a similarity between the transformed multidimensional feature vectors.Yl. The computer-implemented method of any preceding claim, wherein the first and / or second driving policy is a neural network.

18. The computer-implemented method of any preceding claim, wherein the first dataset of driving scenes is created based on random sampling of driving scenes.

19. The computer-implemented method of any preceding claim, further comprising using the first and / or second trained driving policy in a motion planner of an autonomous vehicle.

20. The computer-implemented method of any preceding claim, further comprising controlling an autonomous vehicle or robot based on the first and / or second trained driving policy.

21. The computer-implemented method of claim 20, further comprising collecting real-world driving data based on controlling the autonomous vehicle or robot.

22. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method of claims 1 to 21.

23. An apparatus comprising a processor configured to carry out the method of claims 1 to 21.

24. A computer-readable medium comprising a driving policy trained according to the method ofclaims 1 to 21.

25. An apparatus comprising memory comprising a driving policy trained according to the method of claims 1 to 21.

26. The apparatus of claim 25, wherein the apparatus is a robot or an autonomous vehicle, for example an autonomous car or autonomous airborne vehicle.

27. A method for controlling an autonomous vehicle, comprising controlling the autonomous vehicle based on a driving policy trained according to the method of claims 1 to 21.

Images