Driving policy training

Abstract

A computer‐implemented method for training a driving policy comprising: obtaining a plurality of trajectories output from one or more driving policies based on inputting an initial state of a d

Description

FIELD AND BACKGROUND

[0001] The present techniques relate to the field of autonomous driving. More particularly, but not exclusively, the present techniques relate to training driving policies for controlling autonomous vehicles.

[0002] In recent times, the use of autonomous vehicles (also known as self-driving vehicles, SDVs) has become more widespread. Autonomous vehicles are vehicles capable of sensing their own environment and operating without human involvement. The degree of autonomy may vary from fully autonomous requiring no human involvement (sometimes known as Level 5 - Full Driving Automation), to lower levels of autonomy requiring some degree of human involvement (Levels 1 to 4). As used herein, an autonomous vehicle refers to a vehicle or robot having at least some capability of operating without human involvement.

[0003] Autonomous vehicles may be controlled or operated using a driving policy (also known as a motion planner). The driving policy may be a machine learning model trained using machine learning techniques. SUMMARY

[0004] Particular aspects and embodiments are set out in the appended claims.

[0005] Viewed from a first aspect, there is provided a computer-implemented method for training a driving policy comprising: obtaining a plurality of trajectories output from one or more driving policies based on inputting an initial state of a driving scene to the one or more driving policies; simulating driving behaviour in the driving scene based on each of the plurality of trajectories; determining, based on the simulated driving behaviour, one or more performance metrics associated with each of the plurality of trajectories; awarding scores to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories; first training, using the scores and the plurality of trajectories, a reward model to output a score associated with a trajectory; and second training, using reinforcement learning and the trained reward model and based on a plurality of driving scenes, a driving policy.

[0006] Hence, the present approach provides an improved method for training a driving policy with reinforcement learning that uses closed-loop evaluation and a reward model optimised to select trajectories that perform particularly well in closed-loop evaluation. The present inventors have identified that previous techniques fortraining a driving policy may not utilise feedback from closed-loop evaluation. Further, previous techniques may use imitation learning rather than reinforcement learning when training a driving policy. However, imitation learning may be susceptible to co-variate shifts under a closed-loop 1 evaluation. For example, if a driving policy is evaluated in open-loop, it may not be possible to determine whether the policy has executed on a previous step, i.e. whether the driving policy is following its own predicted steps (whereas in closed-loop evaluation this may be possible). Over time, errors can compound. Further, a certain level of performance in an open-loop evaluation does not necessarily correspond to a similar level of performance in a closed-loop evaluation. The present inventors have identified that closed-loop behaviour may better represent real world driving and thus it would be advantageous to provide a technique that may utilise the results of a closed-loop evaluation to train a driving policy.

[0007] However, the present inventors have further identified that incorporating real-world rules into reinforcement learning can be impractical. Thus, the present technique overcomes this by using a combination of closed-loop evaluation to train a reward model which may then be used in the reinforcement learning of the driving policy.

[0008] Indeed, the simulation results in the awarding of scores and then the reward model may be trained based on these scores. Thus, the reward model may be trained to score unseen trajectories (and unscored trajectories) and output scores for the trajectories that imitates the behaviour of running the closed-loop simulation without actually performing the closed-loop simulation. This results in the determination of a reward model that may then be repeatedly deployed to score trajectories (without also performing a closed-loop simulation). As such, training efficiency is increased (as the closed-loop simulation may only need to be performed once to determine the reward model). Furthermore, the reward model may be trained to score trajectories in a similar way to that which a closed-loop simulation would score trajectories. Accordingly, disadvantages associated with open-loop simulation as discussed herein may be avoided.

[0009] Thus, improved closed-loop performance and thus improved real-world performance of the driving policy may be achieved. Further, by training and then using a reward model in the reinforcement learning of the driving policy, training of the driving policy may be more efficient. For example, less time and computing resources may be required for the driving policy to meet a predetermined level of performance.

[0010] In some examples, the second training uses the output of the trained reward model in the reinforcement learning to train the driving policy. Thus, as discussed herein, the efficiency of training and performance of the driving policy may be increased.

[0011] In some examples, the second training uses the one or more trained driving policies as an input. Thus, in some cases, the present techniques may be used to 'fine-tune' or further improve pre-existing driving policies. For example, the same driving policy or policies used to generate the plurality of 2 trajectories may then be trained using reinforcement learning and the reward model to further train the driving policy or driving policies. This increases configurability of the technique and also allows for the efficient fine-tuning of pre-existing or trained driving policies using a closed-loop evaluation. As a result, the driving policy performance may be increased when evaluated on real-world driving scenarios.

[0012] In some examples, the second training does not use the one or more trained driving policies as an input. Thus, in some examples, a different or new driving policy may be trained using the reward model determined from trajectories obtained from other driving policies. This may be advantageous in cases where a single reward model is determined and used to train a large number of driving policies, or where the one or more driving policies used to obtain the trajectories are 'off-the-shelf' driving policies or otherwise obtained from external sources.

[0013] In some examples, the second training uses a further trained driving policy as an input. Thus, in some cases, the second training may use as input a trained driving policy in addition to one or more driving policies used to obtain the trajectories. Accordingly, the flexibility and configurability of the approach may be increased and may be adapted depending on a given implementation.

[0014] In some examples, simulating driving behaviour comprises simulating driving behaviour in the driving scene based on each of the plurality of trajectories for a plurality of successive time intervals, and wherein determining the one or more performance metrics comprises determining one or more performance metrics associated with each of the plurality of trajectories during each of the plurality of successive time intervals.

[0015] Thus, as described herein, the closed-loop simulation may determine performance metrics across a number of time intervals to evaluate the performance of the different trajectories during the time intervals. As a result, performance of the trajectories may be more effectively determined and closed-loop evaluation is realised. Thus, the likelihood that errors may compound over time, for example as a result of open-loop simulation, can be reduced. Accordingly, real-world performance of a driving policy may be increased.

[0016] In some examples, a simulation model is used to simulate the driving behaviour.

[0017] In some examples, the first training uses regression or classification training. Thus, the reward model may be trained using regression or classification training to output a score associated with a trajectory. The reward model may therefore be trained in an efficient manner.

[0018] In some examples, the one or more performance metrics comprise metrics associated with: acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and / or a difference between the simulated driving behaviour and stored reference driving data. Thus, the closed-3 loop evaluation may evaluate metrics associated with real-world driving to determine the performance of the trajectories. Accordingly, the reward model may be optimised to score trajectories that perform better based on evaluation of real-world driving metrics. Thus, the performance of the driving policy trained using the second training in real-world driving scenarios may be increased.

[0019] In some examples, simulating driving behaviour comprises: controlling driving behaviour of a simulated agent in each driving scene to follow each of the plurality of trajectories. In some examples, the simulation may use simulation software. Thus, a closed-loop simulation may be performed that causes a simulated agent to traverse each of the obtained trajectories, and the performance of the trajectories may be evaluated to determine the better scoring trajectories.

[0020] In some examples, simulating driving behaviour comprises altering behaviour of actors in the driving scenes based on the simulated driving behaviour of the simulated agent. Thus, behaviour of various agents and actors may be simulated, creating a more realistic simulation of real-world driving, where vehicles respond to the action of other vehicles. Thus, the reward model may be trained to 'mimic' a closed-loop simulation that better represents real-world driving. As a result, the scores may be more suitable for scoring real-world driving and so the second training of the driving policy may result in a driving policy with increased real-world performance.

[0021] In some examples, awarding scores to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories comprises: ranking each of the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories; and awarding scores to the plurality of trajectories based on the ranking. Thus, the trajectories may be scored based on rankings of how well each trajectory performs for a given performance metric. Hence, scores may be efficiently awarded based on performance of the trajectories. In some examples, the scoring may be based on rankings for a plurality of different performance metrics. Thus, in these examples, a more detailed assessment of performance across different metrics may be realised. In some examples, a higher ranking corresponds to a higher score being awarded to that trajectory.

[0022] In some examples, the ranking is based on comparing the one or more performance metrics associated with each of the plurality of trajectories to performance metrics associated with others of the plurality of trajectories. Thus, a cross-comparison of performance between trajectories may be performed to determine the ranking of trajectories.

[0023] In some examples, the plurality of driving scenes comprises real-world driving. In some examples, the driving scene comprises a real-world driving scene. Thus the performance of the trajectories may be simulated using real-world driving scenes. This provides a more accurate determination of performance 4 of trajectories, ultimately resulting in a driving policy that performs better in real-world driving scenarios. The real-world driving scenes may correspond to driving scenes collected by one or more vehicles as they are driven in the real-world. The real-world driving scenes may be taken from an existing corpus of real-world driving scenes that cover a range of different driving scenarios.

[0024] In some examples, the one or more trained driving policies and / or driving policy is an artificial neural network. It will be appreciated that the neural network may be an artificial neural network implemented as hardware or software, or indeed emulated in software and thus considered hardware. Examples of such artificial neural network driving policies that may be used with the techniques described herein include models like SafetyNet, SafePathNet, and Wayformer.

[0025] In some examples, the method further comprises: initial training, based on driving scene data and using a machine learning technique, one or more initial driving policies to generate the one or more trained driving policies. In some examples, the machine learning technique used in the initial training is imitation learning. Thus, the present technique may initially train a driving policy using imitation learning, and then this driving policy may be enhanced or 'fine-tuned' using the reward model and the reinforcement learning. Accordingly, configurability of the approach is increased and various machine learning techniques may be combined depending on implementation.

[0026] In some examples, the obtaining, simulating, determining, awarding, and first training steps are repeated for a plurality of different driving scenes. Thus, these steps may repeated for a number of different driving scenes, for example for a predetermined number of driving scenes, to further enhance the accuracy of the reward model's ability to accurately predict scores associated with the trajectories.

[0027] In some examples, the method further comprises using the driving policy output from the second training in a motion planner of an autonomous vehicle. Thus, the driving policy may be used in an autonomous vehicle, for example to control operation of the autonomous vehicle.

[0028] According to a second aspect, there is provided a computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method as described above and herein.

[0029] According to a third aspect, there is provided an apparatus comprising one or more processors configured to carry out the method described herein.

[0030] According to a fourth aspect, there is provided a computer-readable medium comprising a driving policy trained according to the method described herein.

[0031] According to a fifth aspect, there is provided an apparatus comprising memory comprising a driving policy trained according to the method described herein.

[0032] In some examples, the apparatus comprising memory described above is a robot or an autonomous vehicle, for example an autonomous car or autonomous airborne vehicle.

[0033] Other aspects will also become apparent upon review of the present disclosure, in particular upon review of the Brief Description of the Drawings, Detailed Description and Claims sections. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Examples of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:

[0035] Figure la: schematically illustrates an autonomous vehicle.

[0036] Figure lb: schematically illustrates an autonomous vehicle.

[0037] Figure 2: schematically illustrates a driving scenario.

[0038] Figure 3: schematically illustrates steps for training a driving policy as described herein.

[0039] Figure 4: schematically illustrates a simulation of a driving scene as described herein.

[0040] Figure 5: schematically illustrates steps for training a driving policy as described herein.

[0041] Figure 6a: schematically illustrates steps for training a reward model as described herein.

[0042] Figure 6b: schematically illustrates steps for training a driving policy as described herein.

[0043] Figure 7: schematically illustrates a computing device that may implement the techniques described herein.

[0044] While the disclosure is susceptible to various modifications and alternative forms, specific example approaches are shown by way of example in the drawings and are herein described in detail. It should be understood however that the drawings and detailed description attached hereto are not intended to limit the disclosure to the particular form disclosed but rather the disclosure is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the claimed invention.

[0045] It will be recognised that the features of the above-described examples of the disclosure can conveniently and interchangeably be used in any suitable combination. DETAILED DESCRIPTION

[0046] Figure la schematically illustrates an example autonomous vehicle 1 (also known as an SDV). Autonomous vehicle 1 comprises perception circuitry 2. Perception circuitry 2 is configured to perceive an environment of the autonomous vehicle 1, for example using one or more sensors, such as one or more image, video, and / or sensing apparatuses. For example, perception circuitry 2 may comprise one or more camera, radar, LiDAR, SONAR, and / or LASER systems for capturing data of an environment of the autonomous vehicle 1. Perception circuitry 2 may also comprise one or more sensors for sensing a state of the autonomous vehicle 1. Such sensors may include position (e.g. GPS), speed, acceleration, braking, steering and / or movement sensors. Vehicle state refers to properties associated with a current state of the vehicle, such as driving properties, for example a current speed, position, acceleration, deceleration, and / or steering state.

[0047] Perception circuitry 2 may determine or otherwise obtain data relating to the environment of the autonomous vehicle 1 and / or the vehicle state and provide this data as an input to a driving policy 3 of the autonomous vehicle 1 (as indicated by the line between perception circuitry 2 and driving policy 3 in figure la). For example, an autonomous vehicle (such as autonomous vehicle 1) may collect, determine, or sense data relating to its environment using on-board camera, radar, and / or LiDAR devices, and additionally collect, determine, or sense data relating to its state (e.g. properties relating to the current state of the vehicle such as driving state), using on-board sensors. Autonomous vehicle 1 may also obtain sensor data from one or more other autonomous vehicles using one or more vehicle-to-vehicle communication protocols. The sensor data may then be used as an input to a trained driving policy, such as driving policy 3.

[0048] Driving policy 3 may correspond to one or more of the driving policies described herein, for example a driving policy trained according to the techniques described herein.

[0049] Driving policy 3 may be a trained artificial neural network, configured or trained to take as input data relating to the environment of the autonomous vehicle and vehicle state (such as that determined and provided by perception circuitry 2) and output data relating to, or indicative of, control of the autonomous vehicle 1 in the environment. For example, the data output may comprise instructions or data indicative of instructions for controlling movement or behaviour of the autonomous vehicle 1 in the environment and / or a trajectory or data indicative of a trajectory that the autonomous vehicle or robot 1 is to follow in a subsequent time period. Further detail of the training of the driving policy 3 is provided below.

[0050] It will be appreciated that the driving policy 3, or driving policies described herein, may be a hardware or software-based trained artificial neural network, or indeed emulated in software and thus considered hardware.

[0051] The data output from the driving policy 3 is provided to control circuitry 4 (as indicated by the line between driving policy 3 and control circuitry 4 in figure la). Control circuitry 4 is configured to control the autonomous vehicle 1 based on the output of the driving policy 3. For example, the control circuitry 4 is configured to control movement and / or behaviour of the autonomous vehicle 1 in its environment. For example, one or more systems of the autonomous vehicle 1 may be controlled, such as a propulsion system (like an engine or electric motor), a braking system, and / or a turning system, etc., to follow a trajectory predicted by the driving policy 3 for a subsequent time period or based on data output by the driving policy 3 indicative of a predicted trajectory. As an example, one or more propulsion systems of the autonomous vehicle 1, such as an engine system or motor system, may be controlled to provide power to wheels or actuators to move the autonomous vehicle 1 based on the output of the driving policy 3.

[0052] It will be appreciated that, in examples involving an autonomous robot, one or more actuators, motors, grippers, end effectors, etc. may be controlled based on the output of the driving policy 3, such as based on a predicted trajectory or data indicative of a predicted trajectory. It will be appreciated that, in examples involving an autonomous vehicle 1, the driving policy 3 may output data used by control circuity for controlling various systems of the autonomous vehicle. For example, an engine system, motor system, braking system, and / or steering system, etc.

[0053] An example involving an autonomous vehicle will now be described in further detail with reference to figure lb. It will be appreciated that the term "autonomous vehicle" may refer to a selfdriving vehicle, such as a car, a van, a lorry or other vehicle, or an unmanned vehicle such as an unmanned delivery robot, an aerial vehicle, for example an unmanned aerial vehicle or drone, or a robot with at least some autonomous capability. In examples involving unmanned aerial vehicles, it will be appreciated that the driving scenes described herein may refer to three dimensional scenes of UAV driving, for example airborne flight, landing and / or take-off. Indeed, it will be appreciated that the present techniques provide an advantageous training technique applicable to any type of computer-controlled vehicle or device that uses a driving policy (also known as a motion planner) to control its movements.

[0054] Figure lb shows a schematic illustration of a vehicle to which the teachings of the disclosure can be applied. Vehicle 100 is depicted as comprising a first image sensor 110, a second image sensor 120, further image sensors 130 and a control device 140. It will be understood that, in some examples, the second image sensor 120 and further image sensors 130 are optional and that a single image sensor may be provided. One or more of sensors 110, 120, and 130 may correspond to the perception circuitry of figure la.

[0055] In some examples, one or more of the first image sensor 110, the second image sensor 120, and the further image sensors 130 is a camera. The camera uses visible and / or invisible light. Thereby performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths. In other examples, one or more of the image sensors may be a LIDAR system, a RADAR system, a SONAR system and / or a LASER scanning system. Thereby, higher spatial accuracy may be achieved and performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths.

[0056] In the present example, control device 140 is a general-purpose computer, for example of the form depicted in figure 7. Thereby costs may be reduced, for example, by using non-specialised hardware. In other examples, the control device 140 may comprise a specialist computing component such as an ASIC or FPGA. Thereby power efficiency, speed and / or latency may be enhanced.

[0057] Control device 140 may correspond to control circuitry 4 of figure la. Further, control device 140 may comprise one or more processors for evaluating the driving policy 3 of figure la, and data storage for storing the trained driving policy 3 and the program instructions for evaluating the driving policy 3. It will be appreciated that control device 140 may be loaded with a trained driving policy, for example as described herein, and control device 140 may be configured to control the vehicle 100 based on the output of the driving policy.

[0058] Operation of an autonomous vehicle traversing a roadway will now be described with reference to figure 2. Figure 2 schematically illustrates a driving scenario 5. Driving scenario 5 includes an autonomous vehicle 6 driving along a roadway 7. Autonomous vehicle 6 may correspond to the autonomous vehicle 1 of figure la, and / or the vehicle 100 of figure lb. As such, autonomous vehicle 6 comprises perception circuitry for sensing data related to an environment of the autonomous vehicle 6 and sensing data related to a state of the autonomous vehicle 6. In this example, autonomous vehicle 6 includes on-board camera, radar and LiDAR systems and on-board vehicle sensor systems for sensing the vehicle environment, like those described in relation to figures la and lb. The data captured by these systems may be processed by one or more on-board processors, such as control device 140 of figure lb.

[0059] For example, this captured data may be processed into an input format for the trained driving policy. It will be appreciated that this step will vary depending on the implementation and in some cases may not be performed. Data relating to the perceived environment and sensed vehicle state may then be input to a trained driving policy to determine how to control the autonomous vehicle 6, for example the trained driving policy 3 of figure la.

[0060] As shown in figure 2, also present in the driving scenario 5 and in the opposite and oncoming lane of the roadway 7, is another vehicle 8. Vehicle 8 may be an SDV or a human-operated vehicle (or indeed any other road user such as a pedestrian, cyclist, motorcyclist etc.). Autonomous vehicle 6 may perceive 9 vehicle 8 using its on-board camera, radar and / or LiDAR systems. An on-board computing or processing circuitry processes the data captured by the camera, radar, and / or LiDAR systems to perform analysis of the data so as to determine the presence of the vehicle 8. Various image, video, and data analysis techniques may be used that could be envisaged by the skilled person, such as machine learning techniques to identify objects in the data.

[0061] On-board processing circuitry of the autonomous vehicle 6 may also determine properties associated with the vehicle 8 and successive inputs from data recorded by the camera, radar, and / or LiDAR systems, for example to determine a speed, a changing position, and / or an acceleration of the vehicle 8.

[0062] Further, the driving scenario 5 comprises a building 9 and tree 10. It will be appreciated that driving scenarios may comprise an enormous range of objects such as structures, foliage, and street furniture. The perception circuitry of the SDV 6 perceives these objects of the scenario and the on-board processing circuitry may identify and classify the various objects.

[0063] In order for autonomous vehicle 6 to navigate the roadway 7 safely, autonomous vehicle 6 utilises a trained driving policy, also known as a motion planner, to make decisions as to how to control its movement and driving behaviour, for example a trajectory 11 to follow in a subsequent time period. In the example of figure 2, the output from the driving policy may be data indicative of a trajectory 11 that causes the autonomous vehicle 6 to remain at a constant speed and in the centre of its lane on the roadway as the other vehicle 8 passes by in the opposite direction.

[0064] For the driving policy or motion planner (e.g. a neural network policy) to be able to control the driving behaviourof the autonomous vehicle 6 across a range of driving scenarios in a safe and predictable manner, the driving policy is trained using one or more machine learning techniques.

[0065] A method for training a driving policy according to the present techniques will now be described with reference to figure 3. The method of figure 3 may be performed to train a driving policy, which may then be used by the autonomous vehicle of any of figures la, lb, or 2. The method of figure 3 may be performed by a computing device like that described with reference to figure 7.

[0066] At step 301, a plurality of trajectories output from one or more driving policies are obtained based on inputting an initial state of a driving scene to the one or more driving policies. In some examples, each trajectory may be output from a different driving policy and in other examples a single driving policy may output the plurality or a subset of the plurality of trajectories.

[0067] As described herein, a driving policy may be an artificial neural network trained to take as input an initial state of a driving scene (i.e. data relating to the environment of the autonomous vehicle and vehicle state) and output data relating to, or indicative of, a trajectory that the autonomous vehicle is to 10 follow for a subsequent time period. It will be appreciated that in some cases a trajectory may be the output of the driving policy and in other cases the output may be data from which a trajectory may be derived or determined.

[0068] Thus, an initial state of a driving scene may be input to one or more driving policies and a plurality of trajectories may then be output. In some examples, an initial state of a driving scene corresponds to data that describes the environment and / or vehicle state at the beginning of the driving scene. These trajectories, from one perspective, correspond to a prediction for how the autonomous vehicle should traverse the driving scene in the subsequent time period, based on the initial state of the driving scene. It will therefore be appreciated that in some cases a plurality of trajectories may be output from a single driving policy, each trajectory being different to other trajectories. Indeed, in some examples a multimodal driving planner is used that predicts multiple different trajectories from a single input driving scene initial state.

[0069] The performance of each of these trajectories may then be evaluated using closed-loop simulation. In particular, while only the initial state of the driving scene was input to the one or more driving policies to obtain the plurality of trajectories, the driving scene may comprise driving scene data for a time period, such as a predetermined number of seconds (for example 5 seconds). Thus, by stepping through the driving scene in a simulation in time steps, while a simulated agent traverses each of the trajectories, the performance of each of the trajectories may be determined in a closed-loop evaluation.

[0070] The closed-loop simulation may use software configured to replay a recorded driving scene and alter actions performed by actors present in the driving scene using the output (i.e. trajectories) of a driving policy. As discussed above, a driving scene may include data relating to the environment and vehicle state for a time period, and so the simulation may use this recorded data to recreate the driving scene in software. In particular, in the simulation, the recording vehicle in the driving scene (for example autonomous vehicle 6 of figure 2) may be replaced with a simulated vehicle controllable using the trained driving policy. In other words, the simulated vehicle drives in the reconstructed driving scene according to the outcomes of the trained driving policy (i.e. the trajectory), and replaces the actions of the recording vehicle. A comparison between the trajectory of the simulated vehicle may therefore be made against the trajectory of the recording vehicle (which represents the ground truth for driving behaviour), or one or more other performance metrics may be determined based on the simulated driving behaviour of the simulated agent following a given trajectory.

[0071] At step 302, driving behaviour in the driving scene is simulated based on each of the plurality of trajectories. This may comprise using a software simulation to simulate driving behaviour of a simulated agent to follow each of the obtained trajectories throughout the duration of the driving scene as described above. As mentioned, a driving scene may comprise data indicative of a driving scene (the environment, a vehicle state of the vehicle) across a period of time. Indeed, the driving scene data may be driving scene data obtained by a vehicle equipped with sensors as it traverses a roadway in the real-world. This recording vehicle may be an autonomous vehicle or may be a manned recording vehicle. Thus, with the simulation, the driving scene may be recreated in software and the recording vehicle (i.e. the autonomous vehicle that recorded the driving scene) may be replaced with a simulated agent, and the driving behaviour of the simulated agent may be controlled to follow each of the obtained trajectories.

[0072] At step 303, one or more performance metrics associated with each of the plurality of trajectories may be determined based on the simulated driving behaviour. For example, an overall performance metric for the performance of each of the trajectories may be determined. In some cases, the performance of each trajectory may be evaluated at the end of the driving scene. However, in other cases, the performance of each trajectory may advantageously be determined at multiple time intervals during the recreation of the driving scene.

[0073] The one or more performance metrics may comprise a variety of performance metrics associated with real-world driving, for example, acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and / or a difference between the simulated driving behaviour and stored reference driving data. The stored reference data may correspond to the trajectory that the vehicle actually took when recording the driving scene and may be considered to represent a 'ground truth' trajectory for that particular driving scene.

[0074] Once performance metrics for each of the plurality of trajectories have been determined using the closed-loop simulation, at step 304, scores may be awarded to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories.

[0075] In some examples, the scores may be awarded based on ranking the trajectories using the performance metrics. For example, the trajectories may be ranked based on how close the simulated driving behaviour was to the stored reference driving data, or based on positional differences between the trajectory and the stored reference trajectory, or how each trajectory performed for a given performance metric (such as a maximum acceleration or maximum braking value reached during simulation of the trajectory). The trajectories that are determined to be closer to the reference trajectory may be awarded a higher score than trajectories determined to be further from the reference trajectory. Other performance metrics may be used in the ranking, for example to provide a weighting factor. Thus, while the trajectories may be ranked based on a first performance metric (i.e. how close to a reference trajectory each trajectory is throughout the driving scene), factors like substantial acceleration or braking may also be considered.

[0076] Thus, in some examples, the scores may be awarded based on determining whether one or more predetermined metric thresholds are violated during simulation of the driving behaviour. For example, a predetermined maximum acceleration or braking threshold may be configured, for example based on human comfort levels or predetermined safety measurements.

[0077] At step 305, a reward model is trained (first training) to output a score associated with a trajectory using the scores and the plurality of trajectories. Thus, from one perspective, the reward model is trained to imitate the behaviour of the closed-loop simulation and awarding of scores without actually performing the closed-loop simulation. The reward model may be trained using regression or classification training. In some examples, the reward model may be trained using a stochastic gradient descent technique. Steps 301, 302, 303, 304, and 305 may be repeated until a predetermined training condition is satisfied. For example, a predetermined number of training iterations, or until performance of the reward model (based on a comparison of scores output from the reward model with scores output from the closed-loop simulation) changes less than a predetermined amount between repetitions of these steps.

[0078] As a result, a reward model is determined that is able to score unseen trajectories with scores. Example types of reward models can be any neural network that can score or rate the trajectories output from the driving policy. Indeed, in some examples, the reward model is an artificial neural network trained to score / rank trajectories based on input trajectories and driving scene data. An example reward model is the reward model detailed in 'Training language models to follow instructions with human feedback', OpenAI, Ouyang et. al, available at https: / / arxiv.org / pdf / 2203.02155.pdf.

[0079] Once trained, the reward model may then be used to train a driving policy. At step 306, a driving policy is trained (second training) using reinforcement learning and the trained reward model and based on a plurality of driving scenes. The reinforcement learning may use a proximal policy optimisation, PPO, technique.

[0080] The second training of step 306 may comprise selecting the highest scoring trajectory using scores awarded by the reward model. In some examples, the reward model gives a reward to each trajectory (i.e. the score / rank), and the goal of the reinforcement learning algorithm (which may be PPO but may be a different optimisation) is to maximise these rewards. In the example of PPO, the trained driving policy weights are updated in a way that maximizes the output reward from the reward model.

[0081] It will be appreciated that the driving policy trained in the second training step 306 may be multimodal in some examples and may not be multimodal in other examples. Indeed, the reward model may be used to train a driving policy with any reinforcement learning technique that uses a reward model.

[0082] As discussed herein, the second training may use the one or more driving policies (that output the plurality of trajectories). In this case, these one or more driving policies may be further enhanced or 'finetuned' using the output of the trained reward model in the reinforcement learning of the second training. In other cases, the second training does not use the driving policies as input. In this case, a new or different driving policy may be trained in the second training.

[0083] An example representation of a driving scene simulation will now be described with reference to figure 4. Figure 4 shows three successive snapshots of a driving scene 12 at times tl, t2, and t3 during a simulation. Driving scene 12 shows a vehicle 13. In this example, vehicle 13 represents the simulated agent in the simulation that is traversing a trajectory!along a roadway 14. As shown, each time snapshot also includes other vehicles 15.

[0084] At each time, tl, t2, and t3, one or more performance metrics associated with the trajectory are determined. For example, as described above, a deviation from a reference trajectory (a ground truth trajectory) may be determined and evaluated at each time step. While figure 4 shows a simulated agent 13 traversing trajectory T, it will be appreciated that a similar simulation is performed for each of the trajectories in the plurality of trajectories.

[0085] An example of simulation software that maybe used for the simulation described herein is CARLA, an open-source simulator available at carla.org. An example of a driving scene training dataset is the Woven by Toyota Prediction Dataset, titled One Thousand and One Hours: Self-driving Motion Prediction Dataset, authored by Houston, J. and Zuidhof, G. and Bergamini, L. and Ye, Y. and Jain, A. and Omari, S. and Iglovikov, V. and Ondruska, P., published in 2020, and available at https: / / woven.toyota / en / prediction-dataset, the contents of which is hereby incorporated by reference. A further example of a driving scene dataset is "nuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles.", Holger, Caesar, et al., arXiv preprint arXiv:2106.11810 (2021), and available at https: / / www.nuscenes.org / nuplan.

[0086] A driving policy method will now be described with reference to figure 5. Steps 1, 2, and 3 may be performed by a computing device such as that described in figure 7. Step 4 may be performed by a controller of an autonomous vehicle (such as controller 140 of figure lb). Step 4 may be considered an inference step.

[0087] Optionally, the method of figure 5 may comprise step 1. At optional step 1, a multi-modal driving policy may be trained using a plurality of driving logs (driving scenes). These driving logs / scenes may be stored in cloud storage.

[0088] At step 2, a plurality of trajectories output from the trained multi-modal driving policy are used in the closed-loop evaluation to score the trajectories. For example, the trajectories may be simulated and scored / ranked as described herein.

[0089] Optionally, a 'human in the loop' step may be performed which comprises manual human scoring / ranking of the trajectories that have been generated by the closed-loop simulation so as to verify the scores of the trajectories that have been generated by the closed-loop simulation.

[0090] A reward model is then trained to select the best scoring trajectory from the plurality of trajectories as described herein.

[0091] At step 3, the reward model is used in reinforcement learning to re-train the driving policy (i.e. the same starting driving policy from step 1 before training was performed), to fine-tune the driving policy (i.e. to train the driving policy of step 1 after training was performed), or to train a new training policy (i.e. not the training policy of step 1).

[0092] The trained driving policy may then be deployed on an autonomous vehicle or self-driving vehicle.

[0093] At step 4, the autonomous vehicle may be controlled based on the trained driving policy, and may take as input perception data and vehicle data. As the autonomous vehicle is driven, the autonomous vehicle may collect driving logs / scenes which are sent to the cloud storage and stored as part of a driving scene / log repository.

[0094] An example process for training the reward model will now be described with reference to figure 6a. The process of figure 6a may be performed by the computing device of figure 7, and the process of figure 6a may correspond to the first training, for example the first training of step 305 of figure 3 or step 2 of figure 5. The process of figure 6a relates to supervised training of the reward model using the outputs of the closed-loop evaluation.

[0095] At step 1, the driving scene and predicted trajectories are input to the reward model and the closed-loop evaluation ranks / scores are used as the labelled output for training the reward model to accurately predict the score / rank of unseen trajectories (i.e. to mimic the closed-loop evaluation without having to perform the closed-loop evaluation).

[0096] At step 2, the reward model is optimised (for example using stochastic gradient descent, SGD) to predict the correct rank / scores for the trajectories given in the driving scene. Thus, the reward model learns to rank or score the output trajectories in a way that best matches the closed-loop evaluation. In this way, instead of defining an explicit reward function, a learned reward model can be used to rank / score the trajectories.

[0097] An example process for training a driving policy using the trained reward model (such as a reward model trained using the process of figure 6a) will now be described with reference to figure 6b. The process of figure 6b may be performed by the computing device of figure 7, and may correspond to the second training, for example the second training of step 306 of figure 3 or the training of step 3 of figure 5.

[0098] At step 1, driving logs (i.e. a driving scene) are input to the pre-trained model (i.e. a trained driving policy configured to output / predict multiple trajectories based on input driving scene data).

[0099] At step 2, the driving policy predicts / outputs a plurality of trajectories. At step 3, these predicted trajectories and driving scene data are input to the trained reward model (such as a reward model trained using the process of figure 6a). At step 4, the reward model ranks or scores the driving policy's (also known as a driving / motion planner) predicted trajectories. At step 5, depending on the ranking / scoring from the reward model, the driving policy (shown as 'pre-trained model') uses reinforcement learning techniques to update itself (such as proximal policy optimisation, PPO). In some examples, the reward model gives a reward to each trajectory (i.e. the score / rank), and the goal of the reinforcement learning algorithm (which may be PPO but may be a different optimisation) is to maximise these rewards. In the example of PPO, the trained driving policy weights are updated in a way that maximizes the output reward from the reward model.

[00100] An example computing apparatus will now be described with reference to figure 7. Figure 7 schematically illustrates an example of a computing device 700 which can be used to implement teachings described above, for example the method of figures 3, and 5, and the simulation of figure 6. In some examples, the computing device 700 may correspond to a computing device provided at the autonomous vehicle or robot (or SDV) described herein, which may be loaded with the trained driving policy as described herein. In addition, computing device 700 can, in some examples, correspond to control device 140 of figure lb.

[00101] The computing device 700 has processing circuitry 710 for performing data processing in response to program instructions and data storage 720 for storing data and instructions to be processed by the processing circuitry 710. It will be appreciated that in some examples (such as those corresponding to method 400) computing device 700 may be a distributed computing device, for example using a cloud computing infrastructure comprising high-powered compute nodes. It will be appreciated that in some examples (such as those corresponding to method 400) the data storage 720 may also be provided using a cloud storage server. In some examples, the processing circuitry 710 includes one or more caches for caching recent data or instructions. It will be appreciated that Figure 7 is merely an example of possible hardware that may be provided in the computing device and other components may also be provided. For example, the device may include a dedicated sensor interface 730 for communicating with sensors. As another example, for some devices for which user interaction is expected, the device may be provided with one or more user input / output device(s) 740 to receive input from a user or to output information to a user. The computing device 700 may additionally or alternatively have a communications interface 750 for communicating with external devices. For example, communications interface 750 could use any of a range of different communication protocols, such as Ethernet, WiFi®, Bluetooth®, ZigBee®, etc.

[00102] The methods discussed above may be performed under control of a computer program executing on a computing device. Hence a computer program may comprise instructions for controlling a computing device to perform any of the methods discussed above. The program can be stored on a computer-readable medium. A computer readable medium may include non-transitory type media such as physical storage media including storage discs and solid state devices. A computer readable medium may additionally or alternatively include transient media such as carrier signals and transmission media. A computer-readable storage medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[00103] In the present application, the words "configured to..." are used to mean that an element of an apparatus has a configuration able to carry out the defined operation. In this context, a "configuration" means an arrangement or manner of interconnection of hardware or software. For example, the apparatus may have dedicated hardware which provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not imply that the apparatus element needs to be changed in any way in order to provide the defined operation.

[00104] The various embodiments described herein are presented only to assist in understanding and teaching the claimed features. These embodiments are provided as a representative sample of embodiments only, and are not exhaustive and / or exclusive. It is to be understood that advantages, embodiments, examples, functions, features, structures, and / or other aspects described herein are not to be considered limitations on the disclosure scope defined by the claims or limitations on equivalents to the claims, and that other embodiments may be utilised and modifications may be made without departing from the scope of the invention as defined by the claims.

Claims

1. A computer-implemented method for training a driving policy comprising:obtaining a plurality of trajectories output from one or more driving policies based on inputting an initial state of a driving scene to the one or more driving policies;simulating driving behaviour in the driving scene based on each of the plurality of trajectories;determining, based on the simulated driving behaviour, one or more performance metrics associated with each of the plurality of trajectories;awarding scores to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories;first training, using the scores and the plurality of trajectories, a reward model to output a score associated with a trajectory; andsecond training, using reinforcement learning and the trained reward model and based on a plurality of driving scenes, a driving policy.

2. The computer-implemented method of claim 1, wherein the second training uses output of the trained reward model in the reinforcement learning to train the driving policy.

3. The computer-implemented method of claim 1 or claim 2, wherein the second training uses the one or more driving policies as an input.

4. The computer-implemented method of claim 1 or claim 2, wherein the second training does not use the one or more driving policies as an input.

5. The computer-implemented method of claim 1 or claim 2, wherein the second training uses a further trained driving policy as an input.

6. The computer-implemented method of any preceding claim, wherein simulating driving behaviour comprises simulating driving behaviour in the driving scene based on each of the plurality of trajectories for a plurality of successive time intervals, and wherein determining the one or more performance metrics comprises determining one or more performance metrics associated with each of the plurality of trajectories during each of the plurality of successive time intervals.

7. The computer-implemented method of any preceding claim, wherein the first training uses regression or classification training.

8. The computer-implemented method of any preceding claim, wherein the one or more performance metrics comprise metrics associated with: acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and / or a difference between the simulated driving behaviour and stored reference driving data.

9. The computer-implemented method of any preceding claim, wherein simulating driving behaviour comprises: controlling driving behaviour of a simulated agent in the driving scene to follow each of the plurality of trajectories.

10. The computer-implemented method of claim 9, wherein simulating driving behaviour comprises altering behaviour of actors in the driving scene based on the simulated driving behaviour of the simulated agent.

11. The computer-implemented method of any preceding claim, wherein awarding scores to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories comprises:ranking each of the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories; andawarding scores to the plurality of trajectories based on the ranking.

12. The computer-implemented method of claim 11, wherein the ranking is based on comparing the one or more performance metrics associated with each of the plurality of trajectories to performance metrics associated with others of the plurality of trajectories.

13. The computer-implemented method of any preceding claim wherein the plurality of driving scenes comprises real-world driving scenes.

14. The computer-implemented method of any preceding claim, wherein the one or more trained driving policies and / or driving policy is an artificial neural network.

15. The computer-implemented method of any preceding claim, the method further comprising: initial training, based on driving scene data and using a machine learning technique, one or more initial driving policies to generate the one or more trained driving policies.

16. The computer-implemented method of claim 15, wherein the machine learning technique is imitation learning.

17. The computer-implemented method of any preceding claim, in which the obtaining, simulating, determining, awarding, and first training steps are repeated for a plurality of different driving scenes.

18. The computer-implemented method of any preceding claim, further comprising using the driving policy output from the second training in a motion planner of an autonomous vehicle.

19. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method of claims 1 to 18.

20. An apparatus comprising a processor configured to carry out the method of claims 1 to 18.

21. A computer-readable medium comprising a driving policy trained according to the method of claims 1 to 18.

22. An apparatus comprising memory comprising a driving policy trained according to the method of claims 1 to 18.

23. The apparatus of claim 22, wherein the apparatus is a robot or an autonomous vehicle, for example an autonomous car or autonomous airborne vehicle.AMENDMENTS TO THE CLAIMS HAVE BEEN FILED AS FOLLOWS:-18 02 25CLAIMS1. A computer-implemented method for training a driving policy comprising:obtaining a plurality of trajectories output from one or more driving policies based on inputting an initial state of a driving scene to the one or more driving policies;simulating driving behaviour in the driving scene based on each of the plurality of trajectories;determining, based on the simulated driving behaviour, one or more performance metrics associated with each of the plurality of trajectories;awarding scores to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories;first training, using the scores and the plurality of trajectories, a reward model to output a score associated with a trajectory; andsecond training, using reinforcement learning and the trained reward model and based on a plurality of driving scenes, a driving policy, wherein the second training does not use the one or more driving policies as an input.

2. The computer-implemented method of claim 1, wherein the second training uses output of the trained reward model in the reinforcement learning to train the driving policy.

3. The computer-implemented method of any preceding claim, wherein simulating driving behaviour comprises simulating driving behaviour in the driving scene based on each of the plurality of trajectories for a plurality of successive time intervals, and wherein determining the one or more performance metrics comprises determining one or more performance metrics associated with each of the plurality of trajectories during each of the plurality of successive time intervals.

4. The computer-implemented method of any preceding claim, wherein the first training uses regression or classification training.

5. The computer-implemented method of any preceding claim, wherein the one or more performance metrics comprise metrics associated with: acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and / or a difference between the simulated driving behaviour and stored reference driving data.

6. The computer-implemented method of any preceding claim, wherein simulating driving behaviour comprises: controlling driving behaviour of a simulated agent in the driving scene to follow each of the plurality of trajectories.18 02 257. The computer-implemented method of claim 6, wherein simulating driving behaviour comprisesaltering behaviour of actors in the driving scene based on the simulated driving behaviour of the simulated agent.

8. The computer-implemented method of any preceding claim, wherein awarding scores to the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories comprises:ranking each of the plurality of trajectories based on the one or more performance metrics associated with each of the plurality of trajectories; andawarding scores to the plurality of trajectories based on the ranking.

9. The computer-implemented method of claim 8, wherein the ranking is based on comparing the one or more performance metrics associated with each of the plurality of trajectories to performance metrics associated with others of the plurality of trajectories.

10. The computer-implemented method of any preceding claim wherein the plurality of driving scenes comprises real-world driving scenes.

11. The computer-implemented method of any preceding claim, wherein the one or more trained driving policies and / or driving policy is an artificial neural network.

12. The computer-implemented method of any preceding claim, the method further comprising: initial training, based on driving scene data and using a machine learning technique, one or more initial driving policies to generate the one or more trained driving policies.

13. The computer-implemented method of claim 12, wherein the machine learning technique is imitation learning.

14. The computer-implemented method of any preceding claim, in which the obtaining, simulating, determining, awarding, and first training steps are repeated for a plurality of different driving scenes.

15. The computer-implemented method of any preceding claim, further comprising using the driving policy output from the second training in a motion planner of an autonomous vehicle.

16. A computer-readable medium comprising instructions which, when executed by a processor, cause the processor to carry out the method of claims 1 to 15.

17. An apparatus comprising a processor configured to carry out the method of claims 1 to 15.

18. A computer-readable medium comprising a driving policy trained according to the method of claims 1 to 15.

19. An apparatus comprising memory comprising a driving policy trained according to the method of claims 1 to 15.

20. The apparatus of claim 19, wherein the apparatus is a robot or an autonomous vehicle, for example an autonomous car or autonomous airborne vehicle.18 02 25

Images