Driving policy fine-tuning

The method addresses inefficiencies in training and fine-tuning driving policies by generating and evaluating trajectories for autonomous vehicles, using rejection sampling and machine learning to enhance performance and efficiency.

GB2706059APending Publication Date: 2026-11-11TOYOTA JIDOSHA KK
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
GB2025005593
Authority / Receiving Office
GB · GB
Patent Type
Applications
Current Assignee / Owner
Filing Date
2025-04-14
Publication Date
2026-11-11

AI Technical Summary

Technical Problem

Existing approaches to training and fine-tuning driving policies for autonomous vehicles are inefficient, time-consuming, and resource-intensive, with challenges in selecting and improving the performance of trained driving policies.

Method used

A method for fine-tuning a trained driving policy by generating and evaluating trajectories based on performance criteria, using rejection sampling to select a subset of trajectories for training, and employing machine learning techniques like imitation learning to improve efficiency and performance.

Benefits of technology

The method allows for efficient and flexible fine-tuning of driving policies by reducing data processing and storage requirements, enhancing trajectory generation, and improving performance metrics such as safety and compliance with driving rules.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A method of fine tuning a driving policy comprising, generating a set of trajectories for each of one or more driving scene states, inputting a respective scene state to the trained policy to obtain a
Need to check novelty before this filing date? Find Prior Art

Description

Field and Background The present techniques relate to the field of driving policies, such as for autonomous driving. More 5 particularly, but not exclusively, the present techniques relate to techniques for training a driving policy and fine-tuning a trained driving policy. In recent times, the use of autonomous vehicles (also known as self-driving vehicles, SDVs) has become more widespread. Autonomous vehicles are vehicles capable of sensing their own environment and operating without human involvement. The degree of autonomy may vary from 10 fully autonomous requiring no human involvement (sometimes known as Level 5 - Full Driving Automation), to lower levels of autonomy requiring some degree of human involvement (Levels 1 to 4). As used herein, an autonomous vehicle refers to a vehicle or robot having at least some capability of operating without human involvement. Existing approaches to driving policy training and fine-tuning of trained driving policies are not 15 suitable to the complexities and considerations of current demands. Summary Aspects of the invention are set out in the accompanying claims. The present inventors have identified that training a driving policy (also known as a motion planner) can be inefficient, time-consuming, and computing resource-intensive. In particular, the present inventors have identified that it can be difficult to improve the performance of a driving policy once it has been trained, e.g. to fine-tune the trained driving policy. For example, selecting additional training data can be inefficient, and identifying additional training data that may have a positive impact on the performance of the trained driving policy can be challenging. Viewed from a first aspect, there is provided a computer-implemented method for fine-tuning a trained driving policy, the method comprising: generating a set of trajectories by, for each of one or more driving scene states, inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories; evaluating the set of trajectories based on one or more performance criteria; selecting for training a subset of the set of trajectories based on evaluating the plurality of trajectories; and training the trained driving policy using a machine learning technique and the subset of the set of trajectories. In this way, training data (i.e. the subset of trajectories) can be efficiently generated and selected for use to fine-tune the performance of the trained driving policy. Further, trajectories selected for training are evaluated based on performance criteria, and so the trained driving policy can be fine-tuned in a configurable and flexible manner. As a result, the performance of the trained driving policy can be improved in a more computing resource and time efficient manner. In some examples, the machine learning technique comprises imitation learning and / or reinforcement learning. In some examples, the machine learning technique is imitation learning. The present inventor has identified that imitation learning can be particularly well-suited to use with the subset of trajectories. Conventional imitation learning using the subset of trajectories as ground truths for associated driving scene states may be used. In some examples, selecting for training the subset of the set of trajectories comprises rejecting from selection for training one or more of the set of trajectories other than the subset. Hence, trajectories may be rejected from selection based on evaluating the trajectories. As a result, trajectories can be controllably discarded from selection for training. Further, the total amount of data being processed and the associated storage requirements can be significantly reduced by performing rejection sampling of the trajectories. Indeed, this supports an approach where only the best performing trajectories (determined against predetermined criteria) are retained and other trajectories are discarded. In this way, the present technique performs rejection sampling. In some examples, the trained driving policy has been previously trained to output multiple trajectories for a given input driving scene state. In this way, the trained driving policy is a multimodal driving policy. Such driving polices are configured to output multiple possible trajectories for a single input driving scene state. This increases the efficiency of trajectory generation, and also results in a greater variety of trajectories. This is useful because it can be challenging to select a single trajectory for output, in particular when a driving scene is particularly complex. Indeed, the way the trained driving policy has been trained to select a single trajectory for output may not be suitable for a given driving scene, and so by outputting multiple possible trajectories, the likelihood that one of the multiple trajectories satisfies a given performance criterion can be increased. In some examples, evaluating the set of trajectories comprises ranking the set of trajectories based on the one or more performance criteria. The present inventor has identified that ranking supports an efficient subsequent selection of trajectories. In some examples, the subset of the set of trajectories corresponds to a predetermined number of highest ranked trajectories. Hence, the selected subset may be a top K number of highest ranked trajectories, where K is a predetermined number. This provides an efficient selection mechanism for selecting the trajectories to be used for the training. In some examples, generating the set of trajectories is based on simulating driving behaviour of a simulated agent (e.g. an ego vehicle) in a driving scene associated with the one or more driving scene states. By using a simulation, a large number of trajectories can be generated in an efficient manner. Indeed, this can reduce the amount of real-world data (i.e. data that has been collected by recording vehicles traversing real driving scenes) required to perform effective fine-tuning of the driving policy. In some examples, simulating driving behaviour comprises controlling driving behaviour of the simulated agent to follow one of the plurality of trajectories obtained from the trained driving policy. In this way, a trajectory that has been generated may actually be used for the simulation. This supports an efficient simulation of driving behaviour. In some examples, the one or more driving scene states are associated with a state of the driving scene at different time steps through a simulation of action in the driving scene. In this way, the one or more driving scene states may refer to input driving scene states that are input to the trained driving policy at different time steps of the simulation to generate the pluralities of trajectories for each time step. This increases efficiency of trajectory generation. In some examples, simulating driving behaviour comprises altering behaviour of one or more other agents based on the simulated driving behaviour of the simulated agent. Thus, behaviour of various agents (i.e. other vehicles in the driving scene) may be simulated, creating a more realistic simulation of real-world driving, where vehicles respond to the action of other vehicles. In some examples, generating the set of trajectories comprises, for a plurality of time steps each associated with a driving scene state, inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories for the respective time step. In this way, there is a many to one relationship between trajectories and time steps, such that a large number of trajectories can be generated and evaluated. This increases the likelihood of identifying trajectories that are beneficial for fine-tuning the trained driving policy. In some examples, generating the set of trajectories comprises, for a predetermined number of iterations, iteratively inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories for the respective iteration. In some examples, the evaluating and selecting steps may be performed for each iteration. In other examples, the evaluating and selecting steps may be performed once the predetermined number of iterations has been reached and the set of trajectories has been generated. In some examples, the method further comprises selecting one trajectory from the plurality of trajectories for the respective time step for simulation, and simulating driving behaviour of a simulated agent to follow the selected one trajectory to determine a resulting driving scene state for use as a driving scene state input to the trained driving policy for a next time step. Hence, not only are the trajectories used to create a subset for later training, but one of the trajectories at each time step is used for a next stage in the simulation. For example, the selected trajectory is then used as the trajectory for the simulated agent to follow in the simulation. After that trajectory has been followed, the resulting driving scene state forms the input driving scene state for the next time step and generating the next plurality of trajectories from the trained driving policy. In some examples, selecting one trajectory from the plurality of trajectories for the respective time step for simulation comprises selecting a trajectory based on determining performance of each of the plurality of trajectories for the respective time step. This increases the likelihood that trajectories having a suitable performance are generated by the simulation, as the best performing simulation can be selected for the simulation’s next step. This reduces the likelihood of the simulation resulting in an error state, and supports more efficient trajectory generation. In some examples, evaluating the set of trajectories comprises evaluating the plurality of trajectories at each time step, and selecting for training comprises rejecting from selection for training one or more of the plurality of trajectories at each time step. In this way, rather than generating the entire set of trajectories and then selecting the subset and rejecting trajectories other than the subset once the simulation has finished, trajectories can be rejected at each time step and so there is no requirement for these rejected trajectories to be processed further and stored. This can reduce a computing resource requirement for fine-tuning the trained driving policy. In some examples, the one or more performance criteria is associated with a safety level associated with a given trajectory. In this way, the safety performance of the trained driving policy can be increased. The one or more performance criteria may relate to one or more predetermined thresholds for braking, acceleration, jerk, or the observation of one or more driving rules. In some examples, training the trained driving policy comprises using the subset of the set of trajectories as ground truth trajectories for associated driving scene states. Hence, the trained driving policy can be further trained using the subset of trajectories as ground truth for the driving scene state, so that the trained driving policy is trained to imitate the generation of the ground truth trajectory for the respective driving scene state. In some examples, evaluating the set of trajectories comprises: determining one or more performance metrics associated with each of the plurality of trajectories based on simulating driving behaviour associated with each of the plurality of trajectories. The simulation used for the performance metrics may be the same simulation used to generate the set of trajectories. Hence, during simulation performance metrics may be measured for each trajectory, and the plurality of trajectories may then be evaluated based on the measured performance metrics. This supports an efficient evaluation of trajectory performance. In some examples, the one or more performance metrics comprise metrics associated with: acceleration, braking, driving off-route, vehicle proximity, collisions, and compliance with driving rules. In this way, a variety of different metrics are supported and the trained driving policy can be fine-tuned to improve performance relating to a variety of different metrics. This increases flexibility of the fine-tuning. In some examples, evaluating the set of trajectories comprises inputting the plurality of trajectories to a large language model (LLM) with a prompt requesting evaluation of the set of trajectories. The present inventor has identified that it can be challenging to select and define performance metrics for evaluating performance, and have further identified that an LLM is able to provide a more holistic view of performance of what a ‘good’ or safe trajectory would be. This can improve the selection of trajectories for training and thus the performance of the trained driving policy once trained further. In some examples, the prompt is to request the LLM to rank the set of trajectories based on the one or more driving objectives. The present inventors have identified that this is an efficient way to determine a ranking of the trajectories. In some examples, inputting the plurality of trajectories to the LLM comprises inputting, for each trajectory of the plurality of the trajectories, data indicative of positions and headings of one or more vehicles in a driving scene associated with the respective driving scene state. The present inventors have identified that LLMs can evaluate a trajectory based on data describing the positon and heading of vehicles in a driving scene. This can reduce the amount of information required to be processed by the LLM and can thus reduce response time, and therefore training time. In some examples, inputting the plurality of trajectories to a large language model (LLM) with a prompt requesting evaluation of the set of trajectories also comprises inputting the associated driving scene state. In some examples, the trained driving policy is an artificial neural network configured to output a trajectory to be followed by an autonomous vehicle in a subsequent time step based on an input driving scene state. It will be appreciated that the neural network may be an artificial neural network implemented as hardware or software, or indeed emulated in software and thus considered hardware. Examples of such artificial neural network driving policies that may be used with the techniques described herein include models such as SafetyNet, SafePathNet, and Wayformer. In some examples, the trained driving policy is for controlling an autonomous vehicle. In some examples, the method further comprises controlling an autonomous vehicle based on the trained driving policy after training the trained driving policy using imitation learning and the subset of the set of trajectories. In some examples, a different driving policy from the trained driving policy used to generate the set of trajectories is trained using imitation learning and the subset of the set of trajectories. In this way, other driving policies may be trained using the subset of trajectories that have previously been curated. Viewed from a second aspect, there is provided an apparatus comprising one or more processors configured to carry out the method described above. Viewed from a third aspect, there is provided a computer-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method described above. Viewed from a fourth aspect, there is provided a computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method described above. Other aspects will also become apparent upon review of the present disclosure, in particular upon review of the Brief Description of the Drawings, Detailed Description and Claims sections. Brief Description of the Drawings Examples of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which: Figure 1 illustrates an example autonomous vehicle; 5 Figure 2 illustrates an example driving scene; Figure 3 shows steps for training a driving policy; Figure 4 shows steps for training a driving policy; Figure 5 shows steps for training a driving policy; and Figure 6 illustrates an example computing device that may perform the present techniques. 10 While the disclosure is susceptible to various modifications and alternative forms, specific example approaches are shown by way of example in the drawings and are herein described in detail. It should be understood however that the drawings and detailed description attached hereto are not intended to limit the disclosure to the particular form disclosed but rather the disclosure is to cover all modifications, equivalents and alternatives falling within the spirit and scope of the 15 claimed invention. It will be recognised that the features of the above-described examples of the disclosure can conveniently and interchangeably be used in any suitable combination. Detailed Description Figure 1 shows an illustration of an autonomous vehicle 100 that may use a trained driving policy (also referred to as a motion planner) for autonomous driving control as described herein. Vehicle 100 is depicted as comprising a first sensor 110, a second sensor 120, further sensors 130 and a computing device 140. It will be understood that, in some examples, the sensors may be image sensors and that the second sensor 120 and further sensors 130 are optional and that a single sensor may be provided. It will be appreciated that the present techniques may be applied to any vehicle (unmanned, autonomous or otherwise) provided with one or more sensors as described herein. In some examples, one or more of the first sensor 110, the second sensor 120, and the further sensors 130 is a camera. The camera uses visible and / or invisible light. Thereby performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths. In some examples, one or more of the image sensors may be a LIDAR system, a RADAR system, a SONAR system and / or a LASER scanning system. Thereby, higher spatial accuracy may be achieved and performance may be enhanced in certain environmental conditions, for example, when rain or fog reduces visibility at certain electromagnetic wavelengths. In the example of figure 1, first sensor 110 is a camera sensor, second sensor 120 is a LiDAR sensor and further sensors 130 are RADAR sensors. Vehicle 100 may comprise further sensors (not shown) for sensing information relating to the state of the vehicle 100. For example, vehicle 100 may comprise one or more sensors for sensing information associated with / indicative of one or more of a speed, position, jerk, curvature, acceleration, braking, wheel-turning amount, engine power, fuel or battery level, etc. In the present example, computing device 140 is a general-purpose computer, for example of the form depicted in figure 8. In other examples, the computing device 140 may comprise a specialist computing component such as an ASIC or FPGA. Thereby power efficiency, speed and / or latency may be enhanced. Computing device 140 may comprise one or more processors for executing and evaluating a trained driving policy, and data storage for storing the trained driving policy and program instructions for evaluating the driving policy. It will be appreciated that computing device 140 may be loaded with a trained driving policy, and computing device 140 may be configured to control the vehicle 100 based on the output of the driving policy. Indeed, vehicle 100 may collect sensor data from the sensors 110, 120, 130 and / or one or more sensors for sensing the state of the vehicle 100 (such data being an example of data that may be included in driving scene state data) and input the sensor data to the computing device 140. Computing device 140 may then execute a driving policy / motion planner based on the sensor input, and control the autonomous vehicle based on the output of the executed driving policy. For example, the driving policy may be a trained machine learning driving policy (such as a trained artificial neural network) which takes as input sensor data (i.e. driving scene state data that includes data relating to the vehicle environment and vehicle state) and outputs data indicative of a trajectory for the autonomous vehicle to follow during a subsequent time period. The autonomous vehicle 100 may then be controlled to follow the determined trajectory, for example through control of one or more of an engine system or motor system of the vehicle, a turning system of the vehicle, and / or a braking system of the vehicle, etc. It will be understood that autonomous vehicle 100 may comprise additional components and computing circuitry not shown in figure 1. It will be appreciated that the term “autonomous vehicle” may refer to: a self-driving vehicle, such as a car, a van, a lorry or other vehicle; or an aerial vehicle, for example a manned aerial vehicle, an unmanned aerial vehicle or a drone; or a robot, for example an industrial robot or a domestic robot. Autonomous vehicle 100 may also be provided with a network communication apparatus (not shown) for communicating over one or more communication networks. Further, autonomous vehicle 100 may communicate with one or more other vehicles using one or more vehicle-to-vehicle communication protocols. Autonomous vehicle 100 may receive sensor data from other autonomous vehicles using the network communication apparatus, and may send sensor data to other autonomous vehicles in a similar way. As mentioned above, the driving policy may be a trained machine learning model, such as an artificial neural network, that is configured to output a trajectory for the autonomous vehicle to follow based on input indicative of a vehicle environment and / or state (i.e. a driving scene state). Such input may include information relating to the vehicle speed, position, jerk etc. of the driving vehicle and in some cases also other vehicles observed by the driving vehicle. The driving policy may be trained using one or more machine learning techniques, such as reinforcement learning, imitation learning, etc. Figure 2 shows a representation of an example driving scene 200 according to the present techniques. A driving scene is a representation of a driving scenario that has been sensed by a sensing vehicle during driving (e.g. sensed in real-time for real-time processing and control, or recorded and stored for later use as part of a corpus of driving scenes). The data underlying or describing the driving scene is referred to as the driving scene state, which may be input to a trained driving policy to generate trajectories. This data can include position data, acceleration data, etc. of vehicles in the driving scene, including the ego vehicle and other agents in the driving scene. A driving scene may be sensed by an autonomous vehicle during operation, or a non-autonomous vehicle configured to collect driving scene data for adding to a corpus of driving scene data. During driving, a vehicle equipped with sensing arrangements (such as the sensors described in relation to figure 1) senses the vehicle environment (and in some cases also the vehicle state, such as speed, position, etc.) and uses this data as the driving scene state. In some cases, this data may be stored for later use, or may be used immediately to determine a next trajectory for the vehicle to follow. This data corresponds to a driving scene, and can include movement and action of other road users and pedestrians, and data relating to the street layout, street furniture, weather, etc. For example, driving scenes may correspond to data recorded by one or more camera, radar, and / or LiDAR systems over time captured by a recording vehicle (for example a vehicle equipped with environment perceiving circuitry) and also data relating to the vehicle state of the recording vehicle over time for the duration of the recorded driving scene. In some examples, the driving scene state data comprises data identifying one or more road users and their trajectories. In this way, driving scene state data may correspond to a snapshot in time of a driving scenario, thus including one or more road users, their relative positions, and a road layout. Driving scene state data may also be represented by coordinates (i.e. numbers) in a driving scene space, and may also include associated timestamps. In some examples, driving scene state data comprises sensor data (recorded by a vehicle), vehicle GPS data (of the vehicle), and vehicle controlling signals (of the vehicle), each associated with timestamps indicative of when the data was generated. In this way, when driving scene state data has been stored for later use, the driving scene state data may be used to reconstruct the driving scene in software that the vehicle (that sensed the driving scene data) experienced. For example, using the data captured by a recording vehicle, the driving scene, i.e. how the recording vehicle traversed the driving scene, including information relating to its driving behaviour, and how other actors (like vehicles, pedestrians, other road users etc.) behaved, can be reconstructed in software. For example, image processing techniques may be used on video data of the driving scene to identify other road users and their behaviour (such as their movement, speed, position, trajectories, etc.). Once recorded, the driving scenes may be added to a driving scene dataset for later use. Indeed, driving scenes can then be used for various purposes, such as machine learning training of driving policies. For example, driving scene state data for a variety of driving scenarios may be collected and may form a corpus of training data that may be used for training a machine learning driving policy. An example of a driving scene state training dataset is the Woven by Toyota Prediction Dataset, titled One Thousand and One Hours: Self-driving Motion Prediction Dataset, authored by Houston, J. and Zuidhof, G. and Bergamini, L. and Ye, Y. and Jain, A. and Omari, S. and Iglovikov, V. and Ondruska, P., published in 2020, and available at https: / / woven.toyota / en / prediction-dataset, the contents of which is hereby incorporated by reference. A further example of a driving scene dataset is "nuPlan: A closed-loop ml-based planning benchmark for autonomous vehicles.", Holger, Caesar, et al., arXiv preprint arXiv:2106.11810 (2021), and available at https: / / www.nuscenes.org / nuplan. As described herein, the simulation may be used to generate trajectories by stepping through a simulation of one or more driving scenes, and simulating an ego vehicle to follow a generated trajectory at each time step. The resulting driving scene state following traversal of the trajectory may then be used as input for the next iteration of trajectory determination. In this way, a large number of trajectories can be efficiently generated. The simulation can also be used to determine performance metrics for trajectories. For example, each trajectory may be simulated in software and performance metrics may be collected during simulation. These performance metrics may comprise a variety of performance metrics associated with real-world driving, for example, acceleration, braking, driving off-route, proximity, collisions, compliance with driving rules, and / or a difference between the simulated driving behaviour and stored reference driving data. The stored reference data may correspond to the trajectory that the vehicle actually took when recording the driving scene and may be considered to represent a ‘ground truth’ trajectory for that particular driving scene. An example of simulation software that may be used for the simulation described herein is CARLA, an open-source simulator available at carla.org. The closed-loop simulation may use software configured to replay a recorded driving scene and alter actions performed by agents present in the driving scene using the output (i.e. trajectories) of a driving policy. As discussed above, a driving scene may include data relating to the environment and vehicle state for a time period, and so the simulation may use this recorded data to recreate the driving scene in software. Turning back to figure 2, this shows a representation of a driving scene 200 depicting a driving scenario involving a roadway 210 having a three-way intersection (also known as a three-way junction). Driving scene 200 includes data relating to the movement of recording vehicle 220 during driving along the roadway 210 (as shown by the arrow), and other road actors 230. It will be appreciated that other agents in the scene may include pedestrians, cyclists, and / or other road users. It will be appreciated that driving scene 200 shows only one example of a driving scene, and that a large variety and number of driving scenes may be sensed by sensing vehicles and stored, and 12 in some cases used to train one or more driving policies. Each driving scene may depict a driving scenario, for example, motorway scenes, intersections, junctions, parking lots, traffic light scenes, road sign scenes, multi-lane roadways, single lane roadways etc. Thus, a corpus of driving scenes will comprise a great number of driving scenarios and examples of the same driving scenarios. Driving scene 200 may be an example of training data that is used to train one or more driving policies. Figure 3 shows a method 300 including steps fortraining a driving policy. Step 301 includes generating a set of trajectories by, for each of one or more driving scene states, inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories. The trained driving policy may have been previously trained to output multiple trajectories for a given input driving scene state. In this way, a multi-modal driving policy may be used. This can increase the likelihood that a trajectory is generated that satisfies one or more performance criteria. As discussed further below, step 301 can include simulating driving behaviour of a simulated agent in a driving scene associated with the one or more driving scene states to generate the set of trajectories. For example, step 301 can include for a plurality of time steps each associated with a driving scene state, inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories for the respective time step. In this way, driving behaviour is simulated for a number of time steps, where for each time step multiple trajectories are output and collated over the course of the simulation. Step 302 includes evaluating the set of trajectories based on one or more performance criteria. This can include ranking the set of trajectories based on the one or more performance criteria. This supports an efficient evaluation and selection. The simulation of driving behaviour can be used again for step 302 in some examples. For example, step 302 can include determining one or more performance metrics associated with each of the plurality of trajectories based on simulating driving behaviour associated with each of the plurality of trajectories. Indeed, the simulation may also be used to collect performance metrics associated with each of the trajectories. For example, at each time step, all of the generated trajectories may be simulated, e.g. the agent is simulated to follow each of the trajectories for that time step of a driving scene and performance metrics for each trajectory are collected. This can include metrics associated with: acceleration, braking, driving off-route, vehicle proximity, collisions, and compliance with driving rules. This means that each trajectory can be evaluated accurately to assist with the selection. Additionally or alternatively, evaluating the set of trajectories comprises inputting the plurality of trajectories to a large language model (LLM) with a prompt requesting evaluation of the set of trajectories. This may also include inputting the associated driving scene state and / or information relating to other actors in the driving scene. Indeed, in some cases, rather than using a simulation, an LLM may be used. The present inventors have identified that LLMs may be capable of accurately evaluating possible trajectories. For example, a trajectory may be input in a vectorised form to the LLM. Such a vectorised form may include may include position information, heading information, yaw information, acceleration information. An example would be [x, y, yaw], or [Xego, y positionego, yawego, xagent, yagent, yawagent]. LLMs are a type of machine learning model (such as a neural network transformer model) that are trained (for example using deep learning and / or self-supervised learning) on large datasets to perform natural language processing tasks such as text generation. Conventional LLMs may be used with the present techniques, including OpenAI’s GPT series such as ChatGPT, Google’s Bard or PaLM2. The present technique is not particularly limited in this respect. The LLM may be hosted remotely and may be communicated with over a network connection. The LLM may also be prompted to rank the trajectories based on one or more driving objectives. An example prompt may be: “rank the following trajectories based on minimising jerk”. Step 303 includes selecting for training a subset of the set of trajectories based on evaluating the plurality of trajectories. This can include rejecting from selection for training one or more of the set of trajectories other than the subset. In this way, the present technique may be seen as a form of rejection sampling. It will be appreciated that steps 302 and 303 may be performed at each iteration of the inputting of driving scene states to the trained driving policy for a plurality of driving scene states. Such as, at each time step of the simulation. Alternatively, steps 302 and 303 may be performed only once the set of trajectories have been generated by iteratively inputting driving scene states to the trained driving policy. Step 304 includes training the trained driving policy using a machine learning technique and the subset of the set of trajectories. This can include using the subset of trajectories as ground truth trajectories for associated driving scenes. For example, a trajectory in the subset has been selected based on the evaluation, and so the trajectory is then treated as the ‘correct’ trajectory for its associated driving scene. The driving policy can then be fine-tuned using imitation learning with the driving scene states labelled with the ‘correct’ or ground truth trajectory as training data. 14 The driving policy is then trained to imitate generation of the ‘correct’ trajectory. It will be appreciated that conventional machine learning techniques may be used to further train the driving policy, such as conventional imitation learning techniques, but using the curated subset of trajectories. In this way, performance of the driving policy can be increased. Figure 4 shows a method 400 for training a driving policy. Steps 401 to 407 may be performed iteratively, for a predetermined number of iterations. This may correspond to a number of time steps of a driving simulation. Method 400 starts at step 401. Step 401 includes inputting a driving scene state to a trained driving policy to generate a plurality of possible trajectories to be followed during a next time step. In this case, the driving policy is a multi-modal driving policy. At step 402, the plurality of trajectories is evaluated and / or ranked. As described above, this may be done by simulating an agent to follow each of the trajectories using software, and determining performance metrics during the simulation. This may instead be done by inputting each of the trajectories as well as the driving scene state to an LLM with a request for the LLM to evaluate and / or rank the trajectories according to one or more performance criteria. Such a one or more performance criteria may be associated with a safety of the trajectory, or pre-determined human comfort levels for acceleration, jerk, etc. At step 403, the top K trajectories are added to a training subset for later use to train a driving policy. The other trajectories are rejected (e.g. discarded) from being used for training. In order to determine the next driving scene state to be used for the generation of trajectories in a next iteration, a trajectory is selected from the plurality of trajectories for use in the simulation at step 404. This may be the best performing trajectory, i.e. the highest ranked trajectory, or it may be determined in another way, for example randomly. The selected trajectory from step 404 is then used to simulate driving behaviour during a next time step in step 405. In particular, an agent is simulated in software to follow the selected trajectory. This can include simulating driving behaviour of other agents in the driving scene that may take action in response to action of the agent following the selected trajectory. In this way, a second driving scene state can be generated. In particular, at step 406, the driving scene state resulting from the simulation of step 405 is determined. In other words, the driving scene state once the agent has been simulated to follow the selected trajectory is determined. This can include updated positions, speeds, etc. of vehicles in the driving scene since before the agent followed the selected trajectory. Indeed, it will be 15 appreciated that while the agent is being simulated to follow a selected trajectory, other agents in the same may be simulated to take action in response. In this way, over time, the driving scene is simulated to evolve and change, and can be used to iteratively generate a large number of driving scene states and thus trajectories. At step 407 it is determined whether to continue for another iteration. This can be based on whether a stop condition has been reached, such as a predetermined number of iterations. If the process is to continue for another iteration, the method returns to step 401. If the process is not to continue for another iteration, the method continues to step 408. At step 408, a driving policy (such as the driving policy used to generate the trajectories in step 401) is trained using imitation learning and the subset of trajectories collected during the iterative process. As described herein, this can fine-tune the performance of the driving policy using trajectories that are considered to perform well in a given driving scene. As a result, autonomous driving performance can be improved. Figure 5 shows an example method 500 for fine-tuning a driving policy (also referred to as a policy model). Method 500 may optionally include step 501 where a policy model is pre-trained. At step 502, a closed-loop simulation is performed using the policy model to generate a set of trajectories, as described herein. At step 503, metrics associated with the generated set of trajectories are computed. This can include determining the metrics using the closed-loop simulation, or inputting the trajectories to an LLM as discussed above. At step 504, rejection sampling is performed. This can include keeping only the top k trajectories based on the metrics computed at step 503. This can form the subset of trajectories to be used for training. At step 505, the subset of trajectories is used to fine-tune the policy model. This is performed using imitation learning and the subset of trajectories as training data, as described herein. For example, conventional imitation machine learning techniques may be used. Figure 6 illustrates an example of an electronic device 600 which can be used to implement teachings described above, including the methods of figures 3 to 5. The electronic device 600 has processing circuitry 610 for performing data processing in response to program instructions and data storage 620 for storing data and instructions to be processed by the processing circuitry 610. In some examples, the processing circuitry 610 includes one or more 16 caches for caching recent data or instructions. The data storage 620 may have a database 630 which can, for example, store trajectories and / or driving scene states. The device further includes a communication interface 640 which can be used, for example, to obtain / receive driving scenes, and communicate with an LLM. It will be appreciated that Figure 6 is merely an example of possible hardware that may be provided in the device and other components may also be provided. The device 600 may additionally or alternatively be provided with one or more user input / output device(s) 650 to receive input from a user device (e.g. to receive the query) or to output information (e.g. the selected one or more similar driving scenes). The methods discussed above may be performed under control of a computer program executing on a device. Hence a computer program may comprise instructions for controlling a device to perform any of the methods discussed above. The program can be encoded in a computer-readable medium. A computer-readable medium may include non-transitory type media such as physical storage media including storage discs and solid state devices. A computer-readable medium may also or alternatively include transient media such as carrier signals and transmission media. A computer-readable storage medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices. In the present application, the words “configured to...” are used to mean that an element of an apparatus has a configuration able to carry out the defined operation. In this context, a “configuration” means an arrangement or manner of interconnection of hardware or software. For example, the apparatus may have dedicated hardware which provides the defined operation, or a processor or other processing device may be programmed to perform the function. “Configured to” does not imply that the apparatus element needs to be changed in any way in order to provide the defined operation. Although illustrative teachings of the disclosure have been described in detail herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise teachings, and that various changes and modifications can be effected therein by one skilled in the art without departing from the scope and spirit of the invention as defined by the appended claims.

Claims

1. A computer-implemented method for fine-tuning a trained driving policy, the method comprising:generating a set of trajectories by, for each of one or more driving scene states, inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories;evaluating the set of trajectories based on one or more performance criteria;selecting for training a subset of the set of trajectories based on evaluating the plurality of trajectories, wherein selecting for training the subset of the set of trajectories comprises rejecting from selection for training one or more of the set of trajectories other than the subset; andtraining the trained driving policy using a machine learning technique and the subset of the set of trajectories.

2. The method of any preceding claim, wherein the trained driving policy has been previously trained to output multiple trajectories for a given input driving scene state.

3. The method of any preceding claim, wherein evaluating the set of trajectories comprises ranking the set of trajectories based on the one or more performance criteria.

4. The method of claim 3, wherein the subset of the set of trajectories corresponds to a predetermined number of highest ranked trajectories.

5. The method of any preceding claim, wherein generating the set of trajectories is based on simulating driving behaviour of a simulated agent in a driving scene associated with the one or more driving scene states.

6. The method of claim 5, wherein simulating driving behaviour comprises controlling driving behaviour of the simulated agent to follow one of the plurality of trajectories obtained from the trained driving policy.

7. The method of any preceding claim, wherein the one or more driving scene states are associated with a state of the driving scene at different time steps through a simulation of action in the driving scene.

8. The method of any preceding claim, wherein generating the set of trajectories comprises, for a plurality of time steps each associated with a driving scene state, inputting a respective driving scene state to the trained driving policy to obtain a plurality of trajectories for the respective time step.

9. The method of claim 8, further comprising selecting one trajectory from the plurality of trajectories for the respective time step for simulation, and simulating driving behaviour of a simulated agent to follow the selected one trajectory to determine a resulting driving scene state for use as a driving scene state input to the trained driving policy for a next time step.

10. The method of claim 9, wherein selecting one trajectory from the plurality of trajectories for the respective time step for simulation comprises selecting a trajectory based on determining performance of each of the plurality of trajectories for the respective time step.

11. The method of claim 9, wherein evaluating the set of trajectories comprises evaluating the plurality of trajectories at each time step, and selecting for training comprises rejecting from selection for training one or more of the plurality of trajectories at each time step.

12. The method of any preceding claim, wherein the one or more performance criteria is associated with a safety level associated with a given trajectory.

13. The method of any preceding claim, wherein training the trained driving policy comprises using the subset of the set of trajectories as ground truth trajectories for associated driving scene states.

14. The method of any preceding claim, wherein evaluating the set of trajectories comprises: determining one or more performance metrics associated with each of the plurality of trajectories based on simulating driving behaviour associated with each of the plurality of trajectories.

15. The method of claim 14, wherein the one or more performance metrics comprise metrics associated with: acceleration, braking, driving off-route, vehicle proximity, collisions, and compliance with driving rules.

16. The method of any preceding claim, wherein evaluating the set of trajectories comprises inputting the plurality of trajectories to a large language model (LLM) with a prompt requesting evaluation of the set of trajectories.

17. The method of claim 16, wherein the prompt is to request the LLM to rank the set of trajectories based on the one or more driving objectives.

18. The method of claims 16 or 17, wherein inputting the plurality of trajectories to the LLM comprises inputting, for each trajectory of the plurality of the trajectories, data indicative of positions and headings of one or more vehicles in a driving scene associated with the respective driving scene state.

19. The method of any preceding claim, wherein the trained driving policy is an artificial neural network configured to output a trajectory to be followed by an autonomous vehicle in a subsequent time step based on an input driving scene state.

20. The method of any preceding claim, wherein the trained driving policy is for controlling an autonomous vehicle.

21. The method of any preceding claim, further comprising controlling an autonomous vehicle based on the trained driving policy after training the trained driving policy using imitation learning and the subset of the set of trajectories.

22. An apparatus comprising one or more processors configured to carry out the method of any of claims 1 to 21.

23. A computer-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any of claims 1 to 21.

24. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any of claims 1 to 21.

Citation Information

Patent Citations

  • ViewUS2020090042A1onEspacenetopensinnewtab

  • ViewUS2025145176A1onEspacenetopensinnewtab

  • ViewUS2023159047A1onEspacenetopensinnewtab

  • ViewUS12420412B2onEspacenetopensinnewtab