Generating a trajectory for an autonomous vehicle

Abstract

The present invention relates to a computer-implemented method of generating a trajectory 118 for an autonomous vehicle 10, AV, using an autonomy stack, the method comprising: receiving a free-space a

Description

FIELD

[01] The subject-matter of the present disclosure relates to generating a trajectory for an autonomous vehicle. BACKGROUND

[02] An autonomy stack for an autonomous vehicle may include a first component which generates a trajectory based on sensor data, and a second component which assesses potential infractions between the trajectory and those of objects in the scene. In the event that an infraction is likely to occur, the second component is able to adjust the trajectory to avoid the infraction.

[03] Previous attempts have taken a simplistic approach which assumes that all other objects will move along a straight trajectory, at constant speed, and are effectively blind to other objects, including the autonomous vehicle (or ego-vehicle).

[04] It is an aim of the present invention to address such problems and improve on the prior art. SUMMARY

[05] According to an aspect of the present disclosure, there is provided a computer-implemented method of generating a trajectory for an autonomous vehicle, AV, using an autonomy stack, the computer-implemented method comprising: receiving a free-space area in which the AV operates, an object in the free-space area, a current trajectory for the AV, and a heuristic governing how the object will behave in response to the AV; constructing a plurality of trajectories for the object using the heuristic and the free-space area; constructing a plurality of trajectories for the AV using the current trajectory and the free-space area; comparing each of the trajectories for the AV with (or to) each of the trajectories for the object to detect which combinations involve an infraction between the AV and the object; and generating a trajectory for the AV based on the comparing, wherein the heuristic includes constraining a lateral offset of each trajectory of the object to a nearside direction of the object.

[06] Constraining a lateral offset of each trajectory of the object to a nearside direction of the object means that the object will not be constrained to moving only in a straight line. By enabling movement of the object to a nearside direction (e.g. to a left in the UK and to a right in the USA), false positive infractions are less likely to occur in certain driving situations such as driving around a bend on a road.

[07] In an embodiment, generating a trajectory for the AV based on the comparing comprises: when one or more combinations do not involve an infraction, selecting a constructed trajectory associated with one of the combinations not involving an infraction as the trajectory for the AV.

[08] In an embodiment, generating a trajectory for the AV based on the comparing comprises: when all combinations involve an infraction, replacing the current trajectory with a minimal risk manoeuvre.

[09] In an embodiment, the heuristic includes an acceleration of the object being constrained to be less than or equal to 0 m / s2. In other words, the object may continue at constant speed or decelerate. The object may not accelerate. In the event of a potential collision, another object is less likely to accelerate out of danger.

[10] In an embodiment, the heuristic includes constraining a lateral offset of the trajectory of the object to be towards a near side road boundary of the object, the near side road boundary bordering the free-space area.

[11] In an embodiment, the lateral offset is greater than or equal to a predetermined distance from a near side road boundary for the object. In a situation, for example, where the object pulls over to a side of the road, the predetermined distance may equal zero.

[12] In an embodiment, the heuristic includes constraining the object to steering only if another object is in front.

[13] In an embodiment, the other object is the AV.

[14] In an embodiment, computing the plurality of trajectories of the object comprises: setting a start point as a pose of the object and an end point at a lateral offset to a near side of the object; and fitting a spline between the start point and the end point.

[15] In an embodiment, fitting the spline comprises fitting a quintic Bezier curve.

[16] In an embodiment, the end point is in a direction toward a near side road boundary of the object, the near side road boundary bordering the free-space area.

[17] In an embodiment, the end point has a lateral position greater than or equal to a predetermined distance from a near side road boundary of the object.

[18] In an embodiment, the predetermined distance is 0m.

[19] In an embodiment, computing a trajectory of the object comprises: retrieving a plurality of predefined splines for the object; and setting the plurality of predefined splines to have a start point at a pose of the object.

[20] In an embodiment, end points of the respective predefined splines vary in longitudinal offset and lateral offset.

[21] In an embodiment, the free-space area and the object are generic.

[22] In an embodiment, receiving the free-space area and the object comprises receiving a dynamic occupancy grid includes a plurality of cells, wherein each cell corresponds to one of: a free-space area, a non-free-space area, and the object.

[23] In an embodiment, constructing the plurality of trajectories for the AV comprises: adjusting a lateral offset of the current trajectory of the AV to move toward a near side of the AV.

[24] In an embodiment, constructing the plurality of trajectories for the AV comprises: constraining an acceleration of the AV to be less than or equal to 0 m / s2.

[25] In an embodiment, constructing the plurality of trajectories for the AV based on the detected infraction comprises: constraining a lateral offset of an end point of the trajectory of the AV to be greater than or equal to a predetermined distance from a near side road boundary for the AV, the near side road boundary bordering the free-space area.

[26] According to an aspect of the present disclosure, there is provided an autonomous vehicle comprising: an actuator; at least one processor; and storage having instructions stored thereon that when executed by the at least one processor cause the at least one processor to perform the computer-implemented method of any preceding aspect or embodiment, wherein the actuator is configured to move the autonomous vehicle according to the adjusted trajectory.

[27] According to an aspect of the present disclosure, there is provided a transitory, or non-transitory, computer-readable medium, having instructions stored thereon that when executed by at least one processor cause the at least one processor to perform the computer-implemented method of any preceding aspect of embodiment. BRIEF DESCRIPTION OF DRAWINGS

[28] The subject-matter of the present disclosure is best described with reference to the accompanying figures, in which:

[29] Figure 1 shows a schematic block diagram of an autonomous vehicle (AV) according to one or more embodiment;

[30] Figure 2 shows a block diagram of an architecture of an autonomy stack for controlling the AV from Figure 1, according to one or more embodiments;

[31] Figure 3 shows block diagram a part of the architecture of the autonomy stack from Figure 2, according to one or more embodiments;

[32] Figure 4 shows a flow chart of a computer-implemented method of generating a trajectory for the AV, using the autonomy stack from Figures 2 and 3; and

[33] Figure 5 shows a flow chart of training a machine learning model included in the autonomy stack of Figures 2 and 3;

[34] Figure 6 shows a flow chart of a computer-implemented method of generating a trajectory for the AV, using the autonomy stack form Figures 2 and 3, according to at least one embodiment;

[35] Figures 7A and 7B respectively show schematics illustrating a potential infraction between the AV and a stationary object, and an adjusted trajectory of the AV to avoid the potential infraction;

[36] Figures 8A and 8B respectively show schematics illustrating a potential infraction between the AV and an oncoming vehicle, and adjusted trajectories of the AV and the object to avoid the potential infraction;

[37] Figures 9A and 9B respectively show schematics illustrating a potential infraction between the AV and an oncoming object when the AV is in the wrong lane, and adjusted trajectories of the AV and the object to avoid the potential infraction;

[38] Figures 10A and 10B respectively show schematics illustrating a potential infraction between the AV and an oncoming vehicle changing lanes, and adjusted trajectories of the AV and the object to avoid the potential infraction;

[39] Figures 11A and 11B respectively show schematics illustrating a potential infraction between the AV and a vehicle in the same lane in front of the AV, and adjusted trajectories of the AV and the object to avoid the potential infraction;

[40] Figures 12A and 12B respectively show schematics illustrating a potential infraction between the AV and an object travelling around a bend in a road, and adjusted trajectories of the AV and the object to avoid the potential infraction; and

[41] Figures 13A and 13B respectively show schematics illustrating a straight-line trajectory option for the AV, and a curved trajectory option for the AV where a collision may occur with an object. DESCRIPTION OF EMBODIMENTS

[42] The embodiments described herein may be embodied as sets of instructions stored as electronic data in one or more storage media. Specifically, the instructions may be provided on a transitory or non-transitory computer-readable media. When executed by the processor, the processor is configured to perform the various methods described in the following embodiments. In this way, the methods may be computer-implemented methods. In particular, the processor and a storage including the instructions may be incorporated into a vehicle. The vehicle may be an autonomous vehicle (AV).

[43] Whilst the following embodiments provide specific illustrative examples, those illustrative examples should not be taken as limiting, and the scope of protection is defined by the claims. Features from specific embodiments may be used in combination with features from other embodiments without extending the subject-matter beyond the content of the present disclosure.

[44] With reference to Figure 1, an AV 10 may include a plurality of sensors 12. The sensors 12 may be mounted on a roof of the AV 10, or integrated into the bumpers, grill, bodywork, etc. The sensors 12 may be communicatively connected to a computer 14. The computer 14 may be onboard the AV 10. The computer 14 may include a processor 16 and a memory 18. The memory may include the non-transitory computer-readable media described above. Alternatively, the non-transitory computer-readable media may be located remotely and may be communicatively linked to the computer 14 via the cloud 20. The computer 14 may be communicatively linked to one or more actuators 22 for control thereof to move the AV 10. The actuators may include, for example, a motor, a braking system, a power steering system, etc.

[45] The sensors 12 may include various sensor types. Examples of sensor types include LiDAR sensors, RADAR sensors, and cameras. Each sensor type may be referred to as a sensor modality. Each sensor type may record data associated with the sensor modality. For example, the LiDAR sensor may record LiDAR modality data.

[46] The data may capture various scenes that the AV 10 encounters. For example, a scene may be a visible scene around the AV 10 and may include roads, buildings, weather, objects (e.g. other vehicles, pedestrians, animals, etc.), etc.

[47] With reference to Figure 2, the instructions may form an autonomy stack. The autonomy stack includes a first component 30 and a second component 32. The first component 30 is for generating a trajectory for the AV based on sensor inputs 34 and the second component 32 is for adjusting the trajectory based on the sensor inputs 34. The sensor inputs for the first and second components 30, 32, may inputs from the same sensors 12.

[48] As alluded to above, the sensor inputs may be inputs from sensors 12 of different modalities. For example, the inputs may be from a camera 36, a LiDAR sensor 38, a RADAR sensor 40, odometry 42, and inertial measurement units (IMlls) 44.

[49] The first component 30 includes a perception module 46 and a planning module 48. The second component 32 also includes a perception module 50 and a planning module 52. Each of these modules includes further modules as described below. However, in summary, the perception module 46 of the first component 30 comprises a laser (LiDAR) localiser 54, a radar localiser 56, a camera localiser 58, a tracking module 60, a prediction module 62, a pose fusion module 64, and a first part of an end-to-end machine learning model 66, which may be a network such as a neural network. The planning module 48 of the first component 30 includes a planner 68 and a second portion of the end-to-end network 66. Figure 2 provides a condensed version of the second component 32, and a more detailed view is provided in Figure 3. For the purposes of Figure 2, the perception module 50 of the second component 32 includes a model-free perception module 70 and a machine learning (ML) perception module 72. For the purposes of Figure 3, the planning module 52 includes a validate trajectory module 74, a generate minimal risk manoeuvre, MRM, module 76, and a control module 78.

[50] The respective localisers 54, 56, 58, identify objects in a scene and their respective positions relative to the AV 10. The positions of the objects relative to the AV 10 are fused using the pose fusion module 64. The fused positions are output to the tracking module 60, the prediction module 62, and the planning module 68.

[51] The end-to-end network 66 is configured to output a trajectory for the AV 10 based on sensor inputs from the various sensors 12.

[52] At a hidden layer of the network 66, positions of objects are identified relative to the AV 10. The positions are output to the tracking module 60 and decoded by a decoder to a form similar to the form output by the pose fusion module 64. The tracking module 60 is configured to fuse the positions and track them temporally to predict the respective velocity of each object.

[53] The prediction module 62 is configured to receive the object positions from the pose fusion module 64, the object positions and velocities from the tracking module 60, and the object positions and velocities from a hidden layer of the end-to-end network 66. The prediction module 62 is configured to predict positions of the objects relative to the AV 10 at future time points.

[54] The planner module 68 is configured to receive the predicted future positions of the objects from the prediction module 62, the positions of the objects from the pose fusion module 64 and a trajectory from the end-to-end network 66. The trajectory from the end-to-end network 66 may be the output from the network 66. Based on these inputs, the planner module 68 is configured to generate a trajectory for the AV 10. The trajectory may include a nominal trajectory and an MRM trajectory. The trajectory is output to the validate trajectory module 74.

[55] The end-to-end network 66 is a machine learning model. The other modules 54-64, 68, of the first component 30 are rules-based models. In some embodiments, the other modules 54-56, 68, of the first component may be machine learning models each trained separately.

[56] The modules and functionality of the second component 32 are best described with reference to Figure 3.

[57] With reference to Figure 3, the sensors 12 may also include a localisation 80 of the AV 10 from a map. The ML perception module 72 may include a first machine learning model 82 and a second machine learning model 84. The perception module 50 of the second component 32 may further comprise a localisation validation model 86, a first compute control constraint module 88, and a second compute control constraint module 90. The validate trajectory module 74 may include a trajectory adjuster module 92, and a trajectory validator module 94. The localisation validation module 86 is configured to validate the localisation of the AV 10.

[58] The autonomy stack, and more particularly, the second component of the autonomy stack, may be operated as a computer-implemented method. The method may include various steps as outlined below.

[59] In summary, and as shown in Figure 4, the method may be summarised as a computer-implemented method of generating a trajectory for an autonomous vehicle, AV, 10 using an autonomy stack, the autonomy stack including a first component 30 and a second component 32, the first component 30 for generating a trajectory for the AV based on sensor inputs and the second component 32 for adjusting the trajectory based on the sensor inputs, the first and second components 30, 32 each including a perception module 46, 50 and a planning module 48, 52, the computer-implemented method comprising: identifying S100, using the perception module 50 of the first component, objects based on sensor inputs; generating S102, using the planning module 48 of the first component, a trajectory for the AV based on the objects identified by the perception module 46 of the first component; identifying S104, using a perception module 50 of the second component, objects based on the sensor inputs; and adjusting S106, using the planning module 52 of the second component, the trajectory based on the objects identified by the perception module 50 of the second component, wherein the perception module 50 of the second component includes one or more rules-based models and one or more machine learning models.

[60] With further reference to Figure 3, the identification of the objects by the perception module of the second component includes identifying the objects and labelling the objects as generic objects. The term “generic” is used to mean no semantic labels are added. In other words, all objects may be treated equally. This may be in contrast to the object identification occurring in the first component, where the objects may be labelled according to their semantic class, e.g. a vehicle, a pedestrian, a dog, a mailbox, etc. In this way, the computation takes less time to compute the objects and use them for further processing operations. In addition, by not classifying the objects semantically, the second component becomes more conservative.

[61] To identify the objects, the first machine learning model 82 is trained to identify objects from images. The first machine learning model 82 may be a neural network, and preferably a deep neural network. The neural network may comprise, or may be, a convolutional neural network.

[62] To identify the object, the second machine learning model 84 is trained to identify objects from RADAR and / or LiDAR data. The RADAR and LiDAR data may be provided in the forms of point clouds. To achieve this, the second machine learning model 84 may comprise two independent machine learning models one for each modality. Alternatively, the RADAR and LiDAR data may be combined into a single point cloud and the second machine learning model 84 identifies the objects using the single point cloud as inputs.

[63] The second machine learning model 84 may be a neural network, and preferably may be a deep neural network. The neural network may comprise, or may be, a recurrent neural network, or may be, or may comprise, a convolutional neural network.

[64] The identification of the objects by the first and second machine learning models, 82, 84, may include generating first and second occupancy grids 96, respectively. The respective occupancy grids each include a plurality of grid segments. Each segment may be labelled with a state. Overall, the states define whether or not the AV 10 is able to travel in that segment based on the presence of any objects in an area covered by the grid. The states may include occupied 98, where the segment is occupied by an object. The occupied state may be accompanied by a velocity of the object that occupies that state if the respective machine learning model 82, 84, has been trained to detect object velocities in addition to position. The states may also include occluded 100, where the segment is unreachable because of an occupied 98 state of a grid segment between it and the AV 10. The states may also include free-space 102. A free-space 102 state grid is one available for the AV 10 to travel to. The free-space, or free-space area, may thus be an area between opposing road boundaries. The road boundaries may include a near side road boundary for the AV 10 and an offside road boundary for the AV 10. The terms “nearside” and “offside” may be dictated by a frame of reference to an observer. For example, when the near side for the AV 10 may be the offside for an object travelling in a different lane in an opposite direction to the AV 10. The terms nearside and offside may mean left and right, respectively, for countries such as the UK where vehicles travel on a left side of a road, and may mean right and left, respectively, for countries such as USA where vehicles drive on a right side of a road.

[65] The method may also comprise computing, using the compute constraint modules 88, 90, a set of control constraints 104 for each occupancy grid. The set of control constraints defines permitted action spaces for the AV 10. The control constraints may be defined by an intersection of two control planes. A control plane is used herein to mean a dynamic parameter. The two control planes may include one of acceleration, deceleration, steering angle, steering rate, jerk, and velocity. For example, the control constraint may be the variability of one control plane with respect to, or in the domain of, another control plane. For example, the control constraint may be the variability of acceleration in the steering angle domain, or in other words, how much the AV can accelerate at any given steering angle and avoid as infraction, e.g. a collision, with the identified objects. The control constraint 104 is shown graphically in Figure 3. Computing the control constraint for each occupancy grid may comprise computing the control constraint using a graphical processing unit (GPU). Using a GPU at the end of the respective machine learning model reduces bandwidth and central processing unit (CPU) usage with downstream functions of the second component 32.

[66] Calculating the control constraint may be performed based on each of the first and second occupancy grids. In other embodiments, the control constraints 104 may be computed for a combined occupancy grid. Combining the occupancy grids may reduce the impact of false grid predictions, e.g. where a segment is occupied in one grid and free-space in another grid. The method of combining the occupancy grids may be defined as follows.

[67] The method may comprise comparing the first and second occupancy grids. When the states of both occupancy grids match one another, the grids may be combined. When any states of the occupancy grids do not match, the mismatched grid segments may be compared to the same segment of an occupancy grid generated by the rules-based model, e.g. the model-free perception module 70. In some embodiments this may be at a segment-by-segment level, in other embodiments, entire grids will be compared.

[68] It should be noted that this final check with the grid from the rules-based model-free perception module 70 is non-trivial. This is because the grids output by the model-free perception module 70 may not include all states depending on the modality of the sensor inputs.

[69] For example, occupancy states can be checked using grids generated based on either RADAR or LiDAR modality sensor inputs. With the above in mind, it should be noted that the model-free perception module 70, or rules-based perception module, can include a RADAR based model and a LiDAR based model. The RADAR based model identifies objects from RADAR sensor inputs. The LiDAR based model identifies objects from LiDAR sensor inputs.

[70] Occlusion states are difficult to check against RADAR derived grids, and instead should be checked using LiDAR derived grids. Velocity is difficult to check against LiDAR derived grids. A radial component of velocity can be checked using a RADAR derived grids. For velocity in general, a temporal consistency check may be performed using future and / or past grids generated from either the rules based module or the machine learning models.

[71] In some embodiments, the method comprises generating a confidence score associated with each state and by-passing the comparison of the first and second occupancy grids if the score is above a confidence threshold. This may be particularly beneficial for certain circumstances such as not removing true positive states for rare objects or objects only detectable through one modality. For instance, a black object at night will unlikely be detectable from images and LiDAR but may be detectable by radar.

[72] It will be appreciated that the planner module of the first component 30 generates the trajectory based on inputs from various first component modules and also based on the outputs of the perception module of the second component 32. In this way, since the outputs of the perception module are more conservative, due at least in part to them relating to generic objects, the trajectory will be more conservative.

[73] The trajectory generated by the planning module 68 of the first component 30 includes a nominal trajectory for the AV and an MRM trajectory for the AV. The MRM trajectory may involve actions such as changing lane or pulling to the side of a road, for example. The method may include adjusting, using the MRM adjuster module 92, the trajectory based on the objects identified by the perception module of the second component. More specifically, the trajectory may be adjusted based on the occupancy grid(s). This may be directly from the occupancy grids perse, or indirectly by basing the adjustment on the control constraints. This may be achieved by adjusting the nominal trajectory and the MRM trajectory to avoid infractions, e.g. collisions, with objects detected by the perception module of the second component 32, while not compromising on passenger comfort.

[74] The method may also include, generating, using a generate MRM module 76, a further MRM trajectory based solely on the objects identified by the perception module of the second component 32. The further MRM trajectory may be configured to avoid any collisions with the objects identified by the perception module 50 of the second component 32. The further MRM trajectory will be more conservative compared to the MRM trajectory because it is determined solely based on generic objects.

[75] Next, the method includes selecting a final trajectory from the adjusted nominal trajectory, the adjusted MRM trajectory, and the further MRM trajectory. This is done using the trajectory validator module 94. The trajectory validator module 94 may also select the final trajectory based on comfort of occupants in the AV 10, e.g. jerk being below a threshold. The selection of a final trajectory may have a fixed priority order from the three trajectories (adjusted nominal, adjusted MRM, and further MRM). The fixed priority order is determined by the trajectory validator module 94. The order may be decided based on a risk of collisions between the AV 10 and an object, for example. The order may be based on determining if the adjusted nominal trajectory, the adjusted MRM trajectory, and the further MRM trajectory are collision free by comparison with the objects identified by the perception module of the second component; selecting the adjusted nominal trajectory as a final trajectory if the adjusted nominal trajectory is collision free; if the adjusted nominal trajectory is not collision free, selecting the adjusted MRM trajectory as the final trajectory if the adjusted MRM trajectory is collision free; and if the further MRM trajectory is collision free, selecting the further MRM trajectory as the final trajectory

[76] The control module 78 may be configured to convert the final trajectory to a set of actuator configurations so the AV 10 can execute the final trajectory.

[77] Since the first and second machine learning models 82, 84, are data based, they require training. The computer-implemented method for training the machine learning models can be summarised with reference to Figure 5.

[78] With reference to Figure 5, a summary of the computer-implemented method of training a machine learning algorithm of a perception module of a second component in an autonomy stack for controlling an autonomous vehicle, AV, comprises: identifying S200, using the one or more rules-based models, objects using sensor inputs; labelling S202 the sensor inputs and the identified objects automatically as paired data; and training S204 the one or more machine learning models to identify objects using the paired sensor inputs and identified objects.

[79] The objects identified by the rules-based models can include generating an occupancy grid as described above. Training data can be generated by pairing the occupancy grids with the sensor inputs used to generate them. The machine learning model can be trained to generate their own occupancy grids based on sensor inputs.

[80] Such training data works well for states such as occupancy, occlusion, and free-space. However, velocity is more difficult. To train the machine learning models to determine velocity of an object, the sensor inputs may be paired with occupancy grids of future time points generated by the rules-based perception model. In this way, the training data will be temporal paired data. In this way, the machine learning models will be trained to generate an occupancy grid with the occupancy state including velocity of an occupying object using the temporal paired data.

[81] With reference to Figure 6, a computer-implemented method of generating a trajectory for an autonomous vehicle, AV, using an autonomy stack, the computer-implemented method comprising: receiving S300 a free-space area in which the AV operates, an object in the free-space area, a current trajectory for the AV, and a heuristic governing how the object will behave in response to the AV; constructing S302 a plurality of trajectories for the object using the heuristic and the free-space area; constructing S304 a plurality of trajectories for the AV using the current trajectory and the free-space area; comparing S306 each of the trajectories for the AV with each of the trajectories for the object to detect which combinations involve an infraction between the AV and the object; and generating S308 a trajectory for the AV 10 based on the comparing, wherein the heuristic includes constraining a lateral offset of each trajectory of the object to a nearside direction of the object. This last clause may be expressed differently as wherein the heuristic includes constraining a lateral movement of the object to be towards a near side direction of the object. In other words, lateral movement of the object is permitted only in a nearside direction (e.g. movement to the left only for UK driving).

[82] Generating a trajectory for the AV based on the comparing may be performed differently depending on the outcome of the comparisons. For example, when all comparisons have been performed, there may be a cases where one or more of the combinations used in the comparisons do not involve an infraction. In this case, a constructed trajectory associated with one of the combinations not involving an infraction is selected as the trajectory for the AV. In other words, the current trajectory is adjust based on the selected trajectory. More specifically, the current trajectory is replaced with the selected trajectory.

[83] For example, if there are 4 trajectories for the AV 10 and 4 trajectories for the object, there may be 4x4 combinations to compare. If 3 of those combinations do not result in an infraction, then one of those trajectories is selected for use as the trajectory for the AV.

[84] Generating a trajectory for the AV based on the comparing may be performed when all combinations involve an infraction. In this case, the current trajectory may be replaced with a MRM. The MRM may be one of the constructed trajectories. The constructed trajectories may include MRM trajectories and non-MRM trajectories.

[85] The AV 10 may also be called the ego-vehicle.

[86] Constructing the plurality of trajectories for the AV using the current trajectory and the free-space area may include using the current trajectory as one of the constructed plurality of trajectories. The other trajectories of the plurality of constructed trajectories may be adjusted versions of the current trajectory.

[87] In addition, the method may comprise not adjusting the trajectory of the AV if an infraction will not occur between the trajectory of the AV and the trajectory of the object.

[88] This method may be implemented using the MRM trajectory adjuster module 92. For instance, the MRM trajectory adjuster module 92 may receive the free-space area and an object associated with a scene in which the AV is operating. More specifically, the MRM trajectory adjuster module 76 may receive the dynamic occupancy grid including cells corresponding to free-space, non-free-space, and an object. The non-free-space cells correspond positionally to regions outside the opposing road boundaries. As mentioned above, the free-space area and the object are generic. In other words, they include no semantic information. For example, the objects have not been classified as, for example, a car, a person, etc. Similarly, the free-space has not been classified as, for example, terrain type.

[89] The method comprises receiving a heuristic governing how the object will behave in response to the AV. The heuristic may be a hard coded rule, and may not be learned.

[90] The heuristic includes that a lateral direction of the trajectory computed for the object is constrained to an onside direction. Therefore, if the trajectory for the object moves laterally, e.g. the trajectory does not proceed in a straight forward line, the object may only move toward an onside direction. For objects in the UK, that means the object may only move to a left.

[91] The heuristic may also include an acceleration of the object being constrained to be less than or equal to 0 m / s2. In other words, the object may only move at a constant velocity or may only decelerate.

[92] The heuristic may also include constraining a lateral offset of the trajectory of the object to be greater than or equal to a predetermined distance from a near side road boundary for the object. The predetermined distance may be greater than or equal to zero metres. For example, when the predetermined offset is zero metres, the lateral offset of the trajectory, may equal the near side road boundary. The lateral offset may be a lateral offset of an end point of the trajectory compared to a starting point of the trajectory.

[93] The end point of the trajectory may literally be an end point in the sense that the AV 10 comes to a complete stop. Alternatively, the end point may mean an end point of an adjustment made to the current trajectory. Since the validate trajectory 74 module can output to the planner of the first component, the trajectory from the first component will update using the end point of the adjusted trajectory and modify a future trajectory so that the AV 10 continues moving, at least where no infraction exists.

[94] The heuristic may also include constraining the object to steering only if another object is in front (of the object). The other object may be the AV 10. In other words, if the AV 10 is behind the object, the object will not steer to avoid a collision with the AV 10. However, if the AV 10 is in front of the object, the object is able to steer, e.g. move laterally (in a near side direction of the object) to take evasive action to avoid the collision. In other words, the object is effectively blind to anything that is not in front of it.

[95] The method comprises receiving the trajectory for the AV 10. The method comprises constructing a plurality of trajectories for the object using the heuristic and the free-space area. This may be achieved by: setting a start point as a pose of the object and an end point at a lateral offset to a near side of the object; and fitting a spline between the start point and the end point. In other words, the start point of the spline may equal the pose of the object. The end point of the spline may be in a nearside direction of the object, e.g. in the UK the end point may be to a left of the object. The end point may also have a longitudinal offset in front of the object. The longitudinal offset may be set according to a braking distance of the object. The longitudinal offset may also be extended further forward if required according to one or more of the heuristics. For example, the end point may be greater than or equal to a predetermined distance from a near side road boundary of the object. Where the end point is intended to be the near side road boundary, the predetermined distance may equal zero. Therefore, if the heuristic includes that the object must terminate travel at a near side road boundary, the longitudinal offset may be increased to accommodate this heuristic. It should be noted that the lateral offset and / or the longitudinal offset may thus be set according to the one or more heuristics.

[96] Fitting the spline may comprise fitting a quintic Bezier curve between the start and end points.

[97] Alternatively, computing a trajectory of the object using the heuristic comprises: retrieving a plurality of predefined splines for the object; and setting the plurality of predefined splines to have a start point at a pose of the object. The plurality of predefined splines may each be Bezier curves between start and end points. As above, the end points may have a longitudinal offset and a lateral offset compared to the start point. The end points of the respective predefined splines may vary in longitudinal offset and lateral offset.

[98] The method also comprises computing a plurality of trajectories for the AV 10 using the current trajectory and the free-space area.

[99] This may be achieved using rules that are similar to the heuristics used to govern how the object moves in relation to the AV 10. For instance, a lateral direction of the trajectory of the AV 10 may be adjusted to move toward a nearside of the AV 10. In the UK, this means the AV’s 10 trajectory is adjusted to move left, and in the USA, this means the AV’s 10 trajectory is adjusted to move right.

[100] Additionally, or alternatively, the trajectory of the AV 10 may be adjusted to maintain a constant speed or decelerate only. The AV 10 may not be permitted to accelerate. In other words, the trajectory of the AV 10 may be adjusted by constraining an acceleration of the AV 10 to be less than or equal to 0 m / s2.

[101] Additionally, or alternatively, the trajectory of the AV 10 may be adjusted by constraining a lateral offset of the trajectory of the AV to be greater than or equal to a predetermined distance from a nearside road boundary for the AV 10. This predetermined distance may be 0 m. In other words, the trajectory may be adjusted such that the end point of the trajectory may have a lateral offset that is at a position of the nearside road boundary.

[102] Therefore, the plurality of trajectories for the AV 10 include the current trajectory for the AV received from the first component, and a plurality of adjusted trajectories adjusted using the rules set out above.

[103] The method also comprises comparing mutually exhaustively each of the trajectories for the AV and each of the trajectories for the object to detect which combinations involve an infraction between the AV and the object. An infraction may occur when the trajectories intersect. In some embodiments, the control constraints may be used to detect a collision, for instance where the movement required to avoid a collision exceeds one or more control constraints.

[104] When one or more combinations do not involve an infraction, the method comprises selecting a constructed trajectory associated with one of the combinations not involving an infraction as the trajectory for the AV. In other words, one of the combinations not involving a trajectory will be selected and the trajectory for the AV used in that combination will be used as the trajectory for the AV 10. If there is more than one combination not involving an infraction, one of those combinations may be selected. If the current trajectory from the first component is one of those combinations, then the current trajectory will be selected. If there is more than one combination not including the current trajectory, one of those combinations may be selected. The selection may be arbitrary. In more sophisticated algorithms, a trajectory for the AV requiring a minimum adjustment from the current trajectory will be selected.

[105] When all combinations involve an infraction, the method comprising replacing the current trajectory with a MRM. The MRM may include an emergency stop, for example.

[106] The comparison of each of the trajectories of the AV with each of the trajectories of the object may be exhaustive. In other words, every trajectory of the AV is compared with every trajectory of the object.

[107] The AV 10 may be moved by controlling the actuator 22 to move the AV according to the adjusted trajectory.

[108] Various scenarios between the AV 10 and the object may be encountered. Some example scenarios are described subsequently. The behaviour of the AV 10 and the object during those scenarios will be dictated by the heuristics used to constrain how the object responds to the AV 10.

[109] With reference to Figures 7A and 7B, in a first scenario, the object 110 is a stationary object in the same lane 112 of the road 114 as the AV 10. The road 114, which is classed as a free-space area, is bordered by a near side road boundary 116 and an offside road boundary 118.

[110] This scenario is addressed by assuming that objects 110 can only decelerate during their safety procedures. With this assumption, the static object remains stationary. Since the AV 10 is only able to move left or decelerate, the trajectory 118 of the AV 10 is adjusted to brake to a stop position, e.g. emergency stop, or to pull over to the near side road boundary 112 (e.g. the left kerb for UK driving) and stop. To do this, the end point of the trajectory is positioned to coincide with the near side road boundary, or be a predetermined distance therefrom.

[111] With reference to Figures 8A and 8B, in a second scenario, a head-on collision is foreseen between the AV 10 and the object 110.

[112] This scenario is addressed using the heuristics that the object 110 can only decelerate and move to a near side of the object during a safety procedure (when a collision has been detected. In addition, the AV 10 is constrained to decelerate and also move to a near side of the AV 10. Therefore, the trajectory 120 of the object 110 is constructed to have an end point with a lateral offset at a predetermined distance from a near side road boundary 116 of the object 110. The trajectory 118 of the AV 10 is adjusted to have a lateral offset at a predetermined distance from the near side road boundary 112 of the AV 10.

[113] With reference to Figures 9A and 9B, in a third scenario, a head-on collision is predicted when the AV 10 is travelling in an opposing lane. For instance, the AV 10 is in a right-hand lane when driving in the UK or is driving in a left-hand lane when driving in the USA.

[114] Using the heuristic that the object 110 can only move in a near side direction, and can only decelerate, it can be seen that the object 110 is unable to move very far to the left because it is already in the correct lane. Therefore, the trajectory for the object 110 has an end point that is constrained to a predetermined distance from the near side road boundary 116, which results only in a small lateral movement of the object 110.

[115] Therefore, when adjusting the trajectory 118 of the AV 10, the end point is offset laterally to a predetermined distance from the near side road boundary 112. This implicitly captures the right-of-way concept of being in the wrong lane, and returns the AV 10 back to its correct lane.

[116] A symmetric response may not be reasonable if other objects have a clear right-of-way. For example, if the ego vehicle (AV 10) is driving in an oncoming lane, the expectation is that we should move out of the way, not the other way around. This can be addressed by only allowing vehicles to move left (for UK roads). The road boundaries then induce the asymmetric response, producing a larger displacement for the ego vehicle. While this would be a “later response” than the simple straight-line assumption, it is reasonable to assume that in an imminent collision, everyone should do their part for the collision to be avoided.

[117] With reference to Figures 10A and 10B, in a fourth scenario, an object, e.g. an oncoming vehicle, is encroaching from an opposing lane of the road. The oncoming vehicle is performing a lane change. The trajectory 120 of the object 110 is constrained to move laterally to a near side road boundary 116, and decelerate. Therefore, the trajectory of the object 110 is constructed as shown in Figure 10B. The trajectory 118 of the AV 10 is adjusted to move laterally toward a near side road boundary of the AV and come to a stop. In this way, a minimal risk manoeuvre is performed by both the AV 10 and the object 110 to avoid a collision.

[118] With reference to Figures 11A and 11B, in a fifth scenario, the AV 10 is following an object 110, e.g. another vehicle. In this right-of-way scenario, the AV 10 should not expect the lead object to move out of its way. This is addressed by the heuristic that objects only react, e.g. steer, for other objects (e.g. the AV 10) that are in front of them. In other words, this is addressed by assuming that objects only react to others in front of them, and determine which objects are “relevant” for each actor in the trajectory generation algorithm. In this case, since no other objects are in front of the lead object. Therefore, the AV 10 is able to adjust its trajectory 118 to have an end point that is laterally offset to a near side road boundary direction, and the AV 10 also decelerates. The end point may not necessarily mean the AV comes to a complete stop. Since the adjusted trajectory will change a position of the AV, this change in position will be detected by the sensors. Therefore, the inputs to the end-to-end network will change. The end-to-end network may generate a different trajectory in future which will be output to the planner of the first component, and then to the MRM trajectory adjuster 74 to perform the method of constructing the various trajectories, comparing them, checking for infractions, and making adjustments where necessary. The same may be true with the other scenarios too.

[119] With reference to Figures 12A and 12B, in a sixth scenario, the AV 10 is travelling on a road bending in an offside direction. Under previous regimes, the trajectory of the object 110 was assumed to be only straight. This would mean that the AV 10 would have to adjust its trajectory to decelerate and come to a halt, e.g. an emergency stop, since the trajectory 118 of the AV 10 intersects the trajectory 120 of the object 110. Using the previously described heuristics, the trajectory of the object 110 may move to an onside direction. In this way, the object may follow the road, or free-space area, profile, at least for the short distance considered in a stopping manoeuvre. If the AV 10 needs to follow a curved road for a significant distance, the trajectory 118 of the AV 10 may be adjusted to fit a curve to the near side road boundary 112 and using it as the lateral offset reference path may be necessary. In such a situation, the trajectory for the AV 10 may not need to be adjusted since the near side steering of the object 110 will not result in an infraction, e.g. a collision. Therefore, a false positive trigger is avoided. If the near side steering of the object 110 still results in an infraction, e.g. a collision, the trajectory 118 of the AV 110 may be caused to decelerate and steer in a near side direction.

[120] Generally, having an option to steer to the left, aligned with the road boundary, should indicate that this is not an imminent collision. One method is to use the road boundary as a reference path, which can be estimated from the scene geometry. Alternatively, the observed steering circle of the objects can be used as a heuristic for their intended path.

[121] The scenarios described above may be considered to relate to a situation where the trajectory for the object and a trajectory for the AV are constructed using a spline by fitting a Bezier curve between a current pose and an end point.

[122] We assume that all objects should conduct an evasive manoeuvre (speed and steering), subject to reasonable heuristics. If the default safety procedures still result in a collision, then an intervention is required. This captures the idea that everyone will try to avoid a collision, and if all options are expired, we can still measure the severity of the collisions. We now consider a potential implementation. Based on the scenarios above, we can encapsulate the desired behaviour with three simple heuristics: 1. Decelerate 2. Steer only if there is another object in front 3. Pull over to the near side only

[123] With reference to Figures 13A and 13B, in a seventh scenario, a pull-over trajectory (computed in open loop) is inappropriate if there is a static object 110 in the way. Considering this type of trajectory would prevent overtakes, since there will be some point during the manoeuvre where a collision would be deemed unavoidable. Instead, of devising an alternative formulation for the trajectory of the AV 10 that avoids collisions, it is more efficient to sample a set of predefined trajectories per object and choose one that is collision free. The longitudinal and lateral offsets of the end point can be varied, and even a straight line trajectory can be selected as a fall-back. In other words, there are two predefined trajectories shown, one in Figure 13A and one in Figure 13B. The trajectory in Figure 13A is a straight line that does not result in an infraction. The trajectory in Figure 13B is a curve that does result in an infraction with a stationary object, e.g. a parked vehicle. Since the straight line trajectory does not result in an infraction, that trajectory is selected as the trajectory for the AV.

Claims

1. A computer-implemented method of generating a trajectory for an autonomous vehicle, AV, using an autonomy stack, the computer-implemented method comprising:receiving a free-space area in which the AV operates, an object in the free-space area, a current trajectory for the AV, and a heuristic governing how the object will behave in response to the AV;constructing a plurality of trajectories for the object using the heuristic and the free-space area;constructing a plurality of trajectories for the AV using the current trajectory and the free-space area;comparing each of the trajectories for the AV with each of the trajectories for the object to detect which combinations involve an infraction between the AV and the object; andgenerating a trajectory for the AV based on the comparing,wherein the heuristic includes constraining a lateral offset of each trajectory of the object to a nearside direction of the object.

2. A computer-implemented method of Claim 1, wherein adjusting the current trajectory comprises:when one or more combinations do not involve an infraction, selecting a constructed trajectory associated with one of the combinations not involving an infraction as the trajectory for the AV.

3. A computer-implemented method of Claim 1 or Claim 2, wherein adjusting the current trajectory comprises:when all combinations involve an infraction, replacing the current trajectory with a minimal risk manoeuvre.

4. The computer-implemented method of any preceding claim, wherein the heuristic includes an acceleration of the object being constrained to be less than or equal to 0 m / s2.

5. The computer-implemented method of any preceding claim, wherein the heuristic includes constraining a lateral offset of the trajectory of the object to be towards a near side road boundary of the object, the near side road boundary bordering the free-space area.

6. The computer-implemented method of Claim 5, wherein the lateral offset is greater than or equal to a predetermined distance from a near side road boundary for the object.

7. The computer-implemented method of any preceding claim, wherein theheuristic includes constraining the object to steering only if another object is in front.

8. The computer-implemented method of Claim 7, wherein the other object is the AV.

9. The computer-implemented method of any preceding claim, wherein computing the plurality of trajectories of the object comprises:setting a start point as a pose of the object and an end point at a lateral offset to a near side of the object; andfitting a spline between the start point and the end point.

10. The computer-implemented method of Claim 9, wherein fitting the spline comprises fitting a quintic Bezier curve.

11. The computer-implemented method of Claim 9 or Claim 10, wherein the end point is in a direction toward a near side road boundary of the object, the near side road boundary bordering the free-space area.

12. The computer-implemented method of Claim 11, wherein the end point has a lateral position greater than or equal to a predetermined distance from a near side road boundary of the object.

13. The computer-implemented method of Claim 12, wherein the predetermined distance is 0m.

14. The computer-implemented method of any of Claims 1 to 8, wherein computing a trajectory of the object comprises:retrieving a plurality of predefined splines for the object; andsetting the plurality of predefined splines to have a start point at a pose of the object.

15. The computer-implemented method of Claim 14, wherein end points of the respective predefined splines vary in longitudinal offset and lateral offset.

16. The computer-implemented method of any preceding claim, wherein the free-space area and the object are generic.

17. The computer-implemented method of any preceding claim, wherein receiving the free-space area and the object comprises receiving a dynamic occupancy grid includes a plurality of cells, wherein each cell corresponds to one of: a free-space area, a non-free-space area, and the object.

18. The computer-implemented method of any preceding claim, wherein constructing the plurality of trajectories for the AV comprises:adjusting a lateral offset of the current trajectory of the AV to move toward a near side of the AV.

19. The computer-implemented method of any preceding claim, wherein constructing the plurality of trajectories for the AV comprises:constraining an acceleration of the AV to be less than or equal to 0 m / s2.

20. The computer-implemented method of any preceding claim, wherein constructing the plurality of trajectories for the AV based on the detected infraction comprises:constraining a lateral offset of an end point of the trajectory of the AV to be greater than or equal to a predetermined distance from a near side road boundary for the AV, the near side road boundary bordering the free-space area.

21. An autonomous vehicle comprising:an actuator;at least one processor; andstorage having instructions stored thereon that when executed by the at least one processor cause the at least one processor to perform the computer-implemented method of any preceding claim,5 wherein the actuator is configured to move the autonomous vehicle accordingto the adjusted trajectory.

22. A transitory, or non-transitory, computer-readable medium, having instructions stored thereon that when executed by at least one processor cause the 10 at least one processor to perform the computer-implemented method of any of Claims 1 to 20.

Images