Autonomous Vehicle Control

Abstract

The present invention relates to a computer-implemented method of controlling an autonomous vehicle, the method comprising the steps of receiving current sensor data defining a current scenario, determining a distance between the current scenario and a closest scenario from a database of known scenarios, and controlling the autonomous vehicle by steering the autonomous vehicle within steering constraints associated with the closest scenario if the distance is less than a first threshold, performing a minimum risk maneuver if the distance exceeds a second threshold greater than the first threshold, and interpolating steering constraints associated with the closest scenario and steering the autonomous vehicle within the interpolated steering constraints if the distance is between the first and second thresholds.

Description

[Technical field]

[0001] The subject matter of this disclosure relates to computers, computer-implemented methods for controlling an autonomous vehicle, computer-implemented methods for determining steering constraints for an autonomous vehicle during a known scenario, and transitory or non-transitory computer-readable media. [Background technology]

[0002] An autonomous vehicle (AV) can capture multiple scenarios. Each scenario can be associated with a driving range. If the AV encounters a scenario in real time, it can traverse within the driving range that matches the scenario.

[0003] Problems arise when there is no known scenario to match.

[0004] It is an object of the present invention to address these problems and improve upon the prior art. Summary of the Invention

[0005] According to one aspect of the disclosure, a computer-implemented method for controlling an autonomous vehicle is provided, the method including receiving current sensor data defining a current scenario, determining a distance between the current scenario and a closest scenario from a database of known scenarios, and controlling the autonomous vehicle by maneuvering the autonomous vehicle within maneuvering constraints associated with the closest scenario if the distance is less than a first threshold, performing a minimum risk maneuver if the distance exceeds a second threshold greater than the first threshold, and interpolating maneuvering constraints associated with the closest scenario and maneuvering the autonomous vehicle within the interpolated maneuvering constraints if the distance is between the first and second thresholds.

[0006] In one embodiment, the current scenario and each known scenario may be a descriptor that represents a multivariate distribution from the bottleneck of the variational autoencoder.

[0007] In one embodiment, determining the distance may include calculating the distance using multiple weighted matchers.

[0008] In one embodiment, the multiple weighted matchers may include one or more of a time warping matcher, a Euclidean distance matcher, and a trained matcher.

[0009] In one embodiment, the steering constraints may include driving ranges constructed in space-time within which violations are predicted not to occur.

[0010] In one embodiment, the minimum risk maneuver may include one or more of an emergency stop and a pull-over.

[0011] According to one aspect of the invention, a computer-implemented method for determining steering constraints for an autonomous vehicle during a known scenario is provided, the method including: running the known scenario for the autonomous vehicle as a simulation in a simulator, iteratively running the simulation by applying different dynamic parameters to the autonomous vehicle, identifying a maximum dynamic parameter for the autonomous vehicle from the different dynamic parameters, below which no violation occurs during the simulation and above which a violation occurs during the simulation, and storing the maximum dynamic parameter in a constraint database as a steering constraint for the autonomous vehicle for the known scenario.

[0012] In one embodiment, the computer-implemented method further includes retrieving a descriptor of the known scenario from a database, where the descriptor is a multivariate distribution from a bottleneck of a variational autoencoder, and decoding the descriptor using a decoder from the variational autoencoder that generates the known scenario.

[0013] In one embodiment, iteratively running the simulation may include increasing and / or decreasing a dynamic parameter and checking for violations at each increase and / or decrease.

[0014] In one embodiment, the computer-implemented method may further include continuing to increase and / or decrease the dynamic parameter until there are no more adjustments.

[0015] In one embodiment, the increasing and / or decreasing steps may include increasing and / or decreasing at predetermined intervals.

[0016] In one embodiment, the dynamic parameters may be selected from a list including velocity, acceleration, deceleration, and jerk.

[0017] In one embodiment, the steering constraints include driving ranges constructed in space-time within which violations are predicted not to occur.

[0018] According to a further aspect of the present invention there is provided a transitory or non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform a computer implemented method according to any of the preceding claims. [Brief description of the drawings]

[0019] The subject matter of the present disclosure is best explained with reference to the accompanying drawings.

[0020] [Figure 1] 1 shows a schematic diagram of an AV according to one or more embodiments. [Diagram 2] FIG. 1 shows a block diagram of features used in the methods described herein. [Diagram 3] FIG. 2 shows a block diagram of features used to determine tracking and prediction of objects in a scenario captured by sensors of the AV of FIG. [Figure 4]FIG. 1 shows a block diagram of features used to determine tracking and prediction of objects in a scene in a simulator. [Diagram 5] 1 shows a block diagram of the features used to train the scenario encoder. [Figure 6] FIG. 6 shows a block diagram of features used to compare scenarios using the trained scenario encoder of FIG. [Figure 7] A block diagram of the scenario comparator shown in FIG. 6 is shown. [Figure 8] 1 illustrates a flowchart of a method for determining steering constraints according to one or more embodiments. [Figure 9] 2 shows a flow chart of a method for controlling the AV of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The embodiments described herein are embodied as a set of instructions stored as electronic data on one or more storage media. In particular, the instructions may be provided on a transitory or non-transitory computer-readable medium. When executed by a processor, the processor is configured to perform various methods described in the following embodiments. As such, these methods may be computer-implemented methods. In particular, the processor and the storage including the instructions may be incorporated in a vehicle. The vehicle may be an AV.

[0022] The following embodiments provide specific illustrative examples, which should not be construed as limiting, and the scope of protection is defined by the claims. Features of certain embodiments may be used in combination with features of other embodiments without extending the subject matter beyond the content of this disclosure.

[0023] As shown in FIG. 1 , the AV 10 may include a number of sensors 12. The sensors 12 may be mounted on the roof of the AV 10. The sensors 12 may be communicatively connected to a computer 14. The computer 14 may be on-board the AV 10. The computer 14 may include a processor 16 and a memory 18. The memory may include the non-transitory computer-readable medium described above. Alternatively, the non-transitory computer-readable medium may be located remotely and communicatively linked to the computer 14 via a cloud 20. The computer 14 may be communicatively linked to one or more actuators 22 to control the actuators 22 to move the AV 10. The actuators may include, for example, a motor, a braking system, a power steering system, etc.

[0024] The sensors 12 may include various sensor types. Examples of sensor types include LiDAR sensors, RADAR sensors, cameras, etc. Each sensor type may be referred to as a sensor modality. Each sensor type may record data associated with the sensor modality. For example, a LiDAR sensor may record LiDAR modality data.

[0025] The data may capture various scenes encountered by the AV 10. For example, the scene may be the visible scene surrounding the AV 10, which may include roads, buildings, weather, objects (other vehicles, pedestrians, animals, etc.), etc.

[0026] 2, the vehicle control system 30 includes a plurality of sensors 12. The plurality of sensors 12 includes an inertial measurement unit (IMU) 12_1, a camera 12_2, a LiDAR sensor 12_3, and a radar sensor 12_4. The vehicle control system 30 further includes an odometry module 32, a localization module 34, a perception module 36, and a tracking and prediction module 38. The system 30 further includes a high-level command module 40, a mission planning module 42, a mission execution module 44, a trajectory planning module 46, and a controller 48.

[0027] IMU 12_1 may include, for example, an accelerometer and a gyroscope.

[0028] Odometer module 32 may be configured to use inputs from multiple sensors 12 to estimate the position of AV 10 over time relative to a reference point (eg, a starting point).

[0029] The localization module 34 may be configured to use inputs from multiple sensors 12 to estimate the global pose of the AV 10 in its environment.

[0030] The perception module 36 may be configured to generate a model of the environment of the AV 10, including objects in the environment. The model may include segmentation maps, bounding boxes, cuboids, depth perception, etc., using inputs from multiple sensors 12.

[0031] The tracking and prediction module 38 may be configured to track objects (static and dynamic) in the scene based on the outputs from the odometry module 32, the localization module 34 and the recognition module 36.

[0032] High level command module 40 may be configured to receive input regarding a route, e.g., a destination, from a user and output commands to move AV 10 from a current location to the destination, e.g., from point A to point B.

[0033] The mission planning module 42 may include a map and, when cooperating with the high level command module 40, may be configured to construct a route to execute the route from the high level command module 40. The route is the driving distance that the AV 10 needs to travel to successfully traverse the route.

[0034] The trajectory planning module 46 may be configured to plan a trajectory for the AV 10 within a driving range defined by the mission execution module 44 taking into account the object's trajectory and position from the tracking and prediction module and the position of the AV 10 from the odometry module 32 and the position determination module 34.

[0035] Controller 48 may be configured to control one or more actuators of AV 10 to execute the trajectory from trajectory planning module 46.

[0036] The features of the vehicle control system 30 are shown in more detail in Figure 3. In particular, Figure 3 shows a flow chart for making object tracking and prediction decisions based on real-time, online sensor data, where online means that the AV 10 is driving in the area or environment.

[0037] The sensor 12, which includes a camera 12_2 and a radar 12_4, may further include a laser 12_5 and an imaging radar 12_6.

[0038] The images of the camera 12_2 may be used in combination with a shared map 52 to generate an intersection state 50. An intersection may be defined as an area with dynamically changing entry conditions, meaning whether the AV 10 can enter the controlled area. Examples of intersections include traffic lights, push button crosswalks, pedestrian crossings, yield signs, etc.

[0039] The camera 12_2, laser 12_5, and imaging radar 12_6 data may be used to generate a semantic vision 54. As used herein, the term semantic generally refers to any form of class or embedding. Semantic vision may include embeddings of objects seen in vision, camera images, modalities.

[0040] Laser 12_5 may be used to generate a semantic laser 56, which means an embedding associated with a laser point in the laser domain.

[0041] The laser may be used to generate a model-free laser 58. The term model-free is used herein generally to mean raw data points from a particular modality. A model-free laser may be a raw laser point.

[0042] The imaging radar 12_6 may be used to generate a semantic radar 60. The semantic radar 60 may be an embedding that defines the features captured by the imaging radar 12_6. As used herein, the term semantic may refer to a first representation that has been processed to result in a second representation that has semantic information.

[0043] The radar 12_5 may be used to generate a model-free radar 62. The model-free radar may include raw radar points.

[0044] The intersection conditions 50, semantic vision 54, semantic laser 56, model-free laser 58, semantic radar 60, and model-free radar 62 may be used to generate the tracking and prediction module 38.

[0045] Figure 4 shows a similar diagram to Figure 3, except that the inputs are simulated inputs. Thus, the tracking and prediction module 38 uses data from a simulated camera 12_2', a simulated laser 12_5', a simulated imaging radar 12_6', and a simulated radar 12_4'.

[0046] Referring now to FIG. 5, a flow chart illustrating a method for training an encoder to generate scenario encodings is shown.

[0047] The tracking and prediction module 38 determines the positions of static objects and the positions and trajectories of dynamic objects which are input to the scenario encoder 70 together with the object and free space data (obtained from the model-free data). The term free space refers to the driving range that avoids violations. The driving range may be in space-time. A violation represents an event that reduces the safety of the maneuver or reduces the perceived quality of the maneuver. A violation may include, for example, a collision and hard braking.

[0048] The scenario encoder 70 may receive further input from a field bug library 72, which is a database containing responses of the AV 10 to recorded online violations. The scenario encoder 70 may be an autoencoder. The autoencoder may be a variational autoencoder (VAE). A multivariate distribution may be extracted at the bottleneck of the VAE and used as the scenario encoding 74.

[0049] The scenario encodings 74 may be stored in a scenario library 76 .

[0050] A scenario comparator 78 may also be provided. The scenario comparator 78 may be configured to compare the scenario encoding 74 with other known scenarios contained in a scenario library 76. The scenario encoder 70 may, for example, be trained in a supervised manner to generate scenario encodings 74 that match known scenarios of the same scene.

[0051] FIG. 6 shows a flow chart illustrating how to encode additional scenarios online using a scenario encoder 70 trained in the manner outlined with reference to FIG.

[0052] The dynamic objects, static objects (or static maps), and free space observations are combined online into a bird's-eye view 80. The bird's-eye view 80 may be a representation of the scene as seen from a reference point. The reference point may be from above. Additionally, the bird's-eye view may be constructed entirely from spatiotemporal sensor measurements from the AV itself, e.g., a single "frame" LIDAR / RADAR scan or multiple scans integrated over time.

[0053] The bird's-eye view 80 may be used as input to a trained scenario encoder 70, which generates a scenario encoding 74. The scenario encoding may be compared to known scenarios from a scenario library 76 by a scenario comparator 78.

[0054] 7, the scenario comparator 78 is a matcher. A matcher may include multiple matching means. For example, a dynamic time warping (DTW) matcher 82 may be used to compare points in time from each encoding being compared.

[0055] The output from the DTW matcher 82 may also be input to a subcomponent matcher 84 that distributes the paired points from the DTW matcher 82 to multiple matching mechanisms, each of which calculates the distance between each point in the pair.

[0056] The matching means may include a Euclidean distance matcher 86, a trained distance matcher 88, and a custom matcher 90. The Euclidean distance matcher 86 calculates the Euclidean distance between points of the compared encodings. The trained distance matcher 88 may be a neural network trained by contrastive learning to calculate the distance between points of the compared encodings. The custom matcher may calculate the distance by comparing hybrid weighted similarities between different heuristics.

[0057] The output of each matching means is a match score 92A, 92B, 92C. The match scores may be numerical. A respective score weight 94A, 94B, 94C may be applied to each match score. The term "apply" may mean multiplication. A summation block 96 may add the weighted match scores to generate a final score 98. The score weights may vary. The score weights may be pre-determined. In other embodiments, the score weights may be learned. For example, the score weights may be adjusted using an optimization algorithm for training samples that have a desired final score for a particular encoding pair.

[0058] Referring now to FIG. 8, a flowchart outlining a computer-implemented method for determining steering constraints for an autonomous vehicle during a known scenario is shown in accordance with one or more embodiments.

[0059] The method includes a step 100 of retrieving a descriptor of the known scenario from a scenario library 76 stored in a database, the descriptor being a multivariate distribution from the bottleneck of a variational autoencoder, as described above, and decoding the descriptor by a decoder, which may be from the variational autoencoder used to generate the known scenario.

[0060] The method includes running 102 a simulation of a known scenario of the AV in a simulator (SIM). The SIM is run iteratively by applying different dynamic parameters to the AV. The dynamic parameters may be selected from a list including velocity, acceleration, deceleration, and jerk.

[0061] A first set of dynamic parameters may be applied to the AV in a known scenario. For example, the AV may travel at 20 miles per hour (mph) with zero acceleration. Step 104 checks for safe operation of the AV. Safe operation may be understood to mean no violations. A violation may be an event that negatively impacts a user's perception of safe driving. A violation may include a collision with an object in the scenario. A violation may further include a close encounter, such as the AV coming within a predetermined distance of another object in the scenario. These examples are not exhaustive and are provided for illustrative purposes only.

[0062] If there is a violation, the method includes step 106 of checking whether the dynamic parameters reach a limit. In other words, checking to determine whether there is no more adjustment to the dynamic parameters. For example, if the current speed is 20 mph, the speed may be decreased. The decrease (or increase) may be performed at predetermined intervals. The decrease may be performed at 5 mph intervals.

[0063] The method includes adjusting the dynamic parameters 108. In the above example, the new speed may be 15 mph.

[0064] In step 104, the safe operation of the AV is rechecked at the new speed, for example 15 mph.

[0065] The method continues by iteratively running the simulation by increasing and / or decreasing the dynamic parameters and checking for violations at each increase and / or decrease. The method continues to increase and / or decrease the dynamic parameters until there are no more dynamic parameter adjustments. For example, if the speed of the AV is 5 mph, the dynamic parameter speed may be exceeded.

[0066] If step 104 determines that operation is safe, then step 110 identifies a maximum dynamic parameter for the AV, below which no violation occurs and above which a violation occurs during the SIM.

[0067] The maximum dynamic parameters are stored as maneuvering constraints for the AV for the known scenario in a constraint library, step 112. The maneuvering constraints may include driving envelopes constructed in space-time (e.g., using a space-time graph) within which violations are predicted not to occur.

[0068] Alternatively, in step 106, in the event of no adjustment, a minimum dynamic parameter may be determined as a trigger to initiate a minimal risk manoeuvre (MRM). A minimal risk manoeuvre may include, for example, an emergency braking event or pulling onto the road shoulder. These MR examples are not exhaustive and are provided herein for illustrative purposes only.

[0069] In step 114, an MRM trigger is identified.

[0070] In step 112, the MRM is stored in a constraint library.

[0071] As shown in FIG. 9, in accordance with one or more embodiments, a computer-implemented method for controlling an autonomous vehicle is provided.

[0072] The method includes step 120, which involves encoding the current scenario using an encoder.

[0073] Step 122 obtains the encoding from the encoded scenario.

[0074] In step 124, the encoding of the current scenario is compared to known scenarios from a scenario library. The comparison may be performed by a matcher of Figure 7. Thus, determining the distance includes calculating the distance using multiple weighted matchers.

[0075] The comparison may be exhaustive in that it compares all known scenarios. In other embodiments, an unsupervised clustering model may be used to cluster the known scenarios in a scenario library to group similar scenarios together. In this way, fewer comparisons need to be made.

[0076] As mentioned above, the current scenario and each known scenario may be a descriptor that represents a multivariate distribution from the bottleneck of the variational autoencoder.

[0077] In step 126, the distance between the current scenario and the closest known scenario from a database of known scenarios (scenario library) is determined.

[0078] In step 128, the safety orchestrator is used to determine the maneuver to be performed by the AV. The safety orchestrator searches for maneuver constraints associated with the closest known scenario from the constraint library of FIG.

[0079] The safety orchestrator may proceed to step 130 and determine that if the distance is less than the first threshold, it is optimal to control the AV by steering the autonomous vehicle within steering constraints associated with the closest scenario. In other words, the steering constraints of the closest scenario may be used as steering constraints for the current scenario. The steering constraints may include driving ranges constructed in space-time within which violations are predicted not to occur.

[0080] The safety orchestrator may determine, in step 132, that if the distance exceeds a second threshold, it is optimal to control the AV by performing a minimum risk maneuver, the second threshold being greater than the first threshold. The minimum risk maneuver may include one or more of an emergency stop and a pull-over.

[0081] The safety orchestrator may proceed again to step 130 and determine that if the distance is between the first and second thresholds, it is optimal to interpolate the steering constraints associated with the closest scenario and steer the autonomous vehicle within the interpolated steering constraints.

[0082] Once the mode of execution (e.g., get from closest scenario, get by interpolation, or get steering constraints with MRM) has been defined, in step 134 the planner determines the control commands for the AV's actuators to execute the execution mode. At the same time, the watchdog monitors the operation of the panner. In certain circumstances, the watchdog may override the planner.

[0083] The foregoing embodiments are described to illustrate the subject matter of the present disclosure, but the features of the embodiments should not be construed as limiting the scope of protection, which, for the avoidance of doubt, is defined by the following claims.

Claims

1. 1. A computer-implemented method for controlling an autonomous vehicle, comprising: receiving current sensor data defining a current scenario; determining a distance between the current scenario and a closest scenario from a database of known scenarios; if the distance is less than a first threshold, steering the autonomous vehicle within steering constraints associated with the closest scenario; performing a minimal risk maneuver if the distance exceeds a second threshold that is greater than the first threshold; and and if the distance is between the first and second thresholds, interpolating steering constraints associated with the closest scenario and controlling the autonomous vehicle by steering the autonomous vehicle within the interpolated steering constraints.

2. The computer-implemented method of claim 1 , wherein the current scenario and each known scenario are descriptors that represent a multivariate distribution from a bottleneck of a variational autoencoder.

3. The computer-implemented method of claim 1 or claim 2, wherein determining the distance comprises calculating the distance using a plurality of weighted matchers.

4. 4. The computer-implemented method of claim 3, wherein the plurality of weighted matchers comprises one or more of a time warping matcher, a Euclidean distance matcher, a cosine distance matcher, and a trained matcher.

5. The computer-implemented method of claim 1 , wherein the steering constraints include driving ranges constructed in space-time within which no violations are predicted to occur.

6. The computer-implemented method of any one of claims 1 to 5, wherein the minimum risk maneuver includes one or more of an emergency stop and a pulling over.

7. 1. A computer-implemented method for determining steering constraints for an autonomous vehicle during a known scenario, comprising: running a simulation of a known scenario of the autonomous vehicle in a simulator; iteratively running the simulation by applying different dynamic parameters to the autonomous vehicle; identifying a maximum dynamic parameter of the autonomous vehicle from the different dynamic parameters, where if the maximum dynamic parameter is less than the maximum dynamic parameter, a violation does not occur during the simulation, and if the maximum dynamic parameter is exceeded, a violation occurs during the simulation; and and storing the maximum dynamic parameters in a constraint database as steering constraints for the autonomous vehicle for the known scenario.

8. Retrieving a descriptor of the known scenario from a database, the descriptor being a multivariate distribution from a bottleneck of a variational autoencoder; The computer-implemented method of claim 7 , further comprising: decoding the descriptors using a decoder from the variational autoencoder that generates known scenarios.

9. The step of iteratively performing a simulation includes: Increasing and / or decreasing said dynamic parameters; and checking for violations at every increment and / or decrement.

10. The computer-implemented method of claim 9 , further comprising continuing to increase and / or decrease the dynamic parameter until there are no more adjustments to the dynamic parameter.

11. 11. The computer-implemented method of claim 9 or claim 10, wherein the increasing and / or decreasing step comprises increasing and / or decreasing at predetermined intervals.

12. A computer-implemented method according to any one of claims 7 to 11, wherein the dynamic parameters are selected from the list comprising velocity, acceleration, deceleration and jerk.

13. The computer-implemented method of any one of claims 7 to 12, wherein the steering constraints include driving ranges constructed in space-time within which no violations are predicted to occur.

14. A transitory or non-transitory computer readable medium having stored thereon instructions which, when executed by a processor, cause the processor to perform the computer-implemented method of any one of claims 1 to 13.

Images