Method for determining potential trajectory when controlling robotic device
By using machine learning models and weighted factors to optimize potential trajectory sets, the diversity and safety issues of trajectory planning in autonomous vehicles are addressed, achieving robust and efficient trajectory planning.
Patent Information
- Application Number
- CN202511067486.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-08-02
- Filing Date
- 2025-07-31
- Publication Date
- 2026-02-03
AI Technical Summary
In trajectory planning for autonomous vehicles, existing technologies struggle to ensure the diversity of potential trajectories under diverse future scenarios and to prevent unnecessary trajectory diversity in single-mode future scenarios, thus affecting the safety and efficiency of planning.
By using machine learning models to identify potential trajectories and assigning weighting factors to each trajectory, the potential trajectory set is optimized to generate robust and safe control parameters by combining minimum mean distance error and road deviation metric.
It improves the diversity and safety of trajectory planning, reduces planning time, and ensures robust planning in diverse and single-mode futures.
Smart Images
Figure CN121448428A_ABST
Abstract
Description
BACKGROUND
[0001] When driving at least partially automatically (e.g. autonomously), a planner of a vehicle can plan a future behavior of the vehicle in dependence on a predefined number of potential future trajectories of other traffic participants (e.g. other vehicles, pedestrians, cyclists, etc.) and can control the vehicle accordingly. In this case, the quality of the planning (e.g. in terms of safety) can significantly depend on the selection of the potential future trajectories.
[0002] Filos et al.: “Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?” arXiv:2006.14911v2, 2020 (hereinafter referred to as reference [1]) describe an evaluation of potential trajectories by means of a set of expert probability models and a selection of a subset of potential trajectories in dependence on the evaluation. SUMMARY
[0003] The present disclosure relates to a method for controlling a robotic device, wherein potential trajectories (of e.g. traffic participants) are not (randomly) selected from a set of potential trajectories (as e.g. in reference [1]), but are determined directly in a way that optimizes a metric disclosed therein. Thereby, for example, a diversity of (selected) potential trajectories can be ensured in case of a diverse future (i.e. in case of multiple mutually different potential trajectories) and an unnecessary diversity of (selected) potential trajectories can be prevented in case of a unimodal (i.e. non-diverse) future. The potential trajectories determined in this way can guarantee a robust and safe planning by a planner. Thereby, the efficiency of the planning can be significantly improved. Since the planning is only performed for the selected potential trajectories and not for all potential trajectories, the runtime of the planning can also be reduced thereby.
[0004] Various aspects relate to a method for controlling a robotic device, the method comprising: determining a plurality of potential trajectories of an object (e.g. of another robotic device) in a surrounding environment of the robotic device in a manner that the one or more potential trajectories are determined by means of one or more machine learning models (e.g. a set of Bayesian neural networks), respectively, and that for each of the one or more potential trajectories a weighting factor assigned to the potential trajectory is determined; determining a plurality of weighted similarity values in a manner that for each of the plurality of potential trajectories a respective weighted similarity value is determined, wherein determining a weighted similarity value for a potential trajectory comprises: determining a similarity value representing a similarity between the potential trajectory and a set of suggested trajectories according to one or more similarity measures (e.g. distance measures); and determining the weighted similarity value in a manner that the (determined) similarity value is weighted according to the weighting factor assigned to the potential trajectory; summing the plurality of weighted similarity values into an error value; adapting (e.g. optimizing) the set of suggested trajectories in order to determine an adapted (e.g. optimized) set of suggested trajectories that results in a reduced (e.g. minimized) error value; generating control parameters for controlling the robotic device using the adapted set of suggested trajectories; and controlling the robotic device according to the control parameters.
[0005] In the following various embodiments are explained.
[0006] Example 1 is a method for controlling a robotic device as described above.
[0007] Example 2 is established according to example 1, wherein each of the one or more machine learning models has (a set of) (e.g. Bayesian) neural networks; and / or wherein the weighting factor assigned to a potential trajectory of the one or more potential trajectories represents an uncertainty of the potential trajectory.
[0008] In this way it is possible to consider uncertainties in terms of respective model parameters and respective architectures of the one or more machine learning models.
[0009] Example 3 is established according to example 1 or 2, wherein the plurality of trajectories is determined in a manner that the plurality of trajectories is determined by means of exactly one machine learning model.
[0010] The method disclosed herein allows to determine a predefined number of potential trajectories even in case the plurality of potential trajectories is predicted by means of a single machine learning model.
[0011] Example 4 is established according to one of examples 1 to 3, wherein the one or more similarity measures have a minimum average distance error (minADE) between the potential trajectory and the set of suggested trajectories.
[0012] Example 5 is according to one of the examples 1 to 4, wherein the robotic device is an at least partially automated vehicle, wherein the object is another traffic participant, and wherein the one or more similarity measures have an Abseits-der-Straβe measure that increases a similarity value when a proposed trajectory of the set of proposed trajectories is off the road.
[0013] As set out herein, a similarity measure can be used, or multiple similarity measures can also be combined. The inventors have recognized that the minADE as a similarity measure generally leads to more robust and safer planning. In the case of at least partially automated driving, (e.g. in combination with the minADE similarity measure), the Abseits-der-Straβe measure can lead to additional improvements, as trajectories that are off the road to other vehicles (i.e. when the object is another vehicle) are given less weight.
[0014] Example 6 is according to one of the examples 1 to 5, wherein the proposed trajectories of the (initial) set of proposed trajectories are (randomly) selected as a subset from the plurality of potential trajectories. Thereby, the optimization process can be accelerated.
[0015] Example 7 is a control device having one or more processors that are set up to implement a method according to one of the examples 1 to 6.
[0016] Example 8 is a robotic device (e.g. an at least partially automated vehicle) having a control device according to example 7.
[0017] Example 9 is a computer program having instructions that, when executed by a processor, cause the processor to perform a method according to one of the examples 1 to 6.
[0018] Example 10 is a computer readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to one of the examples 1 to 6. BRIEF DESCRIPTION OF DRAWINGS
[0019] In the drawings, like reference numerals generally refer to like parts throughout the various views. These drawings are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the application. In the following description, various aspects are described with reference to the following drawings, in which:
[0020] Figure 1 An at least partially automated vehicle according to various aspects is shown;
[0021] Figure 2A flowchart illustrating a method for controlling an at least partially automated vehicle according to various aspects is shown;
[0022] Figure 3A and Figure 3B The potential trajectory groups determined according to the method 200 for the traffic participants are shown respectively. DETAILED DESCRIPTION
[0023] The following detailed description relates to the accompanying drawings, which illustrate by way of example specific details and aspects of the present disclosure in which the invention can be practiced. Other aspects can be utilized and structural, logical, and electrical changes can be made without departing from the scope of the present invention. The various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure can be combined with one or more other aspects of the present disclosure, to form a new aspect of the present disclosure.
[0024] Various examples are described in more detail below.
[0025] Figure 1 An at least partially automated vehicle 100 according to various aspects is shown. Figure 1 The at least partially automated vehicle 100 shown in Fig. 1 and described herein is an exemplary computer-controlled device. Although various aspects of the computer-implemented method are described herein with reference to a vehicle 100, it is understood that this is for illustration purposes and that any other type of computer-controlled device can use the computer-implemented method, wherein the trajectories of objects in the surrounding environment of the computer-controlled device play a role, such as an industrial robot (e.g. in the form of a robot arm for moving, assembling or machining workpieces, for picking up containers, etc.), a manufacturing robot, a maintenance robot, a household robot (e.g. a cleaning robot, a mowing robot, etc.), a medical robot, etc.
[0026] For controlling the vehicle 100, the vehicle 100 can have a (vehicle) control device 102 which is set up for implementing the interaction of the vehicle 100 with its surrounding environment according to a control program. The term "control device" can be understood as any type of logic-implementing unit which may, for example, comprise a circuit and / or a processor which is capable of executing software, firmware or a combination thereof stored in a storage medium, and which is capable of issuing commands, for example to actuators in the present example. The control device may, for example, be set up by program code (e.g. software) for controlling the operation of a system (in the present example a robot).
[0027] In the present example, the control device 102 can have a computer 104 and a memory 106 storing code and data, the computer 104 controlling the vehicle 100 based on the code and data. According to various aspects, the control device 102 can control the vehicle 100 based on a control model 108 stored in the memory 106.
[0028] In order to be able to control a driving task of the vehicle 100, the control device 102 can use sensor data representative of a surrounding of the vehicle 100. To this end, the vehicle 100 can have one or more sensors 110, in each of which a respective sensor data representative of at least a part of the surrounding of the vehicle 100 can be provided. A sensor of the one or more sensors 110 can be, for example, an imaging sensor and / or a proximity sensor, like, for example, a video camera (e.g. a standard video camera, a digital video camera, an infrared video camera, a stereo video camera, etc.), a radar sensor, a lidar sensor, an ultrasonic sensor, etc. A sensor of the one or more sensors 110 can be set up to detect an image showing at least a part of the surrounding of the vehicle 100. The image can be an RGB image, an RGB-D image or a depth image (also referred to as D image). A depth image described herein can be any kind of image having depth information. A depth image can intuitively have 3-dimensional information about one or more objects in the surrounding of the vehicle 100. A depth image described herein can have, for example, a point cloud provided by a lidar sensor and / or a radar sensor. The depth image can be, for example, an image having depth information provided by a lidar sensor. It is understood that the vehicle 100 can furthermore have other sensors, like, for example, a global navigation satellite system (GNSS, e.g. global positioning system, GPS), a speed sensor, an acceleration sensor, an altitude measurement sensor, a gyroscope, etc., and that the control device 102 can also use sensor data provided by these other sensors for controlling the vehicle 100. The control device 102 can be set up to control the vehicle 100 in response to inputting the sensor data into the control model 108 based on an output of the control model 108.
[0029] The vehicle 100 can have a drive device 112 for driving the vehicle 100. The control device 102 can be set up to determine a control parameter for controlling the vehicle 100 using an output of the control model 108. The control device 102 can be set up to control an operation of the vehicle 100 according to the control parameter (e.g. by controlling the drive device 112 by means of a control signal).
[0030] The at least partially automated vehicle 100 can be an automated vehicle or an autonomous vehicle. The level of autonomy of the vehicle can be determined or specified by a SAE (Society of Automotive Engineers) level, e.g. as defined in SAE J3016. For example, the at least partially automated vehicle 100 can be a partially automated vehicle (according to SAE level 2), a highly automated vehicle (according to SAE level 3), a fully automated vehicle (according to SAE level 4) or an autonomous vehicle (according to SAE level 5).
[0031] The at least partially automated vehicle can generally take over the driving task autonomously. In order to ensure the safety of the occupants and other road users, e.g. cyclists, pedestrians, etc., it is necessary for such a system that takes over the driving task autonomously to be very safety-critical.
[0032] In order to plan the future behavior of the vehicle 100, the control model 108 can have a planner. For this purpose, the planner can be fed potential (future) trajectories of other road users. The potential trajectories of other road users can in this case be limited to a predefined number in order thus to limit the size of the search tree. The robustness and safety of the planning can in this case be related to the potential trajectories fed.
[0033] Figure 2 A flowchart of a (computer-implemented) method 200 for controlling an at least partially automated vehicle 100 according to various aspects is shown.
[0034] In the case of the method 200, the predefined number of potential trajectories is not (as in, for example, reference [1]) selected (randomly) from a set of potential trajectories, but is determined directly in such a way that an optimization measure is optimized. Thereby, for example, the diversity of the (selected) potential trajectories can be ensured in the case of a diverse future, i.e. in the case of a plurality of mutually different potential trajectories, and unnecessary diversity of the (selected) potential trajectories can be prevented in the case of a single-mode, i.e. non-diverse, future. The potential trajectories determined in this way can guarantee a robust and safe planning by the planner.
[0035] The method 200 can comprise (in 202) determining a plurality of potential trajectories of other road users in the surrounding environment of the vehicle 100 in such a way that one or more potential trajectories are determined by means of one or more machine learning models, respectively, and for each of the one or more potential trajectories a weighting factor assigned to the potential trajectory is determined.
[0036] Thereby, for example, it can be ensured in the planning in accordance with the potential trajectories of other road users that areas possibly occupied by other road users are not driven on.
[0037] The method 200 can comprise (in 204) determining a plurality of weighted similarity values in a manner that a respective weighted similarity value is determined for each of a plurality of potential trajectories. Determining a weighted similarity value for a potential trajectory can comprise determining a similarity value representing a similarity between the potential trajectory and the set of suggested trajectories according to one or more similarity measures (e.g. distance measures) and determining the weighted similarity value in a manner that the determined similarity value is weighted according to a weighting factor assigned to the potential trajectory.
[0038] The method 200 can comprise (in 206) summing the plurality of weighted similarity values to an error value.
[0039] The method 200 can comprise (in 208) adapting (e.g. optimizing) the set of suggested trajectories in order to determine an adapted (e.g. optimized) set of suggested trajectories that results in a reduced (e.g. minimized) error value.
[0040] The method 200 can comprise (in 210) generating control parameters for controlling the robotic device using the adapted set of suggested trajectories.
[0041] The method 200 can comprise (in 212) controlling the robotic device according to the control parameters.
[0042] Various aspects of the method 200 are described in more detail in the following.
[0043] In 202, one or more machine learning models f Each machine learning model f n (x) can determine one or more potential trajectories and can determine for each of the one or more potential trajectories a weighting factor assigned to the potential trajectory (i.e. Intuitively, it is possible to determine K*N potential trajectories (as the plurality of potential trajectories) with respectively assigned weighting factors in this way In order to determine the one or more potential trajectories The above described sensor data can for example be fed to the machine learning model f n (x). The potential trajectories described herein are predictions of future trajectories.
[0044] In this case, “N” can be any integer greater or equal to one and “K” can be any integer greater or equal to one as long as the factor K*N is greater than one. Intuitively, in the case of N=1, a single machine learning model f N (x) can predict a plurality of potential trajectories (where K > 1), and in case K = 1, the plurality of machine learning models (where N > 1) can each predict exactly one potential trajectory K.
[0045] The machine learning models may, for example, be (e.g. Bayesian) neural networks, and the weighting factors assigned to the potential trajectories determined by means of the (Bayesian) neural networks can represent an uncertainty of the potential trajectories.
[0046] The inventors have realized that the plurality of potential trajectories may be considered as weighted Dirac delta functions (also referred to as weighted Dirac distributions):
[0047]
[0048] This consideration enables a measure for directly determining a (well-optimized) set of potential trajectories (where 1 < S < K*N) as disclosed herein. This measure is thus optimized according to various aspects, such that a set of potential trajectories can be directly determined without selecting from the plurality of trajectories If a set of potential trajectories would be selected directly from the plurality of trajectories, possible scenarios might be neglected, which can be safety-endangering (e.g. can lead to an accident).
[0049] In 204, a respective weighted similarity value can then be determined for each potential trajectory of the one or more potential trajectories .
[0050] To this end, a similarity value can be determined, which represents a similarity between the potential trajectory and the set of suggested trajectories according to one or more similarity measures (e.g. distance measures). For illustration, the one or more similarity measures can be the minimum average distance error (minADE). In this case, the similarity value can be derived from:
[0051]
[0052] Since a similarity between each potential trajectory and the set of suggested trajectories is considered, a diversity of the (selected) potential trajectories can be ensured in case of a diverse scenario (i.e. in case of a plurality of mutually different, e.g. in mutually different directions, potential trajectories). If a plurality of trajectories For example having a left trajectory, a right trajectory and a straight trajectory, the method 200 can ensure a set of suggested trajectories Also having at least one left trajectory, at least one right trajectory and at least one straight trajectory.
[0053] The minimum average distance error minADE outputs the minimum deviation error between the set of potential trajectories and the set of suggested trajectories (to be optimized). It is understood that minADE is used for illustration and additionally or alternatively any other (optimizable) similarity measure can be used. For example, a (off-road) measure can additionally or alternatively be used, according to which the similarity value is increased when the potential trajectories of other traffic participants are off-road.
[0054] In this example, the smaller the similarity value, the greater the similarity between the potential trajectories and the set of suggested trajectories It is understood that this is exemplary and can be common for various similarity measures (e.g. distance measures), but it can also be the other way around.
[0055] A weighted similarity value can then be determined in that the (determined) similarity value is weighted according to a weighting factor assigned to the potential trajectories . Thus, the weighted similarity value can be derived from:
[0056]
[0057] In 206, the K*N weighted similarity values can then be summed into an error value. Thus, the error value can be derived from:
[0058]
[0059] In 208, the set of suggested trajectories may then be adapted (e.g. optimized) in order to reduce (e.g. minimize) the error value. This optimization can intuitively be performed according to:
[0060]
[0061] wherein is the set of potential trajectories to be optimized, and is the optimized set of potential trajectories. This optimization of the set of potential trajectories can be seen as an approximate empirical risk minimization under a distribution from multiple trajectories .
[0062] In Algorithm 1, the use of minADE as a similarity metric is exemplarily shown in steps 202 to 206 of method 200 for potential trajectory groups. The determination of the initial potential trajectory set. The trajectory can be randomly selected, and can be chosen from multiple trajectories. They are randomly selected, and / or may be selected using a machine learning model.
[0063]
[0064] Figure 3A The first traffic scenario 300A is shown, and Figure 3B The second traffic scenario 300B is shown. For traffic participants 302 who have already traversed a past trajectory 304 (shown as parallel shaded lines), multiple potential (future) trajectories 306 (shown as solid lines with solid endpoints) with their respective weighting factors are determined according to method 200. And based on these potential (future) trajectories, for S=5, a (optimized) set of potential (future) trajectories 308 is determined. (Seen as a dashed line with crosshairs at its endpoints). The dotted line 310 represents the actual trajectory (Grundwahrheitstrajektorie), that is, the trajectory that traffic participant 302 will traverse in the future. For example, in Figure 3B As can be seen intuitively, the (optimized) potential trajectory set 308... The trajectory is not selected from multiple potential trajectories 306 because the potential trajectory indicated by 308 is not consistent with any of the trajectories in multiple potential trajectories 306.
[0065] Although in the above discussion Figure 2 The scheme is applied to control vehicle 100, but it can generally be applied to determine control signals for controlling any technical system in scenarios where a limited number of potential (future) trajectories are at play, such as computer-controlled machines like robots, vehicles, household appliances, power tools, manufacturing machines, personal assistants, or access control systems. Depending on various aspects, the method for predicting the behavior of traffic participants may have steps 202 to 208 of method 200.
Claims
1. A method (200) for controlling a robotic device, the method (200) comprising: • determining (202) a plurality of potential trajectories (306) for an object (302) in a surrounding environment of the robotic device (100) in a manner that one or more machine learning models determine one or more potential trajectories, respectively, and determine for each of the one or more potential trajectories a weighting factor assigned to the potential trajectory; • determining (204) a plurality of weighted similarity values in a manner that for each of the plurality of potential trajectories (306) a respective weighted similarity value is determined, wherein determining a weighted similarity value for a potential trajectory comprises: o determining a similarity value representing a similarity between the potential trajectory and a set of suggested trajectories according to one or more similarity measures; and o determining the weighted similarity value in a manner that the similarity value is weighted according to the weighting factor assigned to the potential trajectory; • summing (206) the plurality of weighted similarity values into an error value; • adapting (208) the set of suggested trajectories in order to determine an adapted set of suggested trajectories (308) that results in a reduced error value; • generating (210) control parameters for controlling the robotic device using the adapted set of suggested trajectories (308); and • controlling (212) the robotic device (100) according to the control parameters.
2. The method (200) according to claim 1, wherein each of the one or more machine learning models has a Bayesian neural network; and / or wherein the weighting factor assigned to a potential trajectory of the one or more potential trajectories represents an uncertainty of the potential trajectory.
3. The method (200) according to claim 1 or 2, wherein the plurality of trajectories (306) is determined in a manner that the plurality of trajectories is determined by means of exactly one machine learning model.
4. The method (200) according to any one of claims 1 to 3, wherein the one or more similarity measures have a minimum average distance error between the potential trajectory and the set of suggested trajectories.
5. The method (200) according to any one of claims 1 to 4, wherein the robotic device (100) is an at least partially automated vehicle, wherein the object (302) is another traffic participant, and wherein the one or more similarity measures have a measure that increases a similarity value when a suggested trajectory of the set of suggested trajectories is in an off-road position.
6. The method (200) according to any one of claims 1 to 5, wherein suggested trajectories of the set of suggested trajectories are selected as a subset from the plurality of potential trajectories.
7. A control device (102) set up for implementing the method (200) according to any one of claims 1 to 6.
8. A robotic device (100) having the control device (102) according to claim 7.
9. A computer program having instructions which, when executed by a processor, cause the processor to carry out the method (200) according to any one of claims 1 to 6.
10. A computer readable medium storing instructions which, when executed by a processor, cause the processor to carry out the method (200) according to any one of claims 1 to 6.