Method for identifying potential trajectories in controlling a robotic device

Abstract

Various aspects relate to a method for controlling a robotic device.SOLUTION: Determining a plurality of potential trajectories of the object in the surroundings of the robotic device by determining one or more potential trajectories using one or more machine learning models, respectively, and determining weighting factors associated with the potential trajectories, wherein similarity values representing a similarity between the potential trajectories and the set of proposed trajectories according to at least one similarity metric are determined and weighted according to the weighting factors associated with the potential trajectories, thereby determining weighted similarity values, adjusting the set of proposed trajectories resulting in a reduction of a sum of the plurality of weighted similarity values, using the adjusted set of proposed trajectories to generate control parameters for controlling the robotic device, and controlling the robotic device according to the control parameters.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Conventional technology In at least partially automated (e.g., autonomous) driving, a planner of the vehicle can plan the future behavior of the vehicle based on a predetermined number of potential future trajectories of other road users (e.g., other vehicles, pedestrians, cyclists, etc.) and control the vehicle accordingly. In this case, the quality of the plan (e.g., with respect to safety) can significantly depend on the selection of the potential future trajectories. [Background technology]

[0002] "Filos et al., "Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?", arXiv:2006.14911v2, 2020" (hereafter referred to as Reference [1]) describes using an ensemble of expert probabilistic models to evaluate potential trajectories and selecting a subset of potential trajectories based on this evaluation. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] “Filos et al., “Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?”, arXiv:2006.14911v2, 2020.” Summary of the Invention [Problem to be solved by the invention]

[0004] Disclosure of the Invention This disclosure relates to a method for controlling a robotic device in which potential trajectories (e.g., of a road user) are identified directly by optimizing a metric disclosed herein, rather than being selected (randomly) from a set of potential trajectories (as in, e.g., Reference [1]). This can ensure diversity in the (selected) potential trajectories when the future is diverse (i.e., when there are many mutually distinct potential trajectories), and can prevent unnecessary diversity in the (selected) potential trajectories when the future is unimodal (i.e., not diverse). The potential trajectories identified in this manner can ensure robust and safe planning by the planner, which can significantly improve planning performance. This can also reduce the execution time of the plan, because planning is performed only for the selected potential trajectories, not for all potential trajectories. [Means for solving the problem]

[0005] Various aspects relate to a method for controlling a robotic device, the method including: identifying a plurality of potential trajectories of objects (e.g., other robotic devices) in a surrounding of the robotic device by identifying one or more potential trajectories using one or more machine learning models (e.g., an ensemble of Bayesian neural networks), respectively, and identifying, for each of the one or more potential trajectories, a weighting factor associated with the potential trajectory; and identifying a plurality of weighted similarity values ​​by identifying a respective weighted similarity value for each potential trajectory among the plurality of potential trajectories, wherein identifying a weighted similarity value for a potential trajectory corresponds to a correlation between the potential trajectory and the proposed trajectory. determining a similarity value representing a similarity between a set of paths according to one or more similarity metrics (e.g., distance metrics); determining a weighted similarity value by weighting the (determined) similarity values ​​according to weighting coefficients associated with the potential trajectories; summing the weighted similarity values ​​into a single error value; adjusting (e.g., optimizing) the set of proposed trajectories to identify an adjusted (e.g., optimized) set of proposed trajectories that results in a reduction (e.g., minimization) of the error value; generating control parameters for controlling the robotic device using the adjusted set of proposed trajectories; and controlling the robotic device according to the control parameters.

[0006] Various examples are described below.

[0007] Example 1 is a method for controlling a robotic device, as described above.

[0008] Example 2 is configured according to Example 1, wherein each machine learning model of the one or more machine learning models includes a (e.g., Bayesian) neural network (e.g., an ensemble thereof), and / or wherein a weighting coefficient associated with a potential trajectory of the one or more potential trajectories represents uncertainty for the potential trajectory.

[0009] In this way, uncertainty regarding each model parameter and each architecture of one or more machine learning models can be taken into account.

[0010] Example 3 can be configured according to example 1 or example 2, where the multiple trajectories are identified by using exactly one machine learning model to identify the multiple trajectories.

[0011] The methods disclosed herein allow for the identification of a predetermined number of potential trajectories, even when multiple potential trajectories are predicted using a single machine learning model.

[0012] Example 4 is configured according to any one of Examples 1 through 3, wherein the one or more similarity metrics comprises a minimum average distance error (minADE) between the potential trajectory and the set of proposed trajectories.

[0013] Example 5 is configured according to any one of Examples 1 to 4, wherein the robotic device is an at least partially automated vehicle, the object is another road user, and the one or more similarity metrics include an (off-road) metric that increases the similarity value if one proposed trajectory in the set of proposed trajectories is off-road.

[0014] As described herein, a single similarity metric may be used, or multiple similarity metrics may be combined. The inventors have recognized that a minimum average distance error (minADE) as a similarity metric generally results in more robust and safer plans. An (off-road) metric may provide additional improvement (e.g., relative to the minADE similarity metric) in the case of at least partially automated driving, because less weight is given to the trajectories of other vehicles located off-road (i.e., when the object is another vehicle).

[0015] Example 6 is configured according to any one of Examples 1 through 5, wherein the proposed trajectories of the (initial) set of proposed trajectories are (randomly) selected as a subset from a plurality of potential trajectories, which can accelerate the optimization process.

[0016] Example 7 is a control device comprising one or more processors configured to perform the method of any one of Examples 1 to 6.

[0017] Example 8 is a robotic device (eg, an at least partially automated vehicle) comprising a control device according to Example 7.

[0018] Example 9 is a computer program comprising instructions that, when executed by a processor, cause the processor to perform the method described in any one of Examples 1 to 6.

[0019] Example 10 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method of any one of Examples 1 through 6.

[0020] In the drawings, like reference numerals generally refer to like parts throughout the various views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings: [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 illustrates an at least partially automated vehicle in accordance with various aspects. [Figure 2] FIG. 1 illustrates a flowchart of a method for controlling an at least partially automated vehicle in accordance with various aspects. [Figure 3A] 2 shows a set of potential road user trajectories, each identified according to the method 200. FIG. [Figure 3B] 2 shows a set of potential road user trajectories, each identified according to the method 200. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0022] The following detailed description refers to the accompanying drawings, which show, by way of illustration, specific details and aspects of the present disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, or electrical changes may be made without departing from the scope of the present disclosure. Various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure may be combined with one or more other aspects of the present disclosure to form new aspects.

[0023] Various examples are described in more detail below.

[0024] FIG. 1 illustrates an at least partially automated vehicle 100 according to various embodiments. The at least partially automated vehicle 100 illustrated in FIG. 1 and specifically described herein for purposes of illustration is one exemplary computer-controlled device. While various aspects of the computer-implemented method are described herein with reference to the vehicle 100, it is understood that the vehicle 100 is used for purposes of illustration only, and that the computer-implemented method may be used by any other type of computer-controlled device in which the trajectory of an object around the computer-controlled device plays a significant role, such as, for example, an industrial robot (e.g., in the form of a robotic arm for moving, assembling, or machining a workpiece, retrieving a container, etc.), a manufacturing robot, a maintenance robot, a domestic robot (e.g., a cleaning robot, a lawnmower robot, etc.), a medical robot, etc.

[0025] To control the vehicle 100, the vehicle 100 may be equipped with a (vehicle) controller 102, which is configured to realize the interaction of the vehicle 100 with its surroundings according to a control program. The term "controller" may be understood as any kind of unit that implements logic, which may include circuits and / or processors that are capable of executing software, firmware, or a combination thereof, for example stored on a memory medium, and that are capable of issuing commands to actuators, for example, in this example. The controller may be configured, for example, to control the operation of a system, in this example, the operation of a robot, by means of program code (e.g., software).

[0026] In this example, the controller 102 may include a computer 104 and a memory 106 that stores code and data based on which the computer 104 controls the vehicle 100. According to various aspects, the controller 102 may control the vehicle 100 based on a control model 108 stored in the memory 106.

[0027] To be able to control the driving tasks of the vehicle 100, the control device 102 can use sensor data representative of the surroundings of the vehicle 100. For this purpose, the vehicle 100 may be equipped with one or more sensors 110, each of which can provide respective sensor data representative of at least a portion of the surroundings of the vehicle 100. For example, one of the one or more sensors 110 may be an imaging sensor and / or a proximity sensor, such as a camera (e.g., a standard camera, a digital camera, an infrared camera, a stereo camera, etc.), a radar sensor, a LIDAR sensor, an ultrasonic sensor, etc. One of the one or more sensors 110 may be configured to detect an image showing at least a portion of the surroundings of the vehicle 100. The image may be an RGB image, an RGB-D image, or a depth image (also referred to as a D-image). A depth image as described herein may be any type of image having depth information. For clarity, a depth image may include three-dimensional information about one or more objects in the surroundings of the vehicle 100. The depth image described herein may include a point cloud provided by, for example, a LIDAR sensor and / or a radar sensor. A depth image may be an image with depth information provided by, for example, a LIDAR sensor. It is understood that the vehicle 100 may further include other sensors, such as, for example, a Global Navigation Satellite System (GNSS, e.g., Global Positioning System GPS), a speed sensor, an acceleration sensor, an altimetry sensor, a gyroscope, etc., and the controller 102 may use sensor data provided by these other sensors to control the vehicle 100. The controller 102 may be configured to control the vehicle 100 based on the output of the control model 108 in response to the input of sensor data to the control model 108.

[0028] The vehicle 100 may include a drive unit 112 for driving the vehicle 100. The controller 102 may be configured to use the output of the control model 108 to identify control parameters for controlling the vehicle 100. The controller 102 may be configured to control the operation of the vehicle 100 in accordance with the control parameters (e.g., by controlling the drive unit 112 with a control signal).

[0029] The at least partially automated vehicle 100 may be an automated vehicle or an autonomous vehicle. The level of vehicle autonomy may be identified or designated by a Society of Automotive Engineers (SAE) level (e.g., as defined in SAE J3016). For example, the at least partially automated vehicle 100 may be a partially automated vehicle (per SAE Level 2), a highly automated vehicle (per SAE Level 3), a fully automated vehicle (per SAE Level 4), or an autonomous vehicle (per SAE Level 5).

[0030] At least partially automated vehicles are generally capable of autonomously assuming driving tasks, and the systems that autonomously assume these tasks need to be highly safety-critical to ensure the safety of occupants and other road users (e.g., cyclists, pedestrians, etc.).

[0031] To plan the future behavior of the vehicle 100, the control model 108 may include a planner. For this purpose, potential (future) trajectories of other road users may be fed to the planner. In this case, the potential trajectories of other road users may be limited to a predetermined number in order to limit the size of the search tree. In this case, the robustness and safety of the plan may depend on the potential trajectories fed.

[0032] FIG. 2 illustrates a flowchart of a (computer-implemented) method 200 for controlling an at least partially automated vehicle 100 according to various embodiments.

[0033] In method 200, a predetermined number of potential trajectories are directly identified by optimizing a metric, rather than being selected (randomly) from a set of potential trajectories (as in, for example, Reference [1]). This ensures diversity in the (selected) potential trajectories when the future is diverse (i.e., when there are many mutually different potential trajectories), and prevents unnecessary diversity in the (selected) potential trajectories when the future is unimodal (i.e., not diverse). The potential trajectories identified in this way can ensure robust and safe planning by the planner.

[0034] Method 200 may include identifying (at 202) a plurality of potential trajectories of other road users around vehicle 100 by using one or more machine learning models to identify one or more potential trajectories, respectively, and, for each of the one or more potential trajectories, identifying a weighting factor associated with the potential trajectory.

[0035] This makes it possible to ensure, for example, that areas potentially occupied by other road users are not traveled on, based on their potential trajectories at the time of planning.

[0036] Method 200 may include identifying (at 204) a plurality of weighted similarity values ​​by identifying a respective weighted similarity value for each potential trajectory of the plurality of potential trajectories. Identifying a weighted similarity value for a potential trajectory may include identifying a similarity value that represents a similarity between the potential trajectory and the set of proposed trajectories according to one or more similarity metrics (e.g., distance metrics), and weighting the (identified) similarity values ​​according to weighting coefficients associated with the potential trajectory to identify the weighted similarity value.

[0037] The method 200 may include summing (at 206) the weighted similarity values ​​into a single error value.

[0038] The method 200 may include adjusting (e.g., optimizing) (at 208) the set of proposed trajectories to identify an adjusted (e.g., optimized) set of proposed trajectories that results in a reduction (e.g., minimization) of the error value.

[0039] The method 200 may include generating (at 210) control parameters for controlling the robotic device using the refined set of proposed trajectories.

[0040] The method 200 may include controlling (at 212) the robotic device according to the control parameters.

[0041] Various aspects of the method 200 are described in more detail below.

[0042] 202, one or more machine learning models

number

number

number

number

number

number

number

number

[0043] In this case, "N" may be any integer greater than or equal to 1, and "K" may be any integer greater than or equal to 1, as long as the coefficient K*N is greater than 1. For simplicity, when N=1, only one machine learning model f N (x) is a set of multiple potential orbits

number

Number

[0044] For example, the machine learning model may well be a (e.g., Bayesian) neural network, and the weighting coefficients associated with the potential trajectories identified using this (Bayesian) neural network can represent the uncertainty of the potential trajectories.

[0045] The inventors of the present invention

Number

Number

[0046] By regarding it in this way, the (optimized) set of potential trajectories

Number

Number

Number

[0047] Then, at 204, one or more potential trajectories are generated.

number

number

[0048] For this reason, potential trajectories

number

number

number

[0049] Each potential trajectory

number

number

number

number

[0050] The minimum average distance error (minADE) is the minimum average distance error (MAD) of the potential trajectory.

number

number

number

[0051] In this example, the potential trajectories

number

number

[0052] The (identified) similarity values ​​are then applied to the potential trajectories.

number

number

number

[0053] The K*N weighted similarity values ​​may then be summed into a single error value at 206, i.e., the error value is:

number

[0054] Then, at 208, the set of proposed trajectories is calculated to reduce (e.g., minimize) the error value.

number

number

[0033] In this case,

number

number

number

[0055] A set of potential trajectories according to steps 202-206 of method 200

number

number

number

[0056] FIG. 3A illustrates a first traffic scenario 300A, and FIG. 3B illustrates a second traffic scenario 300B, in which road users 302 who have followed a past trajectory 304 (shown as horizontal hatching) are assigned corresponding weighting factors according to method 200.

number

number

number

[0057] 2 is applied to control vehicle 100, but in general, the approach of FIG. 2 may be applied to identify control signals for controlling any technological system, such as a robot, a vehicle, a home appliance, a power tool, a manufacturing machine, a personal assistant, or a computer-controlled machine, such as an access control system, in scenarios where a limited amount of potential (future) trajectories plays a key role. According to various aspects, a method for predicting road user behavior may include steps 202 to 208 of method 200.

Claims

1. A method (200) for controlling a robotic device, comprising: The method (200) comprises: Identifying (202) a plurality of potential trajectories (306) for an object (302) around the robotic device (100) by using one or more machine learning models to identify one or more potential trajectories, respectively, and identifying, for each of the one or more potential trajectories, a weighting factor associated with the potential trajectory; identifying (204) a plurality of weighted similarity values ​​by identifying a respective weighted similarity value for each potential trajectory of the plurality of potential trajectories (306), wherein identifying one weighted similarity value for each potential trajectory includes: - determining a similarity value representing the similarity between said potential trajectory and a set of proposed trajectories according to one or more similarity metrics; weighting the similarity values ​​according to the weighting factors associated with the potential trajectories to determine the weighted similarity values; (204) Summing the weighted similarity values ​​into a single error value (206); adjusting (208) the set of proposed trajectories to identify an adjusted set of proposed trajectories (308) that results in a reduction in the error value; generating (210) control parameters for controlling the robotic device using the refined set of proposed trajectories (308); controlling (212) the robotic device (100) in accordance with the control parameters; A method (200) comprising:

2. each machine learning model of the one or more machine learning models comprises a Bayesian neural network; and / or a weighting factor associated with a potential trajectory of the one or more potential trajectories representing the uncertainty of that potential trajectory; The method (200) of claim 1.

3. Identifying the plurality of trajectories (306) by identifying the plurality of trajectories using exactly one machine learning model.

3. The method (200) of claim 1 or 2.

4. the one or more similarity metrics have a minimum average distance error between the potential trajectory and the set of proposed trajectories; The method (200) of any one of claims 1 to 3.

5. the robotic device (100) is an at least partially automated vehicle; said object (302) being another road user; the one or more similarity metrics include a metric that increases a similarity value if a proposed trajectory of the set of proposed trajectories is off-road; The method (200) of any one of claims 1 to 4.

6. the proposed trajectories of the set of proposed trajectories are selected as a subset from the plurality of potential trajectories. The method (200) of any one of claims 1 to 5.

7. A control device (102) configured to implement the method (200) of any one of claims 1 to 6.

8. A robotic device (100) comprising a control device (102) according to claim 7.

9. A computer program comprising instructions which, when executed by a processor, cause the processor to perform the method (200) of any one of claims 1 to 6.

10. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method (200) of any one of claims 1 to 6.

Images