System and method for controlling autonomous vehicle in uncertain environment

Abstract

The present disclosure provides a controller for controlling an ego vehicle in an uncertain environment. The controller is caused to acquire knowledge of the environment from measurements associated with the self-sensor. The measurement is based on a state of the ego-vehicle and sensing instructions associated with operation of a control sensor. The controller is further caused to estimate a state of the environment including an uncertainty of a state of at least one moving object or obstacle in the environment. Further, a sequence of control inputs is determined by multivariate and multi-stage stochastic constraint optimization that solves a model of motion of the ego-vehicle. The controller is then caused to control the ego-vehicle and the sensor based on the sequence of control inputs and the sequence of sensing instructions.

Description

Technical Field

[0001] The present disclosure relates generally to control strategies for autonomous vehicles, and more particularly to control operations of autonomous vehicles in uncertain environments. Background Art

[0002] Autonomous vehicles, such as automated vehicles, unmanned aerial vehicles (UAVs) or drones, autonomous ground vehicles, warehouse mobile robots, etc., typically operate in an environment that is only approximately known. These vehicles must achieve one or more specified control objectives that need to be completed while avoiding unsafe events that may result from the vehicle's operation in the environment. For example, one such unsafe event is the vehicle colliding with an obstacle or leaving a specially marked area or safe navigation area. This is particularly challenging for the following reasons: the environment is not yet exactly known due to the lack of prior information about the environment and because the environment changes over time (for example, other vehicles on the road that the autonomous vehicle travels are also moving).

[0003] To this end, information about the environment can be acquired through sensing using sensors in the vehicle itself or in the environment connected to the vehicle through a communication network, which increases the knowledge of the environment. Although this knowledge is increased compared to the knowledge without sensors, this knowledge currently only provides approximate knowledge due to sensor noise and the limited capabilities of the sensors (such as in terms of their resolution, range, etc.). In addition, sensors can only acquire information about the current environment of the vehicle. For example, one goal of an autonomous vehicle is to reach its target destination without colliding with other vehicles while staying within the driving road by staying within the appropriate lane markings. The positions of other cars and road markings are not known a priori, but can be acquired by sensors such as radar, sonar, lidar, etc. (as long as different objects are within the range of the sensor and within the line of sight of the sensor). These sensors can be located inside or outside the autonomous vehicle. For example, the sensors can be arranged on the vehicle or on a roadside unit (RSU).

[0004] In another example, a warehouse robot transports certain parts to a workbench while staying in designated aisles to avoid endangering human workers and at the same time avoiding collisions with other robots and materials in the warehouse. As human workers and materials move through the warehouse, their positions change in unknown ways, but this can be acquired from sensors carried on the robot (such as lidar and sonar) (as long as the different objects are within the range and line of sight of the sensor), or can even be relayed to the robot from sensors placed in the environment (as long as the robot is within the communication range of the sensor).

[0005] Those skilled in the art will recognize that the amount of information obtained about a particular object in the environment depends on the amount of sensing that object is subjected to. That is, the more an object is observed using sensors, the more information about the object is obtained, that is, the more knowledge of the state of the object is obtained. Therefore, the acquisition of information about objects in the environment for the purpose of increasing knowledge thereof is a dynamic process.

[0006] On the other hand, the amount of information about the environment acquired by the sensor generally depends on how the object acquires this knowledge, or how the observer is operated. For example, if the observer is an autonomous vehicle, the object can only be properly discovered and observed if the object is within the observation range of the sensor used to observe the object and there is an unobstructed line of sight between the sensor placed on the autonomous vehicle and the object. Therefore, if the autonomous vehicle is operated so that the object stays more within the range and line of sight of the sensor, the amount of information acquired about the object will be greater and the knowledge of the state of the object will be more precise. In addition, the amount of information increases while the distance decreases, so if the autonomous vehicle is controlled to be closer to the object, more information can be obtained. However, there may be some risks in getting too close to the assumed position of an object (e.g., an obstacle) before sufficient knowledge of the actual position of the object is available.

[0007] To this end, if the motion of the autonomous vehicle is determined based only on available knowledge, the need to avoid unsafe events caused by a partially unknown environment may cause the motion of the autonomous vehicle to be overly cautious and stay on the most known path, which may result in performance degradation, e.g., longer time to reach the target, use of more energy, etc. On the other hand, if the motion of the autonomous vehicle is determined to obtain the most information about the environment, the motion of the autonomous vehicle may deviate significantly from the motion required to achieve the control goal, resulting in overall performance degradation, e.g., failure to reach the target, reaching the target with significant delay. Summary of the invention

[0008] Technical issues

[0009] Another problem in this motion planning scenario is that in order to plan the motion of the autonomous vehicle, not only the motion of the autonomous vehicle itself needs to be predicted, but also the information acquired based on that motion. However, predicting the acquired information is challenging because it depends on how the environment will change during the prediction period. This in itself is difficult to do.

[0010] Some prior art solutions known in the art disclose a method for controlling an autonomous vehicle in an uncertain environment discovered through sensing, where the operation of the autonomous vehicle affects the sensing and predicts the amount of future uncertainty of the environment based on the sensing actions. However, the prior art does not disclose how to incorporate predictions of future measurements into the decision making problem, which leads to potentially unsafe future operations or the need to choose conservative plans that reduce performance. In addition, the prior art does not disclose a method that can be used to plan the motion of the autonomous vehicle because the prior art only includes convex linear constraints, while the autonomous vehicle needs to avoid collisions with obstacles, which is known to result in non-convex constraints.

[0011] Therefore, there is a need for a control method that determines the optimal operation of an autonomous vehicle while avoiding collisions and other unsafe events resulting from interaction with an uncertain but sensed environment.

[0012] Technical solution to the problem

[0013] To this end, some embodiments provide a technical solution in which future information acquired about the environment is predicted to plan future movements of an autonomous vehicle that is safe and achieves high performance.

[0014] Some embodiments contemplate a controller for controlling an autonomous vehicle operating in a partially unknown environment discovered through sensing. The control objective is to enable the autonomous vehicle to achieve a given goal, such as reaching a specific state or tracking a given reference signal, while ensuring the safety of the autonomous vehicle with respect to the operational requirements and its interaction with the environment.

[0015] Examples of such autonomous vehicles include self-driving cars, as well as aerospace vehicles and mobile or manipulating robots, for which models of motion and dynamics are available, but which operate in an environment that is not completely known. For example, the uncertainty of the environment may be due to the presence of other actors, such as other cars or people, whose positions and movements are not known a priori, but can be discovered using sensors such as cameras, lidars, or radars. Safety is ensured by controlling the autonomous vehicle so that its movements conform to operating specifications, such as limits on speed, acceleration, and force. Safe interaction with the environment is ensured by controlling the autonomous vehicle so that it does not collide with other actors (such as other cars or people) and remains in an area suitable for its stay (such as an appropriate lane on a road or a predefined workspace). Since the knowledge of other actors and possibly also the knowledge of the area of ​​movement is incomplete, ensuring safe interaction may not be completely certain, but knowledge about these can be increased by obtaining more information from sensors, thereby ensuring safe interaction.

[0016] For an autonomous vehicle operating according to the above definitions and examples, the commands issued by control depend on the amount of knowledge available in the environment. For example, to avoid undetected risks, control may want to avoid areas where there is no knowledge. On the other hand, sensing also depends on how control operates the autonomous vehicle through its commands. For example, the distance of the autonomous vehicle from an area determines the extent to which the area is sensed. As another example, the controller can decide in real time the amount of processing done on the data sensed on a certain area, which can increase or decrease the amount of information extracted from the sensor. Therefore, there is a circular interdependency between sensing and control. However, the overall goal of the controller is to achieve a specific control goal for the autonomous vehicle. Therefore, the sensing goal is secondary to the control goal because it does not itself motivate the operation of the autonomous vehicle, although it can affect it.

[0017] In some known solutions existing in the art, the effect of sensing on the uncertainty in the environment has been taken into account, for example, represented by the covariance of the position estimate around a mean value. However, these known solutions do not disclose how to interpret the effect of sensing on the prediction of information about the environment, i.e., the mean around which the covariance is centered.

[0018] To this end, some embodiments are based on the recognition that in order to achieve control objectives with high performance in uncertain environments, it is not sufficient to simply predict how the information obtained from sensing affects the uncertainty (covariance) about the estimate of the environment, but it is also necessary to predict specific measurements (means) obtained about the environment that will affect the estimated information about the environment itself.

[0019] Some embodiments are further based on the recognition that the state of the environment can be represented by a random vector with a first moment and a second moment (mean and covariance). In addition, by predicting the evolution of the estimated covariance using predictive control actions of a model of the vehicle's impact estimation algorithm based on sensor data, the evolution of uncertainty about the environment estimate can be predicted.

[0020] Some embodiments are further based on the recognition that estimates of the environment can also be predicted by predicting the mean of the measurements with their own probabilities according to a multi-stage approach in which the mean is predicted for each admissible behavior of other actors in the environment. This results in a set of possible trajectories for the means of measurements that will be taken about the environment, and therefore, a set of possible trajectories for estimates of the environment's actors, according to the different behaviors that the environment may exhibit.

[0021] Some embodiments are further based on the recognition that by embedding multi-stage predictions of measurements about an environment into a multi-stage optimization problem, control commands for an autonomous vehicle can be obtained that take into account the effects of sensing uncertainty about the environment, as well as the effects of future measurements on estimates of the environment, and the possible behaviors of actors contained in the environment.

[0022] In addition, some embodiments are based on the recognition that, in order to provide a general collision avoidance behavior for an autonomous vehicle, it is necessary to convexify the non-convex constraints for collision avoidance while preserving its uncertainty information. This can be achieved by determining a hyperplane that separates the space occupied by obstacles from the free space that the autonomous vehicle can occupy, and then combining the separating hyperplane with the uncertainty of the environment to obtain a probabilistic constraint that provides collision avoidance guarantees with a user-defined confidence level.

[0023] To this end, some embodiments disclose a method for controlling an autonomous vehicle in an environment that is not known a priori but discovered through sensing, wherein the control method takes into account the effects of actions taken by the autonomous vehicle on the sensing, and takes into account the effects of both uncertainty in the sensing's knowledge of the surrounding environment and predictions of the behavior of actors in the surrounding environment.

[0024] The method implements an interdependent but unbalanced control and sensing, where control is the primary objective and sensing is a secondary objective, while taking into account the effect of sensing on environmental uncertainty and the effect of sensing on the prediction of the behavior of actors in the environment. This is achieved by computing control actions as the solution of a multi-stage multivariable optimization problem, where each stage corresponds to a future time step at which the autonomous vehicle can be controlled, and the elements in a stage correspond to different measurements associated with different discrete actions performed by the actors in the environment at each future time step.

[0025] Some embodiments predict different measurements of actors in the environment in a multi-stage optimization problem and thereby determine control of the autonomous vehicle based on such predicted measurements.

[0026] Some embodiments first predict the behavior of actors in the environment, then predict measurements based on each possible behavior, and thus determine control of the autonomous vehicle based on this prediction of the behavior of actors in the environment as seen based on the measurements of actors in the environment.

[0027] Some embodiments include probabilistic constraints in the multi-stage problem, where the uncertainty in such probabilistic constraints is the predicted reliability of the estimate of the environment. The reliability of the estimate of the environment is calculated using an estimator model that can be built based on dynamic equations or an autonomous vehicle learning model.

[0028] To this end, some embodiments include a convexification of a probabilistic constraint for avoiding mobile actors (e.g., vehicles or people), obtained by constructing a separating hyperplane for the region where obstacles are predicted to be located at any future time based on predictions measured in a multi-stage optimization problem, and combining the separating hyperplane with the estimated uncertainty of the prediction to obtain a probabilistic convex constraint.

[0029] According to one embodiment, a controller for controlling an ego vehicle in an environment surrounding the ego vehicle and including at least one moving object is provided. The controller includes: at least one processor; and a memory having instructions stored thereon, which, when executed by the at least one processor, cause the controller to acquire knowledge of the environment from measurements of at least one sensor of the ego vehicle that senses the environment. The measurements of the at least one sensor are based on at least one of: a state of the ego vehicle; and sensing instructions associated with controlling the operation of the sensor. The controller is further configured to process the acquired environmental knowledge to estimate a state of the environment, the state of the environment including a state of the at least one moving object, and the uncertainty of the state of the environment including an uncertainty of the state of the at least one moving object. The controller is further configured to jointly determine a sequence of control inputs for controlling the ego vehicle, the sequence of control inputs defining a state trajectory of the ego vehicle; and a sequence of sensing instructions to the sensors that define desired knowledge of the environment. The sequence of control inputs is determined by solving a multi-stage stochastic constrained optimization of a motion model of the ego vehicle, the motion model associating a state trajectory with a sequence of control inputs, wherein the sequence of control inputs is subject to chance constraints on the control inputs of the ego vehicle and the admissible values ​​of the state of the ego vehicle defined by the desired knowledge of the environment resulting from at least some possible transitions of the state of the at least one mobile object in different stages of the stochastic constrained optimization and the corresponding uncertainty of the transition states of the mobile objects in the acquired knowledge. The controller is further configured to control the ego vehicle and the sensor based on the sequence of control inputs and the sequence of sensing instructions. The controller is also configured to predict possible future measurements of one or more other mobile objects in the environment; and determine possible future transitions of the state of the at least one mobile object using the predicted possible future measurements of the one or more other mobile objects.

[0030] In some implementations, multivariable and multi-stage stochastic constrained optimization is achieved by conditioning the future transitions of the ego vehicle on predicted possible future measurements of one or more other mobile objects in the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] [Figure 1A ]

[0032] Figure 1A The overall architecture of a controller for controlling an ego vehicle operating in an environment according to an embodiment of the present disclosure is shown.

[0033] [ Figure 1B ]

[0034] Figure 1B A schematic diagram of joint multivariable optimization according to an embodiment of the present disclosure is shown.

[0035] [ Figure 1C ]

[0036] Figure 1C A schematic diagram showing constraints used in unbalanced multivariable constrained optimization according to an embodiment of the present disclosure is shown.

[0037] [ Figure 1D ]

[0038] Figure 1D A schematic diagram showing a principle of a cost function optimized by unbalanced multivariable constrained optimization according to an embodiment of the present disclosure is shown.

[0039] [ Figure 2A ]

[0040] Figure 2A is a block diagram of a method for multi-stage constrained multivariable optimization performed to achieve a joint but unbalanced optimization using predictions of possible future measurements in accordance with an embodiment of the present disclosure.

[0041] [ Figure 2B ]

[0042] Figure 2B It shows that the embodiment according to the present disclosure is implemented Figure 2A A block diagram of the various components of the method.

[0043] [ Figure 3A ]

[0044] Figure 3A An example use case of a controller for controlling an ego vehicle operating in an environment discovered through sensing in accordance with an embodiment of the present disclosure is shown.

[0045] [ Figure 3B ]

[0046] Figure 3B A block diagram of a controller and computing system for controlling an ego vehicle according to an embodiment of the present disclosure is shown.

[0047] [ Figure 4A ]

[0048] Figure 4A An example scenario of the operation of an ego vehicle using a controller in a lane change scenario according to an embodiment of the present disclosure is shown.

[0049] [ Figure 4B ]

[0050] Figure 4B A schematic diagram showing the results of the operation of the controller according to some embodiments of the present disclosure when the controller operates the vehicle to maintain the current lane or change lanes to the left, and if another vehicle ahead does not change lanes, maintaining the current lane seems to be preferable.

[0051] [ Figure 4C ]

[0052] Figure 4C A schematic diagram showing the operating results of the controller according to some embodiments of the present disclosure when the controller operates the vehicle to maintain the current lane or change lanes to the left, and maintaining the current lane seems to be preferred, but another vehicle ahead changes lanes.

[0053] [ Figure 5A ]

[0054] Figure 5A A schematic diagram showing possible results of operation of a controller according to some embodiments of the present invention when the controller operates the ego vehicle to remain in the current lane or change lanes to the left, and another vehicle may remain in its current lane or change lanes to the right at two different times.

[0055] [ Figure 5B ]

[0056] Figure 5B A schematic diagram showing the results of the operation of a controller according to some embodiments of the present disclosure when the controller operates the ego vehicle to remain in the current lane and another vehicle remains in its current lane.

[0057] [ Figure 5C ]

[0058] Figure 5C A schematic diagram showing the result of the operation of the controller according to some embodiments of the present disclosure when the controller operates the ego vehicle to change lanes to the left while another vehicle changes lanes to the right.

[0059] [ Fig. 6A ]

[0060] Fig. 6A A schematic diagram showing the estimated uncertainty in a controller when the controller does not adjust decisions with respect to possible predicted future measurements.

[0061] [ Figure 6B ]

[0062] Figure 6B A schematic diagram showing the estimated uncertainty in a controller according to some embodiments of the present disclosure as the controller adjusts decisions for possible predicted future measurements.

[0063] [ Fig. 7A ]

[0064] Fig. 7A A block diagram of a method executed by a controller for controlling a machine operating in an environment discovered through sensing according to an embodiment of the present disclosure is shown.

[0065] [ Figure 7B ]

[0066] Figure 7B A flow chart of an iterative operation method executed by a controller according to an embodiment of the present disclosure is shown.

[0067] [ Fig. 8A ]

[0068] Fig. 8A A schematic diagram showing a grid of equally spaced possible predicted future measurements and corresponding quantized uncertainties applied by a controller according to an embodiment of the present disclosure.

[0069] [ Figure 8B ]

[0070] Figure 8B A schematic diagram illustrating determination of a grid of possible predicted future measurements and determination of corresponding quantized uncertainties applied by a controller according to an embodiment of the present disclosure, wherein the possible predicted future measurements have equal probability.

[0071] [ Fig. 9 ]

[0072] Fig. 9 A schematic diagram showing convexification of collision avoidance constraints according to an embodiment of the present disclosure is shown.

[0073] [ Fig.10 ]

[0074] Fig.10 A scenario tree including a controller for constructing an unanticipated constraint for an optimization problem in the presence of adjustments for possible predicted future measurements is shown according to an embodiment of the present disclosure.

[0075] [ Fig.11A ]

[0076] Fig.11AA scenario tree according to an embodiment of the present disclosure is shown, which includes a controller for constructing unexpected constraints for an optimization problem in the presence of adjustments for possible predicted future measurements, wherein the adjustments for possible predicted future measurements are only made along branch horizons that are shorter than the prediction horizon.

[0077] [ Fig. 11B ]

[0078] Fig. 11B A scenario tree according to an embodiment of the present disclosure is shown, which includes a controller for constructing unexpected constraints for an optimization problem in the presence of adjustments for possible predicted future measurements, wherein the adjustments for possible predicted future measurements are only made along branch horizons that are shorter than the prediction horizon.

[0079] [ Fig.12 ]

[0080] Fig.12 The relationship between possible behaviors of other vehicles' future behaviors and a scenario tree used by a controller to construct an optimization problem when possible predicted future measurements for adjustment are obtained from possible future behaviors of other vehicles according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0081] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other embodiments, the devices and methods are shown only in block diagram form to avoid obscuring the present disclosure.

[0082] When used in this specification and claims, the terms "for example," "for example," and "such as," and the verbs "include," "have," "comprises," and their other verb forms, when used in conjunction with a list of one or more components or other items, are understood to be open-ended, meaning that the list should not be considered to exclude other additional components or items. The term "based on" means based at least in part on. In addition, it should be understood that the words and terminology used herein are for descriptive purposes and should not be considered limiting. Any headings used in this specification are for convenience only and have no legal or limiting effect.

[0083] The present disclosure provides systems and methods for controlling an object such as an ego vehicle, which may be an autonomous vehicle, in an uncertain environment that is not known a priori but discovered through sensing. The systems and methods disclosed herein provide a multi-stage perception opportunity constrained model predictive control (PAC-MPC) concept that does not require overly conservative estimates of measurement prediction error covariance. For example, the systems and methods disclosed herein can be used in a controller that is configured to control an automated vehicle in situations where obstacles and road boundaries are uncertain and are sensed by variable precision sensors subject to an overall sensing budget constraint and where scenarios are generated based on possible future behaviors of obstacles.

[0084] There is often an interdependent relationship between perception and control: perception of the environment depends on how the ego vehicle is controlled, and control of the ego vehicle depends on the environmental information obtained by the vehicle. The systems and methods disclosed herein are able to exploit this interdependency and provide a solution for controlling objects in uncertain environments in a robust, efficient, and accurate manner. In addition, the implementation of the solution is also based on considering the computational constraints of sensing and prediction, thereby providing a computationally efficient solution for MPC in controllers designed for the task. The systems and methods are also applicable to machines operating in uncertain environments without departing from the scope of the present disclosure, where such machines may include robots, for example.

[0085] Figure 1A An overall architecture of a controller for controlling an autonomous vehicle operating in an environment according to an embodiment of the present disclosure is shown.

[0086] An ego vehicle 101, which may be an autonomous vehicle, is controlled by a controller 102 and operates in an environment 103 surrounding the ego vehicle 101 that affects the operation of the ego vehicle 101 and may be discovered through sensing. It is assumed that the behavior of the ego vehicle 101 in response to one or more commands issued by the controller 102 is known. The environment in which the ego vehicle 101 operates is not completely known a priori, but information and knowledge of the environment 103 may be obtained from measurements of at least one sensor 104 placed on or remotely connected to the ego vehicle 101. In practice, more than one sensor may be carried on the ego vehicle without departing from the scope of the present disclosure.

[0087] The controller 102 comprises information extraction 111 about data of the environment 103 obtained from at least one sensor 104. The information extraction 111 enables the controller 102 to acquire 111a knowledge of the environment 103 and use this information in the control unit 112 together with known information about the ego vehicle 101 to calculate commands 105 for controlling the ego vehicle 101 to achieve control goals while ensuring safety with respect to the operational requirements of the ego vehicle 101 and the interaction with the environment 103. To this end, the data of the environment 103 comprises knowledge about the environment 103 as derived based on measurements 104a of at least one sensor 104 sensing the environment 103. The sensing is based on at least one of a state of the ego vehicle 101 and a sensing instruction for controlling the operation of the at least one sensor 104.

[0088] The controller 102 is configured to determine the optimal operation of the ego vehicle 101 while avoiding unsafe events caused by the environment 103 that is not known a priori and is therefore uncertain but can be discovered through sensing. In addition, the sensing information acquisition capability in turn depends on how the ego vehicle 101 operates so that the acquisition of information does not prevent the ego vehicle 101 from achieving its goals or degrade its performance in achieving its goals.

[0089] Some embodiments are based on the recognition that sensing objectives are secondary to control objectives because they do not themselves motivate the operation of the ego vehicle 101, although sensing objectives may affect the operation of the ego vehicle 101. To this end, some embodiments disclose interdependent but unbalanced control and sensing applications, where control is the primary objective and sensing is a secondary objective.

[0090] To this end, the control unit 112 performs a joint but unbalanced optimization of the control input 105 and the knowledge of the desired environment.

[0091] Figure 1B 1 shows a block diagram of a joint but unbalanced multivariable optimization of control inputs 105 according to an embodiment of the present disclosure. Figure 1B , x represents variables of the ego vehicle 101, while y variables of the environment 103 are subject to admissibility conditions determined as a function of knowledge of the environment.

[0092] To achieve the interdependence between x and y, some embodiments disclose jointly determining the state trajectory of the ego vehicle 101 and the amount of knowledge required for the state of the environment 103 surrounding the ego vehicle 101, i.e., as part of a single, single multivariable optimization. Therefore, some embodiments replace the sequential optimization 110 and the balanced multivariable optimization 115 with an unbalanced multivariable optimization 117. The sequential optimization 110 includes optimizing the state function f(x) and then optimizing the function of the knowledge of the environment g(y) given x. The balanced multivariable optimization 115 includes jointly optimizing the function F(x,y) of x and y in a balanced manner, where the control objectives related to the state of the ego vehicle 101 are given equal weight to the sensing of the environment 103. However, the unbalanced multivariable optimization 117 includes optimizing the state function f(x) subject to the constraint that the function of the knowledge of the environment g(y)>h(x).

[0093] To achieve the imbalance in their importance, some embodiments use multivariable constrained optimization on the state trajectory (which can be defined by f(x)) and the amount of knowledge of the state of the environment (which can be defined by g(y)), where the knowledge of the environment acts on the constraints. In this way, the knowledge of the environment is subordinate to the control objective because the amount of knowledge acting on the constraints does not have an independent optimization objective and it simply constrains the control commands selected for the ego vehicle 101.

[0094] Figure 1C A schematic diagram of the constraints used by the unbalanced multivariable constrained optimization 117 according to some embodiments is shown. It is an object of some embodiments to provide such a multivariable constrained optimization of control commands and required knowledge, which updates the sensing commands only when the required knowledge is insufficient to optimize the state trajectory of the ego vehicle 101. To this end, it is recognized that in addition to or instead of imposing constraints on the acquired knowledge, it is necessary to impose constraints on the admissible states of the ego vehicle 101 based on the acquired knowledge. These admissible states control the optimization space of the state trajectory of the controlled ego vehicle 101, so that the need to increase the acquired knowledge is only created if the lack of knowledge hinders reaching the optimal trajectory. In this way, some embodiments aim to define a given set of constraints. The interaction constraints 120 are structural constraints.

[0095] In some embodiments, the environment 103 is defined by its state, which is referred to herein as the state of the environment or the state of the environment. The state of the environment collects all relevant data about the environment 103 from the perspective of its impact on the controlled ego vehicle 101. The state of the environment can include multiple state variables, such as an occupancy grid in the area around the controlled ego vehicle 101, the movement of points and / or objects in the surrounding environment, the spatial boundaries in which the controlled ego vehicle 101 can operate, and the allowed separation and relative speed of the ego vehicle 101 and objects in the environment 103. The state of the environment captures relevant, available or acquired knowledge. The acquired knowledge can be unambiguously converted into constraints on the permissible values ​​of the state and the control inputs. However, for the purpose of multivariable constrained optimization, it is also necessary to define constraints based not only on the amount of knowledge of the environment available but also on the lack of such knowledge. It is challenging to define constraints based on unknown things.

[0096] Some embodiments are based on the recognition that the constraints on the allowable values ​​of the state and the control inputs can include two parts. The first part can be positively defined by the current knowledge of the environment. The second part has a negative property because it defines the lack of precise knowledge. Some embodiments are based on the recognition that the negative property of the second part can be converted into a positive restriction based on the first part. That is, instead of having the second part describe the unknown, the second part can be positively defined as the uncertainty of the first part. In this way, instead of having constraints defined by known environmental knowledge and unknown knowledge about the environment, some embodiments determine interactive constraints 120 defined by knowledge of the environment and uncertainty about what has been defined by the knowledge of the environment. This conversion allows the unknown knowledge to be represented in a way suitable for constraint derivation.

[0097] Therefore, in some embodiments, the state of the environment and the uncertainty of the state of the environment are used to probabilistically define the interaction constraints 120. For example, when known, i.e., acquired knowledge is used to define a region of permissible state values ​​or equivalently a region of inadmissible state values ​​of the controlled ego vehicle 101, the uncertainty of the lack of knowledge can be positively represented by reducing the region of permissible values ​​and / or increasing the region of inadmissible values. In this way, the probabilistic interaction constraints 120 can reflect the uncertainty of the knowledge.

[0098] However, it is also recognized that the prediction of the future state of the environment affects the interaction constraints 120, and that measurements that will be acquired in the future about the environment 103 affect the prediction of the future state of the environment. Therefore, some embodiments consider different constraints that are adjusted for different predictions of future measurements of the environment, resulting in different predictions of the future state of the environment. It is also recognized that the uncertainty remains the same in different measurements because the uncertainty depends on the quality of the measurement, not its actual value.

[0099] With this understanding, some embodiments disclose combining interaction constraints 120 and representations 125 associated with vectors of random variables using mean and covariance In uncertain environment The chance constraint 127 can be implemented in various ways. For example, the chance constraint 127 can be implemented as a function that combines the state of the self-vehicle 101, the state of the environment 103, and the interaction of the control input through the negative value of a monotonically increasing function of a measure of the uncertainty of the state of the environment when future measurements of the environment are predicted to take corresponding values.

[0100] To this end, some embodiments disclose determining a sequence of control inputs or control commands 105 to the ego vehicle 101 and sensors 104 by solving a multivariable and multi-stage stochastic constrained optimization problem for a model of the motion of the ego vehicle 101 that takes into account information extraction 111 from the uncertain environment 103. The model of the motion of the ego vehicle 101 takes the trajectory x of the state s Associated with a sequence 105 of control inputs are chance constraints 127 on the control inputs of the ego vehicle 101 defined with respect to admissible values ​​of the state of the ego vehicle 101 and based on desired knowledge of the environment 103 resulting from the acquired knowledge of at least some possible transitions of the states of mobile objects in the environment 103 at different stages of the stochastic constrained optimization and the corresponding uncertainties of the transition states of the mobile objects.

[0101] To this end, the motion of the ego vehicle 101 is controlled while satisfying a set of input constraints and a chance constraint on admissible values ​​of the state of the ego vehicle 101, and the sensor 104 is controlled by a sequence of sensing instructions, both of which are based on a sequence 105 of control inputs according to a model of the motion of the ego vehicle 101, the sequence 105 of control inputs being determined by solving a multivariable and multi-stage stochastic constrained optimization problem. Since the environmental measurements depend on the inputs of the system state and / or environmental measurements, the multivariable and multi-stage stochastic constrained optimization problem strikes a balance between achieving the tracking goal of the ego vehicle 101 and reducing the uncertainty of the environment 103 by enhancing the sensing, so that the future tracking performance of the ego vehicle 101 can be improved.

[0102] In order to achieve this goal, a stable system state x is required. s and environmental uncertainty Σ e The cost function of .

[0103] Figure 1DSchematic diagram of a cost function optimized by unbalanced multivariable constrained optimization according to some embodiments. Some embodiments are based on the recognition that the operating principle of the ego vehicle 101 can be captured by a cost function 130, which includes the stage cost and terminal costs They each include a term proportional to the performance of the state of the ego vehicle 101 relative to the control objective and a term proportional to the amount of knowledge of the environment. Some embodiments are based on the recognition that including terms proportional to the amount of knowledge of the environment in the stage cost 133 and the terminal cost 134 in the cost function 130 may help in situations where the goal of control changes in the future, as this may require the ego vehicle 101 to operate in a previously avoided area that has not yet been acquired because the information is not relevant to the previous control objective.

[0104] However, some embodiments are based on the realization that, in order to achieve unbalanced control, the cost function 130 should not depend on the state of the environment. To this end, some embodiments remove the stage and terminal costs from the cost function, for example, by making their weights equal to zero, or impose special constraints 135 that allow the acquired knowledge to be improved in one or a combination of both scenarios. In the first scenario, the controller 102 does not make any effort to improve the knowledge of the environment unless this has a direct benefit in achieving the performance of the control goal. In the second scenario, the controller 102 can still improve the knowledge of the environment as long as it does not hinder the achievement of the control goal.

[0105] Thus, in some embodiments, the controller 102 is configured to determine a cost function associated with a newly formulated multivariable and multi-stage stochastic constrained optimization problem, the cost function being defined as a function of a weighted average of costs obtained by conditioning on a sequence of possible predicted future measurements of one or more mobile objects in the environment 103 surrounding the ego vehicle 101. The weights of these costs are calculated as a function of the probabilities of the points in the grid of possible future measurements as a sequence of predicted corresponding sequences of possible future measurements.

[0106] Figure 2A and Figure 2B A block diagram is shown of a method of performing a multi-stage multivariable constrained optimization 102a by some embodiments of the controller 102 to achieve a joint but unbalanced optimization.

[0107] like Figure 2A As shown, the controller 102 is configured to obtain a state 101a ( Figure 2B) to acquire 210 measurements 215 by at least one sensor 104 that senses an environment 103 surrounding the ego vehicle 101 and senses instructions to control the operation of at least one sensor 104, and processes 220 the measurements 215 to acquire knowledge 225 of the environment 103 and estimate 227 a state of the environment and an uncertainty in the state of the environment.

[0108] Next, the controller is configured to determine possible future measurements 260 of the environment 103 based on the current estimate of the state of the environment and the uncertainty 227 of the state of the environment. The possible future measurements 260 are used to construct the multi-stage multivariable constrained optimization problem 102a (e.g. Figure 2B ), the problem is solved to jointly determine 230 the state trajectory 235a (as defined by the ego vehicle 101) Figure 2B 235 and a desired knowledge 237 of the environment. The desired knowledge 237 of the environment may be defined based on a sequence 250 of (updated) sensing instructions to at least one sensor 104. The multi-stage multivariable constrained optimization problem 102a includes a dynamic model of the ego vehicle 101 that associates a state trajectory 235a with a sequence of control inputs 235, the sequence of control inputs 235 being subject to constraints on allowable values ​​of the state and the control inputs defined based on desired knowledge 237 of the surrounding environment 103 represented by the state of the environment and the uncertainty 227 of the state of the environment; and a prediction of the state of the environment based on possible values ​​260 of future measurements taken about the environment 103. The controller 102 controls 240 the ego vehicle 101 based on the sequence of control inputs 235 using a solution to the multivariable constrained optimization problem to change the state of the ego vehicle 101, and updates 250 sensing instructions based on the desired knowledge 237 of the environment 103, and submits the updated sensing instructions to the at least one sensor 104.

[0109] Exemplary Structural Components

[0110] Figure 3A is a schematic diagram of an ego vehicle 311 including a controller 312 employing the principles of some embodiments of the present disclosure. As used herein, the ego vehicle 311 may be any type of wheeled vehicle, such as a passenger car, a bus, or a rover. In addition, the ego vehicle 311 may be autonomous or semi-autonomous. For example, some embodiments control the motion of the ego vehicle 311. Examples of motion include lateral motion of the ego vehicle 311 controlled by a steering autonomous vehicle 313 of the ego vehicle 311. The steering autonomous vehicle 313 is controlled by the controller 312.

[0111] The ego vehicle 311 may also include an engine 316, which may be controlled by the controller 312 or by other components of the ego vehicle 311. The ego vehicle 311 may also include one or more sensors 315 to sense its current motion and internal state (by way of non-limiting example). Examples of sensors 315 include global positioning autonomous vehicles (GPS), accelerometers, inertial measurement units, gyroscopes, shaft rotation sensors, torque sensors, deflection sensors, pressure sensors, and flow sensors.

[0112] The ego vehicle 311 may be equipped with a transceiver 317 that enables the controller 312 to have communication capabilities through a wireless communication channel via an input interface 318. The vehicle includes one or more sensors 314a, 314b to sense the surrounding environment. Examples of sensors 314a, 314b may be distance range finders, radars, lidars, and cameras. Alternatively or in parallel, sensor data about the surrounding environment may be received through the transceiver 317 for sensors remotely placed relative to the ego vehicle 311. The ego vehicle 311 may be equipped with a map database of an autonomous vehicle storing data about roads in the area in which the ego vehicle 311 is operating, or it may access remotely stored map data through the transceiver 317. As will be well understood by those of ordinary skill in the art, the ego vehicle 311 may be equivalent to the ego vehicle 101, and the sensors 315, 314a, and 314b may be equivalent to at least one sensor 104 described above, without departing from the scope of the present disclosure.

[0113] In this example, vehicle 311 is controlled autonomous vehicle 101, sensors 314a, 314b and any remote sensors whose information is received from transceiver 317 are sensors 104, controller 312 is controller 102, and the road and other vehicles are (in a non-limiting manner) environment 103, without departing from the scope of the present disclosure.

[0114] Figure 3B is a block diagram of a computing platform for controller 102 according to some embodiments. For example, Figure 3B A block diagram of a controller 312 according to one embodiment is shown. The controller 102 includes a hardware processor 301 connected to a memory 302 (e.g., a non-transitory computer readable medium). In some implementations, the memory 302 includes a first portion 321 for storing data about the ego vehicle 101 and a second portion 322 for storing a program of the control unit 112 for calculating control of the ego vehicle 101, a third portion 323 for storing data about the environment 103, and a fourth portion 324 for storing a program for extracting data 111 about the environment from data from at least one sensor 104.

[0115] For example, the first portion 321 of the memory 302 may store parameters for the behavior of the ego vehicle 101, such as maximum acceleration and maximum speed, as well as a model of the ego vehicle 101 and a goal of the ego vehicle 101. The second portion 322 of the memory 302 may have a program implemented thereon that is executable by the processor 301 for calculating commands (e.g., control commands 105) for the ego vehicle 311.

[0116] Still refer to Figure 3B , a third portion 323 of the memory 302 includes data about the environment 103, such as the allowed operating space and the presence of other actors, allowed interactions, possible future actions of other actors, and the amount of update information and historical information available for elements in the environment 103. To this end, the environment 103 may include at least one mobile object that acts as another actor in the environment 103 of the ego vehicle 311. A fourth portion 324 of the memory 302 may have a program implemented thereon that is executable by the processor 301 for processing information obtained from sensors 314a, 314b in the vehicle 311 and remote sensors communicating through the transceiver 317, and for predicting possible measurements according to different conditions.

[0117] The processor 301 may be any computing device capable of performing calculations, and may include one or more physical devices of the same or different types. It is possible that the processor 301 may include multiple computing devices, such as microprocessors. Similarly, the memory 302 may be any logical memory and / or non-transitory computer-readable storage medium capable of storing data, and may include one or more physical data storage devices of the same or different types. The calculations performed by the processor 301 are directed by programs stored in the second portion 322 and the fourth portion 324 of the memory, and use the ego vehicle data stored in the first portion 321 and the third portion 323 of the memory, data obtained from sensors 315, 314a, 314b, and transceiver 317. The calculations of the processor 301 generate commands 304 that change the state of the ego vehicle 101.

[0118] Example Operation

[0119] Some embodiments are based on the recognition that when the ego vehicle 101 is operating in an environment that is unknown but discovered through information acquired via sensing, the environment itself is relevant to the commands applied. For example, Figure 4A and Figure 4BA scenario is shown when the ego vehicle 311 must decide whether to stay in the current lane 411 or change to the left lane 412 and there are other actors (specifically, other vehicles) on the road. The other actors include at least one moving object, such as vehicle 416, vehicle 417, or vehicle 418. Each of these other actors can be an autonomous, semi-autonomous, or manually driven vehicle without departing from the scope of the present disclosure.

[0120] If the controller 102 selects Figure 4A As shown in trajectory 413, due to vehicle 416, sensor fields of view 414, 415 can capture information about moving objects or vehicles, such as a car farther ahead in current lane 411, rather than vehicle 418 ahead in left lane 412. Figure 4B In the trajectory 423 shown, due to the presence of the front vehicle 416, the sensor fields of view 424 and 425 cannot capture information about the vehicle 417 in the current lane 411 that is farther ahead, but can capture information about the front vehicle 418 in the left lane 412. Therefore, the trajectory 413 allows more information about other vehicles in the current lane 411 to be obtained, while the trajectory 423 allows more information about the vehicle in the left lane 412 to be obtained.

[0121] However, the control command 105 also relies on the available knowledge 225 of the environment 103. Thus, if a lane change is desired, the lack of knowledge about the leading vehicle in the left lane 412 may not allow the controller 102 to safely plan a trajectory for changing the ego vehicle 311 to the left lane 412 due to the uncertainty of the position of the vehicle in that lane. Thus, the control command 105 affects the available knowledge 225 about the environment 103, and the availability of knowledge about the environment 103 affects the control command 105.

[0122] In order to predict the future environment to plan the future motion of the ego vehicle 311 , the controller 102 also needs to predict future measurements of actors (eg, other vehicles) in the environment 103 in addition to future uncertainties of the environment based on sensors and vehicle operations.

[0123] like Figure 4C As shown, if vehicle 416 changes lanes to the left lane, from lane 411 to lane 412, the field of view 434, 435 of ego vehicle 311 changes significantly. This will result in different future measurements of the environment (e.g., Figure 2A and Figure 2B260 described in ). Therefore, in some embodiments of the present disclosure, when planning a future control action for the ego vehicle 311, the control action depends on the predicted future realization of the measurement. In some embodiments, the future realization of the measurement is predicted based on the possible behavior of at least one mobile object (416, 417, or 418) present in the environment 103 (e.g., whether the vehicle 416 stays in the current lane 411 or moves to the next lane 412).

[0124] Figure 5A Another scenario on the same road is shown, where the ego vehicle 311 is interacting with another actor or another moving object (e.g., another car 510), and the ego vehicle 311 can point its sensor 104 to sense in the cone 505 according to the pointing direction 504. The ego vehicle 311 can plan different trajectories, such as trajectory 501 and trajectory 502, and the other car 510 can also execute different trajectories, such as trajectory 511, trajectory 512, and trajectory 513, which are a priori unknown to the ego vehicle 311. Depending on the trajectory selected by the other car 510, different combinations of control actions and sensing may be the best combination for the ego vehicle 311 to plan.

[0125] For example, if Figure 5B As shown, if the other car 510 remains in the current lane 412 , the optimal plan for the ego vehicle 311 may be to remain in the same lane 411 and sense within the cones 521 , 522 , 523 at corresponding points of the trajectory 501 to increase information about the other car 510 .

[0126] On the contrary, if Figure 5C As shown, if the other car 510 changes to the right lane 411, the best plan for the ego vehicle 311 may be to change to the left lane 412 and sense within the cones 531, 532, 533 at the corresponding points of the trajectory 502 to also increase the information about the other car 510 that is now moving on a different trajectory. Therefore, when planning, actors such as at least one mobile object in the environment 103 (e.g. Figure 5A , Figure 5B and Figure 5C Different possible behaviors of the car 510).

[0127] Fig. 6AThe effect of uncertainty due to different behaviors (specifically, three trajectories: trajectory 511, trajectory 512, and trajectory 513) of an actor (another car 510) in the environment 103 is shown. Due to the perception of the other car 510 by the ego vehicle 311, each trajectory has uncertainty associated with each point, such as uncertainty ellipsoids 611, 612, and 613 associated with the points of trajectory 511, and uncertainty ellipsoids 631, 632, 633 associated with the points of trajectory 513. Since all three trajectories are possible, without considering information about future measurements, planning for the ego vehicle 311 needs to consider all uncertainty ellipsoids covering all trajectories, such as ellipsoids 601, 602, 603. To ensure safety, the movement of the ego vehicle 311 must avoid the uncertainty ellipsoids. Therefore, larger ellipsoids may significantly restrict the movement of the ego vehicle 311, thereby reducing performance. For example, since the uncertainty ellipsoid 613 covers two lanes, it is impossible for the autonomous vehicle 311 to overtake the other car 510.

[0128] Figure 6B The effect of uncertainty on the decision to control the ego vehicle 311 due to different behaviors of an actor or one or more other moving vehicles in the environment 103 of the ego vehicle 311 is shown. According to this example, the actor is another car 510, specifically, having three trajectories: trajectory 511, trajectory 512, and trajectory 513.

[0129] The controller 102 is configured to perform multivariable and multi-stage stochastic constrained optimization by probabilistically adjusting future transitions of the ego vehicle 311 for predicted possible future measurements of one or more other mobile objects in the environment 103 , such as the car 510 .

[0130] By adjusting the decision of the control for the future realization of the measurement related to the conditional information about the future trajectory of the other car 510, it is not necessary to cover the uncertainty (such as 611, 612, 613) of the different trajectories (511, 512, 513) of the other car 510 by a single uncertainty set (such as 601, 602, 603) because the motion plan of the ego vehicle 311 has been adjusted for the realization of the plan of the other car 510 by adjusting for the realization of the future measurement of the other car 510. Therefore, instead of avoiding the area 601, 602, 603, the controller 102 will formulate a different plan to be selected when the future measurement of the other car 510 is realized, and the different plan avoids the uncertainty ellipsoid 611, 612, 613, or the uncertainty ellipsoid 621, 622, 623, or the uncertainty ellipsoid 631, 632, 633. In each case, the area to be avoided is much smaller, so the ego vehicle 311 has more drivable area on the road and will be able to achieve higher driving performance. For example, there is no longer an uncertainty ellipsoid covering both lanes, so the ego vehicle 311 can always overtake the other car 510.

[0131] Exemplary Controller Architecture

[0132] Fig. 7A 1 shows a block diagram of a controller 102 for controlling an ego vehicle 101 according to some embodiments. The controller 102 uses a predictive model or a model of the motion of the ego vehicle 101 described by a known function such as a discrete time update equation,

[0133]

[0134]

[0135] Among them, x s is the known state of the ego vehicle 101, u s The command 105 issued by the control unit 112 is s is the performance output of the ego vehicle 101, ie, a signal that must achieve a specific goal.

[0136] In some embodiments of the present disclosure, the ego vehicle models (1a), (1b) are kinematic bicycle models, given as follows:

[0137]

[0138] Among them, p x 、p y is the east (horizontal) and north (vertical) position, v is the velocity, a is the acceleration, ψ is the heading angle, δ is the steering angle, and β is the vehicle slip angle, which are given as follows:

[0139]

[0140] Among them, l f , l r are the distances from the center of gravity to the front axle and the rear axle respectively.

[0141] Then, model (1c) is used with a sampling period T s It is represented in discrete time as (1a).

[0142] The operational requirements for the ego vehicle 101 are described by the operational constraints 702 on the ego vehicle state and commands.

[0143] h s (x s ,u s )≤0 (2a)

[0144] For example, for the kinematic bicycle model (1c), the constraints can be given as follows:

[0145]

[0146] It includes upper and lower limits for speed, acceleration, heading angle, and steering angle.

[0147] The ego vehicle 101 has a control target 703, such as output y s Arrival position or tracking reference signal r s ,

[0148]

[0149] For the ego vehicle 101 modeled as (1c), the output y s It can be:

[0150]

[0151] This is the position vector and the reference signal r s A desired position vector varying with time is defined. Alternatively or in parallel, the output vector may include a velocity v and a heading ψ.

[0152] The ego vehicle 101 operates in an environment 103 that is not completely known but about which information can be obtained from sensing to improve knowledge about it. In some embodiments of the present disclosure, the equations describing the dynamics of the environment 705 are known and modeled by the following equations:

[0153]

[0154] Among them, x eis the environment state, which collects all relevant data about the environment from the perspective of its impact on the ego vehicle 101.

[0155] Although equation (4a) may be known, the environmental state x e is not directly known. In addition, the model of the environment may be subject to unknown perturbations w e The unknown disturbance may cause a change in the state of the environment and is unknown to the controller 102. Examples of component models in the environment include a constant model with additive disturbances

[0156] p k+1 =p k +w k …4(b)

[0157] Constant speed model

[0158] p k+1 =p k +v k T s +w k …4(c)

[0159] Stable tracking first-order model

[0160] p k+1 =αp k +(1-α)r k +w k …4(d)

[0161] Different models can be used to represent different components of the environment, resulting in the environment state vector x e of different quantities.

[0162] Information about the environment is obtained from measurements 706y acquired by sensors 104. e Obtained in

[0163]

[0164] Among them, v e is unknown noise that corrupts the measurement, and where the measurement depends on the current state of the ego vehicle 101 , and possibly also on the current command of the ego vehicle 101 .

[0165] In some embodiments of the present disclosure, the state x of the ego vehicle s The quality of the measurement can be affected, for example, by reducing the amount of corruption of the measurement caused by noise. For example, the measurement can be made by , where the noise that corrupts the measurement increases as the distance between the ego vehicle state and the environment state (e.g., a road sign being detected or an actor on the road being tracked (e.g., another car)) increases.

[0166] Some embodiments are based on the recognition that the controller 102 may provide sensing instructions to the sensors 104 that change the amount of knowledge acquired by the sensors 104 about the environment 103 or a portion thereof. For example, knowledge of a particular area of ​​the environment may be acquired by focusing a radar or lidar sensor on the area of ​​interest without changing the state of the ego vehicle 101. However, the decision about where to focus the sensor 104 does not change the state of the ego vehicle 101. Therefore, some embodiments define and include in the optimization of the control commands 105 control inputs that do not affect the change in the state of the ego vehicle 101, but only the amount of knowledge acquired about the environment 103 or a portion thereof.

[0167] To this end, in some embodiments of the present disclosure, the ego vehicle command u s Some elements of do not directly affect the motion of the ego vehicle but affect the quality of the measurement. For example, the measurement can be described by the following equation

[0168] in, It is a command The dth component of has a range between 0 and 1, It does not affect the autonomous driving vehicles (1a) and (1b). In this case, These can represent, for example, the ability to focus the sensor more on a particular area at a particular time, or to perform additional processing on certain measurements to improve their quality, such as by correcting for corruption of the measurement due to noise.

[0169] Thus, in some implementations, the set of commands defined by the control inputs includes commands associated with sensing instructions that have no effect on the state of the ego vehicle 101 while changing the amount of knowledge gained from the environment 103 .

[0170] Some embodiments are based on the recognition that constraints on the permissible values ​​of states and control inputs can be expressed as chance constraints that define the known environment obtained in a positive manner as the state of the environment and the unknown environment as the uncertainty of the state of the environment. For example, the measurements used in the estimator 704 produce the state of the environment and its uncertainty Estimates

[0171]

[0172] The uncertainty of the estimator is the inverse of the knowledge of the state of the environment That is, the more uncertainty there is in the estimate of the state of the environment, the less knowledge there is about the state of the environment. Similarly, the uncertainty of an estimate is the inverse of the reliability of the estimate That is, the greater the uncertainty, the less reliable the estimate of the state of the environment. Information is the change in knowledge between two consecutive steps.

[0173] In some embodiments of the present disclosure, the estimator 704 can be constructed as a dynamic observer.

[0174]

[0175]

[0176] in, is the observer gain, which may depend on the ego vehicle state and command, ∑ w ,∑ v is the uncertainty about interference and noise, such as their covariance, are their nominal values, such as their means. Function Depends on the distribution and the environmental dynamics, for example the linear equation for the environmental dynamics is

[0177]

[0178]

[0179] and Gaussian perturbations w and noise v, with mean and covariance (μ v ,∑ v )、(μ w ,∑ w ), then the estimated update function is

[0180]

[0181] and the function to update the uncertainty of the estimate (here the covariance or the estimation error) is

[0182]

[0183] The environment imposes constraints on the ego vehicle, which are represented as interaction constraints 707

[0184]

[0185] This is what the controller must satisfy in the face of uncertainty. In some embodiments, the function h is called an interaction function.

[0186] For the ego vehicle, examples of interaction constraints include staying between lane boundaries.

[0187]

[0188] Among them, b u 、b l are the upper and lower bounds of the lane, and collision avoidance with other actors (e.g., other cars 510),

[0189]

[0190] Among them, l v ,l o ,l m They are the front length of the ego vehicle, the rear length of the obstacle, and the safety margin.

[0191] In this way, the uncertainty of the state of the environment 103 is the inverse of the acquired knowledge of the environment 103, and the constraints imposed on the allowable values ​​of the state and the control inputs include interaction constraints 707 of variables that allow interaction (these variables include the state of the ego vehicle, the state of the environment, and the control inputs), so that the multivariable constrained optimization is constrained to satisfy the interaction constraints in the presence of uncertainty in the state of the environment.

[0192] Exemplary Control Unit Operation

[0193] In some embodiments, the controller is configured to recursively estimate the environment and the uncertainty of the state of the environment based on a dynamic model of the state of the environment. For example, the control unit 112 operates iteratively in a control cycle, such as Figure 7B shown.

[0194] In control cycle k, the control unit receives 711 the ego vehicle state Then, the control unit constructs an ego vehicle estimator based on the ego vehicle model (1a), (1b), and the ego vehicle operation constraint (2a), and initializes the model at the current ego vehicle state

[0195]

[0196] The control unit 112 receives 712 information about the state of the environment from the estimator. and its uncertainty The current estimate of . Then, the control unit builds the state of the environment and estimates of their uncertainties.

[0197] In some embodiments, the control unit 112 uses the following method, namely, by adding an additional term ∑ in the covariance prediction as the covariance of the measured prediction error. y and include additional uncertainty in the predicted measurements Using the model of estimator (6) as predictor leads to:

[0198]

[0199] However, this leads to an open-loop prediction of the measurement error, which is known to be conservative. Instead, a conditional prediction can be applied, where the prediction is adjusted for the realization of the measurement

[0200]

[0201] in, is the ith value of the measurement that can be achieved, The state estimated for adjusting for this measurement is thus achieved. The adjustment predictor (11b) does not need to rely on the measurement prediction error in the covariance prediction as long as it adjusts for any possible value of the measurement.

[0202] In some embodiments of the present disclosure, it is recognized that the predictor (11b) needs to apply adjustments to all possible measurements, which is computationally challenging, meaning that it would require too many computing resources.

[0203] On the contrary, Figure 7B As shown, the controller selects a discrete set of prediction measurements For adjusting 713, and for the estimator for this discrete set Application Adjustment

[0204]

[0205] Among them, (∑ q ) (i) is the covariance of the quantization error.

[0206] like Fig. 8A As shown, for a predicted measurement range 801 of future measurements, a grid 802 of possible future measurements is defined. Each point of the grid 802 is associated with the measurement range it represents, so that the measurement range associated with the corresponding point provides the probability of the point and the quantized uncertainty of the point. For example, for each measurement possibility, such as point 803 in the grid 802, the controller 102 selects a corresponding associated measurement range 804, i.e., the measurement range represented by the measurement possibility 803. Then, for the measurement range 804, the quantized error (∑ q ) (i)The covariance of represents the quantization uncertainty of point 803. According to the probability distribution of the measurement in range 804, the associated measurement quantization error (∑ q ) (i) , and also associated with the probability of measuring the possibility The area 805 under the probability distribution 800 is measured within the range 804. This represents the probability that the point 803 is in the grid 802.

[0207] In some embodiments of the present disclosure, the measurement possibilities 803 have a constant spacing in the grid 802 of possible future measurements, so that the ranges 804 have a fixed size, which generally results in a varying region 805 and thus a varying measurement quantization error covariance. Thus, the spacing of any two adjacent points in the grid 802 is constant, and all measurement ranges 804 associated with all points 803 in the grid have the same size, which is defined by a predetermined value. The predetermined value may be configurable and may be arbitrarily selected based on the desired application of the controller 102.

[0208] Furthermore, in the grid 802 of possible future measurements, the probability of a measurement being over a corresponding range 804 of measurements associated with a point, eg 803 , of the grid 802 is equal for all points.

[0209] In other embodiments of the present disclosure, Figure 8B As shown, controller 102 selects measurement possibilities 813 to be differently spaced, and associated measurement ranges 814 to also be of different sizes and selected to keep the area 815 under the probability curve 800 of the measurement range constant, thereby providing a constant probability of the measurement possibility occurring.

[0210] In other embodiments of the present disclosure, the controller 102 selects the measurement possibilities, such as points 813 on the grid 802, to be differently spaced and the associated measurement ranges 814 to also be of different sizes and selected to maintain a constant square variation with the measurement possibilities, thereby resulting in a constant measurement quantization error covariance (∑ q ) (i) =∑ q .

[0211] Thus, in the grid 802 of possible future measured points, the quantization error defined above defines a quantization uncertainty associated with a corresponding point (e.g., a point 813 in the grid 802 of discrete points) that is equal for all points. Furthermore, as described above, the quantization uncertainty associated with a corresponding point 813 of the grid 802 is calculated as the covariance (∑ q ) (i) .

[0212] Based on the possible future measurements predicted in this way, the predictor or estimator 704 is initialized at the state of the environment and its uncertainty Current estimates.

[0213]

[0214] Return to reference Figure 7B , the control unit 112 constructs 714 the interaction constraints (9a) based on the predicted state of the environment 103 and its uncertainty. Since the environment 103 is not completely known, the state of the environment is represented by the deterministic component and the uncertainty component of the knowledge about the environment. and its uncertainty To construct constraints, the interaction constraint (9a) is satisfied with the specified confidence despite the uncertainty in the estimate of the environment.

[0215] For example, based on predictors (11a), (11b), one embodiment modifies constraint (9a) to obtain an uncertainty-dependent tightened interaction constraint depending on the estimate of the environment and its uncertainty from (6),

[0216]

[0217] The function γ makes the constraint more stringent to consider the predicted environmental state estimate based on how likely such an error is to occur (i.e., based on the uncertainty of the estimate). The possible error in . The function γ is relative to A measure of , such as its norm, is monotonically increasing.

[0218] If the initial state of the environment 103 is distributed by probability Describe, and if the disturbance w and noise v are also distributed by probability distribution p v (v),p w (w) describes the environment state estimation and its uncertainty is the environmental state estimate The mean and covariance of the probability distribution of .

[0219] Then, the function γ can be designed so that (8) is satisfied with the confidence level 1-ρ to ensure (7),

[0220]

[0221] To this end, the function γ can be defined as

[0222]

[0223] in, is the sensitivity of the constraint to the state of the environment, and α is based on The coefficients are calculated based on the probability distribution and the confidence level 1-ρ. For example, Gaussian distribution, in, is the standard normal inverse cumulative distribution function, and for the general distribution

[0224] In the case where (9a) contains multiple constraints, repeat this step for each constraint separately. and the case of linear autonomous vehicle dynamics, if p v (v), p w (w) is Gaussian, then is Gaussian, so (12) is

[0225]

[0226] The interaction constraints (12) and (15) depend on the estimated mean of the environmental state. and covariance However, while the prediction of the covariance is independent of the measurement, the mean depends on the measurement predicted by one of the predictors in (11a) to (11c)

[0227] Handling of non-convex constraints for collision avoidance, such as non-convex ellipsoid constraints

[0228] (x s -x e )′P(x s -x e )≥d min (16)

[0229] It is challenging when there is uncertainty about the environment because it is difficult to construct the function γ in (12) to correctly represent the uncertainty without over-approximating it. On the other hand, linearizing uncertain row constraints is also difficult because linearization inevitably changes the tightness of the constraints due to the uncertainty. In this disclosure, it is recognized that the effects of uncertain constraints can be preserved by first convexifying the constraints without uncertainty and then imposing the effects of uncertainty on the linearized constraints.

[0230] Fig. 9An ego vehicle 901 is shown operating on a road 902 in the presence of another vehicle 903. Due to the uncertainty in sensing the state of the other vehicle 903, a collision avoidance region 904 is to be avoided, which grows into region 905. In some embodiments of the present disclosure, convexification is equivalent to computing a point 911 on the boundary of the collision avoidance region when there is no uncertainty, and using the tangent line 907 at that point as the boundary of the constraint. When the uncertainty is taken into account, the hyperplane shifts to a new hyperplane 906, which defines an infeasible region 908 represented by a linear constraint. Avoiding such regions ensures collision avoidance, and now only linear constraints of the form of (15) constructed according to (16) need to be implemented.

[0231] In some embodiments of the present disclosure, point 911 is determined as the closest point between the collision avoidance region 904 and the ego vehicle 901 according to a given distance metric, such as Euclidean distance, norm-based distance, etc.

[0232] Example Optimization Techniques

[0233] Some embodiments are based on the recognition that the controller 102 can optimize ego vehicle 101 operation and information acquisition from the environment 103 based on a predicted sequence of measurements acquired from the environment 103 by adjusting decisions to such predicted sequence of measurements,

[0234]

[0235] For example, Figure 7B As shown, the controller 102 receives 715 the current target r of the ego vehicle. s , and constructs a prediction cost function associated with a multivariable, multi-stage stochastic constrained optimization problem for the motion model of the ego vehicle, which is conditioned on the predicted measurement sequence assuming that the jth future measurement sequence is received,

[0236]

[0237] Where F is the terminal cost of the cost function associated with the terminal condition at the end of the prediction horizon N, and l is the stage cost of the cost function controlled by the optimized control input taken along the prediction horizon N. For example, the stage cost may always be greater than or equal to 0, and may have a value of 0 when (or sometimes only when) the goal is achieved.

[0238] The controller 102 then constructs a complete cost function under all assumed predicted measurements for adjustment 713 as a weighted average of the cost functions adjusted for the sequence of predicted measurements

[0239]

[0240] Among them, w j is a non-negative weight. In some embodiments of the present disclosure, the non-negative weight w j is chosen to be equal to the sequence (Y e ) (j) The probability of For example, Fig. 8A and Figure 8B The probabilities defined in grid 802 predict possible future measurements as described in .

[0241] The control unit 112 also constructs 715 the permissible terminal constraints

[0242]

[0243] That is, if (23) is satisfied, then the operation constraint (2a) and the interaction constraint (9a) are also satisfied, and for commands satisfying (17), the terminal constraint is controlled to be unchanged, that is, if (23) is satisfied, there exists a command satisfying (23) So that when

[0244] hour,

[0245]

[0246] Using the system model (1a), (1b) with constraints (2a), the conditioned predictor of the environment 103, e.g. for the sequence (11b) or (11c) to be adjusted, interaction constraints, such as (15) which can also be obtained in whole or in part from (16), cost function (18) using (19), terminal constraints (23) The controller 102 constructs a multivariable and multi-stage optimization problem 719 which determines the optimal decision for the operation of the ego vehicle 101 and the perception of the environment 103.

[0247] However, the optimization problem constructed at 719 must also include some additional constraints known as non-anticipation constraints, which enforce the fact that at any point in time, the controller 102 can make a unique decision based on its past. The controller 102 can be made to impose non-anticipation constraints to obtain a decision based only on measurements that have been made before the decision was made. Another purpose of implementing non-anticipation constraints is to reduce the number of variables in the multivariable and multi-stage optimization problem 719 by imposing control decisions that are generated by the same sequence of past predictions, have the same value, i.e., are equal, so one can be eliminated. Therefore, in order to adjust the sequence of measurement predictions, (Y e ) (1) ,(Y e ) (2) are equal until step Inputs must be equal up to step

[0248]

[0249] This leads to the tree structure in optimization problem 719.

[0250] like Fig.10 As shown, the two adjustment sequences of measurement predictions 1004, 1005 share the same prediction measurement 1010 in the first step, the same input 1002 in the first prediction step, and the same initial input 1001. In contrast, since the second future prediction measurement 1003 is different, the input 1006 in the second prediction step is different.

[0251] The resulting optimal control problem, including the unanticipated constraint (24) and the initial conditions of the ego vehicle 101 and the estimate of the environment 103, becomes

[0252]

[0253] is converted 716 into an optimization based on adjusting the amount of information to be acquired about the state of the ego vehicle 101 and the state of the environment 103 surrounding the ego vehicle 101 jointly determined from the measurement predictions, which achieves Fig.10 The tree structure shown.

[0254] The optimal solution to this problem consists in following the sequence of measured predictions for all adjustments The optimal command sequence for the ego vehicle 101 and possible sensors 104 for a future horizon of length N periods The first part of the command sequence is used as 717 command for the ego vehicle 101 and possibly the sensor 104 in the current control cycle, because

[0255] The control unit then waits 718 until the next cycle. Therefore, the estimator (6) uses to update the estimate of the environment state and its uncertainty, and the state of the ego vehicle 101 is as described by (1a) and (1b) due to the command And changes have occurred.

[0256] Since the number of variables in the optimal control problem grows exponentially with the prediction horizon N, the present disclosure further recognizes that the sequence of adjustment measurement predictions can be made within the branch horizon N. B Then only one value is expanded to reduce the complexity of the optimization problem constructed according to (25).

[0257] Fig.11A shows the branch horizon, NB 1113, the branches of the tree structure extend to N B 1113 for measurement adjustment. Branch horizon N B 1113 is chosen to be smaller than the prediction horizon N 1114. Therefore, it can be determined that

[0258] like Fig. 11B As shown, this results in a first part 1111 of the optimal control problem, in which the number of optimization variables in each stage 1101, 1102, 1103 increases because multiple possibilities of predicted future measurements are considered, and a second part 1112 of the optimal control problem, in which the number of optimization variables in each stage 1104, 1105 remains unchanged because only one possibility of predicted measurements is considered. B is the number of stages of the first part 1111 of the optimal control problem excluding the root node 1100. Therefore, by making N B Smaller, the controller reduces the computational burden of solving (25).

[0259] In some embodiments of the present disclosure, the controller 102 obtains measurement predictions for the adjustment sequence for measurement prediction by evaluating possible operations of other actors (e.g., one or more other moving objects), based on a discrete target set of these other actors, such as vehicles and pedestrians on the road.

[0260] like Fig.12 As shown, the ego vehicle 1201 is operating on a road 1200 where there are one or more other moving objects, such as another vehicle 1202. The other vehicle 1202 can remain in the same lane 1203 or change to the next lane 1204. Therefore, the controller 102 can predict the state trajectory 1210 of the other vehicle 1202 staying in the current lane, or the state trajectory 1211 changing lanes at the current moment, or the state trajectory 1212 changing lanes at a future moment. Adjusting the target for the other vehicle 1202 causes the predicted measurements for the other vehicle to be adjusted, wherein the predicted measurements are predicted along the state trajectory that the other vehicle 1202 will take according to its target. In fact, due to trajectory quantization, the controller 102 associates uncertainty (e.g., confidence ellipsoids 1221-1226) to each point of the state trajectory (e.g., 1211, 1212, 1213). Therefore, the controller 102 is configured to obtain the following: Fig.10 The structure of the multivariable and multi-stage optimization problem (25) is shown, but now the non-deterministic predictor variable is the target of another vehicle 1202.

[0261] In addition, if Fig.12As shown, starting from the node 1250 associated with the current vehicle position, the controller 102 is configured to predict the goal 1251 of the next step length lane change, thereby obtaining the node 1271 and being forced to continue the lane change through the only goal of the lane change 1255, thereby continuing to the node 1275, so that the controller predicts another vehicle trajectory 1211. Starting from the node 1250 associated with the current vehicle position, the first step length of the goal 1252 of maintaining the current lane obtains the node 1272, which can represent the state trajectory 1211 of the other vehicle 1202 staying in the current lane, and can also represent the state trajectory 1213 of the other vehicle in the subsequent step length lane change, because the two uncertainty points 1222 and 1224 in the decision problem are assigned to the node 1272 together. However, starting from node 1252, if the controller 102 predicts that the goal of the other vehicle 1202 is to remain 1255 in the current lane 1203, then node 1275 is reached and the controller 102 predicts trajectory 1210, and if the controller 102 predicts that the goal of the other vehicle 1202 is to change lanes at this current step size (now the second step size 1254), then node 1274 is reached and the controller predicts state trajectory 1212.

[0262] For example, the controller 102 may be configured to use a linear model for the motion of other actors (e.g., another vehicle 1202),

[0263] x o (k+1)=A o x o (k)+B o r o (k)

[0264] y o (k) = C o x o (k)+D o r o (k) (27)

[0265] It is embedded into the environment model (4a), where the target of the other vehicle is related to r o Once r is predicted o The value of the state x of another vehicle is provided based on o The estimated environment 103 of the vehicle is estimated, and the measurement of the other vehicle is predicted to be y obtained by calculating (27) o By solving (25), these values ​​are embedded into y eThe controller 102 is configured to implement constraints, the constraints comprising constraints on the operation of the ego vehicle 101 and probabilistic constraints associated with the ego vehicle 101 and the environment 103, wherein at least one of the constraints is a collision avoidance constraint to avoid collision with one of the moving obstacles, such as another vehicle 1202, wherein the moving obstacle is represented by a convex shape. The controller is further configured to convexify the collision avoidance constraint by a linear random constraint that separates the obstacle from the current vehicle position. The linear random constraint is constructed by determining a first point on the boundary of the convex shape of the obstacle. A tangent plane of the obstacle shape located at the first point is determined and a linear constraint is constructed that keeps the ego vehicle 101 on the same side of the constraint along the predicted horizon. In addition, the controller is configured to add uncertainty to the estimate of the obstacle state as uncertainty of the linear constraint.

[0266] Controller 102 may also be used in applications involving objects other than moving vehicles, such as in industrial applications involving robots that are configured to lift or transport objects according to set goals and trajectories and that may operate in uncertain environments that include humans or other such obstacles.

[0267] The above-mentioned embodiments of the present invention can be implemented in any of a variety of ways. For example, hardware, software or a combination thereof can be used to implement the embodiments. When implemented in software, the software code can be executed on any suitable processor or processor set (whether provided in a single computer or distributed between multiple computers). Such a processor can be implemented as an integrated circuit with one or more processors in an integrated circuit assembly. However, the processor can be implemented using a circuit of any appropriate format.

[0268] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors using any of a variety of operating systems or platforms. In addition, such software can be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and can also be compiled into executable autonomous vehicle language code or intermediate code that is executed on a framework or virtual autonomous vehicle. In general, in various embodiments, the functionality of the program modules can be combined or distributed as desired.

[0269] In addition, embodiments may be implemented as methods for which examples have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in an order different from that shown, which may include performing some actions simultaneously, although shown as sequential actions in the illustrative embodiments.

[0270] Although the present invention has been described by examples of preferred embodiments, it should be understood that various other adjustments and modifications may be made within the spirit and scope of the present invention. Therefore, the purpose of the appended claims is to cover all such changes and modifications that fall within the true spirit and scope of the present invention.

Claims

1. A controller for controlling an ego vehicle in an environment surrounding the ego vehicle and including at least one moving object, the controller comprising: at least one processor; and a memory having instructions stored thereon that, when executed by the at least one processor, cause the controller to: Acquire knowledge of the environment from measurements of at least one sensor of the ego vehicle sensing the environment based on at least one of: a state of the ego vehicle; and sensing instructions associated with controlling the operation of said sensor; processing the acquired knowledge of the environment to estimate a state of the environment, the state of the environment including a state of the at least one mobile object, and an uncertainty in the state of the environment including an uncertainty in the state of the at least one mobile object; jointly determining a sequence of control inputs for controlling the ego vehicle, the sequence of control inputs defining a state trajectory of the ego vehicle; and a sequence of sensing instructions for the at least one sensor defining desired knowledge of the environment, wherein the sequence of control inputs is determined by solving a multivariable and multi-stage stochastic constrained optimization of a model of motion of the ego vehicle that relates the state trajectory to the sequence of control inputs, the sequence of control inputs being subject to chance constraints on admissible values ​​of the control inputs and the state of the ego vehicle defined based on the desired knowledge of the environment resulting from at least some possible transitions of the state of the moving object at different stages of the stochastic constrained optimization and corresponding uncertainties in the transition states of the moving object in the acquired knowledge; and The ego vehicle and the sensors are controlled based on the sequence of control inputs and the sequence of sensing commands.

2. The controller according to claim 1, further causing the controller to: predicting likely future measurements of one or more other mobile objects in the environment; and The predicted possible future measurements of the one or more other mobile objects are used to determine possible future transitions of the state of the at least one mobile object.

3. The controller according to claim 2, wherein: The multivariable and multi-stage stochastic constrained optimization is obtained by adjusting future transitions of the ego vehicle for the predicted possible future measurements of the one or more other mobile objects in the environment.

4. The controller according to claim 2, wherein: The predicted future measurements are selected from a grid of possible future measurements, wherein each point of the grid is associated with the measurement range it represents, such that the measurement range associated with the respective point provides the probability of that point and the quantified uncertainty of that point.

5. The controller according to claim 4, wherein: The grid of possible future measurements is determined such that the spacing between any two adjacent points in the grid is constant and all measurement ranges associated with all points in the grid have the same size, which is defined by a predetermined value.

6. The controller according to claim 4, wherein: The grid of possible future measurements is determined such that the probability of a measurement being over a respective measurement range associated with a point of the grid is equal for all points.

7. The controller according to claim 4, wherein: The grid of possible future measurements is determined such that, due to considering a grid of discrete points, the quantitative uncertainties associated with corresponding points of the grid are equal for all points.

8. The controller according to claim 7, wherein: The quantization uncertainty associated with a corresponding point of the grid is calculated as the covariance of the measurement range associated with the corresponding point.

9. The controller according to claim 1, wherein: The multivariable and multi-stage stochastic constrained optimization is associated with a cost function defined as a function of a weighted average of costs obtained by conditioning on a sequence of possible predicted future measurements.

10. The controller according to claim 9, wherein: The weights are calculated as the probability of the sequence of corresponding sequences of predicted possible future measurements according to the probabilities of the points in the grid of possible future measurements.

11. The controller according to claim 2, wherein: The multivariable and multi-stage optimization includes branching at each stage based on adjustments to the measurements of possible predictions applied to the one or more other moving objects at a time corresponding to the stage.

12. The controller according to claim 11, wherein: The branch horizon, over which adjustments for possible future measurements are made, is shorter than the problem horizon.

13. The controller according to claim 12, wherein: The length of the branch horizon is selected so that the computation time associated with the multivariable and multi-stage optimization is less than the allowed computation time.

14. The controller according to claim 13, wherein: The allowed computation time is equal to a sampling period associated with the controller.

15. The controller of claim 1 further caused to impose unanticipatory constraints to obtain decisions based solely on measurements that have been made before the decision is made.

16. The controller according to claim 15, wherein: The unanticipation constraints are configured to reduce the number of variables in multivariable and multi-stage optimizations.

17. The controller according to claim 1, wherein: The constraints include constraints on the operation of the vehicle and probabilistic constraints related to the vehicle and the environment, wherein at least one of the constraints is a collision avoidance constraint to avoid collision with one of the moving obstacles, wherein the moving obstacle is represented by a convex shape.

18. The controller according to claim 17, wherein: The controller convexifies the collision avoidance constraints by applying a linear random constraint that separates obstacles from the current vehicle position.

19. The controller according to claim 18, wherein: The linear random constraints are constructed in the following way: determining a first point on a boundary of the convex shape of the obstacle; determining a tangent plane of the obstacle shape at the first point; constructing a linear constraint that keeps the vehicle on the same side of the constraint along a prediction horizon; as well as The uncertainty is added to the estimate of the obstacle state as the uncertainty of the linear constraint.

20. The controller of claim 2, further causing the controller to predict possible future measurements by predicting possible future behavior of one or more mobile objects in the environment.

Images