High computational efficiency traffic actor trajectory representation

Abstract

This disclosure generally relates to autonomous vehicles, and more specifically, to computationally efficient techniques for representing the trajectories of traffic participants (e.g., vehicles, pedestrians, cyclists, etc.) (e.g., multi-object tracking for autonomous vehicles). An exemplary method for generating control signals for controlling a vehicle includes: obtaining a parameterized representation of the trajectory of a single object in the same environment as the vehicle; updating the parameterized representation of the trajectory of the single object based on data received by one or more sensors of the vehicle within the framework of a multi-object and multi-hypothesis tracker; and generating control signals for controlling the vehicle based on the updated object trajectory.

Description

Technical Field

[0001] This disclosure generally relates to autonomous vehicles, and more specifically, to computationally efficient techniques for representing the trajectories of traffic participants (e.g., vehicles, pedestrians, cyclists, etc.) (e.g., multi-object tracking for autonomous vehicles). Background Technology

[0002] The safe and comfortable operation of autonomous agents / agents (such as cars, drones, robots, agricultural equipment, etc.) requires a planned movement strategy that considers the potential future dynamics of other agents and objects in the surrounding environment. Since external observers generally cannot discern the intentions of other agents, autonomous agents can only infer the future behavior of their surrounding dynamic environment. Such inferences are based solely on observations of past and present movements in the environment, along with all signals that other agents may actively emit indicating their intentions, such as turn indicators.

[0003] Therefore, autonomous actors need to continuously track the motion of dynamic objects in their surrounding environment using their sensors. Since there are typically multiple dynamic objects in the actor's environment, this process is called multi-object tracking (“MOT”). MOT presents several unique challenges: the number of objects to be tracked may change as objects move in and out of the actor's field of vision; erroneous sensor readings may incorrectly indicate the presence of objects; or existing objects may not provide sensor readings due to temporary occlusion. Thus, the problem arises of how to correctly correlate sensor readings with the tracked objects to update their current motion state estimates—the so-called data association problem.

[0004] A fundamental solution to the uncertainty of data association is to simultaneously assign sensor readings to the tracked object while maintaining several different, consistent assumptions, in a so-called multiple hypothesis tracking (“MHT”) algorithm. This can effectively postpone the final decision on data association to a later point in time, when more conclusive evidence has been collected (e.g., when the occlusion problem has been resolved), during which time all possible alternatives are considered.

[0005] While multi-hypothesis tracking (MHT) algorithms demonstrate superior tracking performance for multi-object tracking (MOT), they are computationally intensive due to the potential need to store numerous associated hypotheses. Therefore, in real-time tracking systems applied to embedded hardware, MHT algorithms are currently limited to maintaining estimates of the object's current motion state, such as its position, velocity, and acceleration.

[0006] However, it is highly beneficial to track not only the current state of an object's movement but also a portion of its historical trajectory, as this can provide valuable information about the intentions of a particular actor. For example, a vehicle that has just changed lanes is less likely to change lanes again compared to another vehicle in the same state of motion that has not recently changed lanes.

[0007] If the Multiple Hypothesis Tracking (MHT) algorithm is used for trajectory tracking, the memory requirement is M*L, which is the product of the number of hypotheses (M) retained and the trajectory length (L) – i.e., the number of time steps in the tracking. Furthermore, when using moving sensors, we must consider the computational cost of converting all hypotheses used for situation interpretation, prediction, and planning into a fixed sensor coordinate system, which is L*m, where m is the number of hypotheses used for these tasks and ≤ M.

[0008] Therefore, it is desirable to generate and maintain trajectory representations that reduce both memory and coordinate transformation computation requirements, as this would enable the multiple hypothesis tracking (MHT) algorithm to also be used for trajectory tracking. Summary of the Invention

[0009] This disclosure relates to methods, electronic devices, systems, apparatuses, and non-transient storage media for generating control signals for autonomous vehicles and operating autonomous agents. Embodiments of this disclosure can represent the trajectories of traffic participants (e.g., the vehicle itself, other vehicles, pedestrians, cyclists, etc.) in a memory-efficient and computationally efficient manner. In some embodiments, the trajectory of an object is represented as a parametric representation such as a Bézier curve. In some embodiments, the trajectory of an object is represented as a generalization of a Bézier curve, i.e., a linear combination of basis functions (e.g., first basis functions, second basis functions, third basis functions, etc.) learned from data but retaining the basic properties of a Bézier curve, ensuring low memory footprint and high computational efficiency in coordinate transformation.

[0010] Using parametric representation methods like Bézier curves to characterize traffic trajectories offers numerous technical advantages. For instance, Bézier curves are well-suited for characterizing typical traffic trajectories observed from autonomous actors (e.g., smooth drive trajectories). Furthermore, historical trajectories, along with the current state of the surrounding environment, provide rich information about the intentions of other actors, thereby significantly reducing uncertainty and simplifying the planning process.

[0011] Furthermore, parameterized representations require significantly less memory than naive trajectory representations. Naive trajectory representations consist of a time series of motion states, each defined by a number of motion state parameters (i.e., position, velocity, acceleration). Therefore, in the multi-hypothesis approach, the computational cost in terms of memory and computation is M*L, the product of the number of hypotheses (M) retained for the object and the length of the tracked trajectory (L) (e.g., the number of time steps tracked). In contrast, Bézier curves, or their generalizations, are parameterized by multiple control points, thus allowing a constant number of parameters (e.g., 8 for cubic Bézier curves, 12 for Günther Bézier curves) to summarize the motion states observed over multi-second time intervals (e.g., tens to hundreds of timestamps / cycles). Bézier curves reduce the computational cost of each trajectory tracking algorithm with a tracked trajectory length (L) to a constant. In other words, embodiments of the present invention can make the trajectory tracking algorithm comparable in size to an algorithm that only tracks the current state, while providing a rich temporal context. This method can reduce memory footprint by more than 95%.

[0012] Furthermore, trajectory parameterization requires significantly fewer computational resources than naive trajectory representation. For example, to obtain a Bézier curve representation seen from different coordinate systems (e.g., due to vehicle motion), the system can move and rotate control points, i.e., curve parameters, thereby obtaining an accurate representation of the curve in a computationally efficient manner. In some implementations, the transformation of these control points can use affine transformations. In other words, their transformation is the same as the transformation of the static position of surrounding environmental features (such as traffic signs). This differs from polynomial representation, which, for example, does not allow direct transformation of its parameters in some coordinate systems. If a polynomial representation is used to fit an object's trajectory, self-motion compensation becomes difficult because the polynomial in the vehicle's coordinate system does not remain polynomial under rotation. Therefore, the system is limited to retaining a list of all measured points for self-motion compensation and densely computing polynomials to refit the trajectory using these points.

[0013] Furthermore, the parameterized representation can be updated in a computationally efficient manner. The motion model required for the prediction step in the tracking algorithm depends only on a time-invariant linear transformation of the cycle time. The observation model is linearly time-invariant when observing the object's position, velocity, and acceleration, thus the Kalman update equations for the trajectory parameters are accurate. These parameters are fully interpretable. The standard kinematic state vectors (position, velocity, acceleration) of the object along the trajectory can be reconstructed through a linear transformation. Instead of fitting the data, the update step directly incorporates the new observation data into the curve parameters. Therefore, it is unnecessary to store the observation sequence for computing the Bézier curve representation or its generalizations.

[0014] As described herein, some embodiments of this disclosure obtain a multivariate Gaussian distribution at the control points of a Bézier curve or its generalization, which, along with an adjusted motion model and measurement model, is directly introduced as a Kalman update equation for a Gaussian distribution on the kinematic state vector used in a state tracker. This transforms any state tracking algorithm into a trajectory tracking algorithm without the computational and memory costs of maintaining the state sequence that constitutes the trajectory. Simultaneously, no information is lost compared to ordinary state tracking, as the final point of the trajectory always corresponds to the current state of the object. Since Bézier curves comfortably and smoothly represent jerk-constrained trajectories, actual measured deviations in the object's state can be used to identify behavioral anomalies in other traffic participants. Compact Bézier representations are also particularly suitable as input to artificial intelligence (AI) algorithms (e.g., machine learning models such as neural networks) for contextual understanding of autonomous actors, as they summarize the actor's behavior in past traffic scenarios.

[0015] An exemplary method for generating control signals for controlling a vehicle includes: obtaining a parameterized representation of a single object trajectory in the same environment as the vehicle; updating the parameterized representation of the single object trajectory based on data received by one or more sensors of the vehicle within a framework of a multi-object and multi-hypothesis tracker; and generating control signals for controlling the vehicle based on the updated object trajectory.

[0016] In some implementations, the control signals are generated based on the updated trajectories of the object and at least one other object in the same environment as the vehicle.

[0017] In some embodiments, the method further includes providing control signals to the vehicle to control its movement.

[0018] In some implementations, the method further includes determining an object-related intent based on an updated trajectory, wherein the control signal is determined based on the intent.

[0019] In some implementations, the intent includes leaving the road, entering the road, changing lanes, crossing the road, turning, or any combination thereof.

[0020] In some implementations, the method further includes inputting the updated trajectory into a trained machine learning model to obtain an output, wherein the control signal is determined based on the output of the trained machine learning model.

[0021] In some implementations, the machine learning model is a neural network.

[0022] In some implementations, obtaining a parametric representation of the trajectory includes retrieving multiple control points from memory.

[0023] In some implementations, the method further includes transforming the obtained parametric representation to a new coordinate system based on vehicle motion.

[0024] In some implementations, the parametric representation obtained by the transformation includes transforming multiple control points of the parametric representation to a new coordinate system.

[0025] In some implementations, updating the parametric representation includes: predicting a desired parametric representation based on the obtained parametric representation and motion model; comparing the desired parametric representation with data received by one or more sensors of the vehicle; and updating the parametric representation based on the comparison.

[0026] In some implementations, the prediction of the expected parameterized representation includes determining multiple control points for the expected parameterized representation.

[0027] In some implementations, determining multiple control points for the desired parameterization includes obtaining the mean and / or covariance of the multiple control points for the desired parameterization.

[0028] In some implementations, the motion model is a linear model configured to forward the obtained parameterized representation by one time period.

[0029] In some implementations, the parameterized representation is updated based on the Kalman filter algorithm.

[0030] In some implementations, the method further includes determining whether the relevant object is abnormal based on comparison.

[0031] In some implementations, the data is first data, the updated parameterized representation is a first parametric curve representation, and the method further includes: obtaining a second updated parameterized representation based on the trajectory parameterized representation updated by the second data received by one or more sensors of the vehicle; and storing the first updated parameterized representation and the second updated parameterized representation as hypotheses related to the relevant objects.

[0032] In some implementations, the object is a traffic participant.

[0033] In some implementations, an example vehicle includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executable by the one or more processors, wherein the one or more programs include instructions to obtain a parameterized representation of the trajectory of a single object in the same environment as the vehicle; updating the parameterized representation of the trajectory of the single object based on data received by one or more sensors of the vehicle within a framework of a multi-object and multi-hypothesis tracker; and generating control signals for controlling the vehicle based on the updated object trajectory.

[0034] In some implementations, an example system for generating control signals for controlling a vehicle includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executable by the one or more processors, wherein the one or more programs include instructions to obtain a parameterized representation of the trajectory of a single object in the same environment as the vehicle; updating the parameterized representation of the trajectory of the single object based on data received by one or more sensors of the vehicle within a framework of a multi-object and multi-hypothesis tracker; and generating control signals for controlling the vehicle based on the updated object trajectory. Attached Figure Description

[0035] To better understand the different implementation methods, the following description is provided with reference to the implementation methods and the accompanying drawings, wherein the same reference numerals refer to corresponding parts throughout the drawings.

[0036] Figure 1A This demonstrates a scenario where the current states of only two vehicles are known, according to some implementation methods.

[0037] Figure 1B This demonstrates a scenario where the current states of only two vehicles are known, according to some implementation methods.

[0038] Figure 1C This demonstrates a scenario where the current state and historical trajectory of an object are known, according to some implementation methods.

[0039] Figure 1D This demonstrates a scenario where the current state and historical trajectory of an object are known, according to some implementation methods.

[0040] Figure 2 This section demonstrates an example of a process for generating vehicle control signals according to some implementation methods.

[0041] Figure 3A Examples of trajectory representations based on some implementation methods are shown.

[0042] Figure 3B Examples of trajectory representations based on some implementation methods are shown.

[0043] Figure 4 Examples of computing devices according to some implementation methods are shown. Detailed Implementation

[0044] The following description enables those skilled in the art to establish and use various embodiments. The descriptions of specific apparatuses, techniques, and applications are provided by way of example only. Different modifications to the examples described herein will be apparent to those skilled in the art, and the general principles defined herein can be applied to other examples and applications without departing from the spirit and scope of the different embodiments. Therefore, the different embodiments are not limited to the examples described and shown herein, but have the scope consistent with the claims.

[0045] This disclosure relates to methods, electronic devices, systems, apparatuses, and non-transient storage media for generating control signals for autonomous vehicles and operating autonomous agents. Embodiments of this disclosure can perform trajectory representation of traffic participants (e.g., the vehicle itself, other vehicles, pedestrians, cyclists, etc.) in a memory-efficient and computationally efficient manner. In some embodiments, the trajectory of an object is represented as a parametric representation. The parametric representation can be a generalization of Bézier curves, i.e., a linear combination of basis functions (e.g., first basis functions, second basis functions, third basis functions, etc.) learned from data and preserving the basic properties of Bézier curves, ensuring low memory footprint and high computational efficiency in coordinate transformation.

[0046] Using Bézier curves for parametric representation of traffic trajectories offers numerous technical advantages. Bézier curves are particularly well-suited for representing typical traffic trajectories observed from autonomous actors (e.g., smooth drive trajectories). Furthermore, historical trajectories, together with the current state of the surrounding environment, provide rich information about the intentions of other actors, thereby significantly reducing uncertainty and simplifying the planning process.

[0047] Furthermore, parameterized representation requires significantly less memory than naive trajectory representation. Naive trajectory representation comprises a time series of motion states, each defined by a number of motion state parameters (i.e., position, velocity, acceleration). Therefore, in the multi-hypothesis approach, the computational cost in terms of memory and computation is M*L, the product of the number of hypotheses (M) retained for the object and the length of the tracked trajectory (L) (e.g., the number of time steps tracked). In contrast, Bézier curves, or their generalizations, are parameterized by multiple control points, thus allowing a constant number of parameters (e.g., 8 for cubic Bézier curves, 12 for Günter Bézier curves) to summarize the motion states observed over multi-second time intervals (e.g., tens to hundreds of timestamps / cycles). Parameterized representation reduces the computational cost of any trajectory tracking algorithm from a scale with the tracked trajectory length (L) to a constant. In other words, embodiments of the present invention can make the scale of a trajectory tracking algorithm comparable to that of an algorithm that only tracks the current state, while providing a rich temporal context. This method can reduce memory footprint by more than 95%.

[0048] Furthermore, trajectory parameterization requires significantly fewer computational resources than naive trajectory representation. For example, to obtain a Bézier curve representation seen from different coordinate systems (e.g., due to vehicle motion), the system can move and rotate control points, i.e., curve parameters, thereby obtaining an accurate representation of the curve in a computationally efficient manner. In some implementations, the transformation of these control points can use affine transformations. In other words, their transformation is the same as the transformation of the static position of surrounding environmental features (such as traffic signs). This differs from polynomial representation, which, for example, does not allow direct transformation of its parameters in some coordinate systems. If a polynomial representation is used to fit an object trajectory, self-motion compensation becomes difficult because the polynomial in the vehicle coordinate system does not remain polynomial under rotation. Therefore, the system is limited to retaining a list of all measured points for self-motion compensation and densely computing polynomials to refit the trajectory using these points.

[0049] Furthermore, the parameterized representation can be updated in a computationally efficient manner. The motion model required for the prediction step in the tracking algorithm depends only on a time-invariant linear transformation of the cycle time. The observation model is linearly time-invariant when observing the object's position, velocity, and acceleration, thus the Kalman update equations for the trajectory parameters are accurate. These parameters are fully interpretable. The standard kinematic state vectors (position, velocity, acceleration) of the object along the trajectory can be reconstructed through a linear transformation. Instead of fitting the data, the update step directly incorporates the new observation data into the curve parameters. Therefore, it is unnecessary to store the observation sequence for computing the Bézier curve representation or its generalizations.

[0050] As described herein, some embodiments of this disclosure obtain a multivariate Gaussian distribution at the control points of a Bézier curve or its generalization, which, along with an adjusted motion and measurement model, is directly introduced as a Kalman update equation for the Gaussian distribution on the kinematic state vector used in the state tracker. This transforms any state tracking algorithm into a trajectory tracking algorithm without the computational and memory costs required to maintain the sequence of states constituting the trajectory. Simultaneously, no information is lost compared to ordinary state tracking, as the final point of the trajectory always corresponds to the current state of the object. Since Bézier curves comfortably and smoothly represent jerk-constrained trajectories, actual measured deviations in the object's state can be used to identify behavioral anomalies in other traffic participants. Compact Bézier representations are also particularly suitable as input to artificial intelligence (AI) algorithms for contextual understanding of autonomous agents, as they summarize the agent's behavior in past traffic scenarios.

[0051] The following description provides examples of methods, parameters, etc. However, it should be acknowledged that such descriptions are not intended to limit the scope of this disclosure, but are provided as examples of implementation methods.

[0052] Although the following description uses terms such as "first" and "second" to refer to different elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of application of the different embodiments described, a first illustration may be referred to as a second illustration, and similarly, a second illustration may be referred to as a first illustration. Both the first illustration and the second illustration are illustrations, but they are not the same illustration.

[0053] The terminology used in the description of the different embodiments illustrated herein is for illustrative purposes only and does not imply limitation on the related embodiments. In the description of the different embodiments illustrated and the appended claims, the singular forms include the plural forms unless the context clearly states otherwise. It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of these. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this description, the presence of the stated features, integers, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof is not excluded.

[0054] Depending on the context, the term "in the circumstances of" can be interpreted as "in the circumstances of" or "when" or "as a response to a relevant determination" or "as a response to a relevant identification." Similarly, the phrases "in the circumstances of determination" or "in the circumstances of identification of [a specific situation or event]" can be interpreted, depending on the context, as "when" or "as a response to a relevant determination" or "when [a specific situation or event] is identified" or "as a response to the identification of [a specific situation or event]."

[0055] Safe and comfortable navigation for autonomous vehicles requires predictive planning, namely the ability to accurately predict and forecast the future behavior of dynamic objects in the surrounding environment. This prediction is based on accurate estimates of the current state of dynamic objects based on past observations. Of course, such state estimates are probabilistic due to uncertainties in the measurement process or unobservable quantities such as driver intentions. State-space models are uniquely suitable for this task because they provide a robust probabilistic framework that sequentially integrates observations into estimates of the current state of dynamic objects and tracks their movement over time. The standard approach in this regard is the Kalman filter or any variations and extensions thereof. The main operational steps of the Kalman filter, some of its weaknesses, and how to improve it are described below.

[0056] Kalman filter algorithm

[0057] Given a sequence of all past observations up to time t, a single object x t The probability distribution on can be regarded as P(x)t |o t ,..,o0)≡P t|t For example, an autonomous vehicle tracking in two-dimensional space is typically represented by a state vector x. t The description includes six items: x-axis position, y-axis position, velocity v, acceleration a, yaw angle ψ, and turning rate. In its simplest case, the probability density of this vector is a Gaussian distribution. It has an average value μ t|t ∑ covariance t|t .

[0058] What is of interest is that at time t+δt, given the current state vector x t : P(x t+δt |x t The further distribution of the future state vector. To reiterate, the simplest case is a Gaussian distribution. Its average value is a linear function The covariance matrix Q(δt) and the matrix F(δt) are given, where Q(δt) depends only on δt.

[0059] In addition, the observed value o should also be considered. t For a given state vector P(o t |x t The probability of (). In the simplest case, it is also a Gaussian distribution. Its average value is a given state vector The matrix is ​​a linear function of the covariance R. As an example, for the aforementioned state vector, and a sensor that only provides position and velocity estimates, the matrix... It could be:

[0060]

[0061] and observation vector o t It includes three items: x-axis position, y-axis position, and velocity v.

[0062] With these distributions, the system can sequentially integrate new observations into a state vector X from time t+δt. t+δt A more accurate estimate. This is based on all the information obtained from previous iterations of the prediction step and subsequent application of Bayesian rules to each observation in the sequence during the update step:

[0063]

[0064] Starting from the Gaussian distribution of the parameter vector, the Gaussian likelihood of the observations, and the linear motion and observation models, these equations can be solved in closed-form, yielding the standard Kalman filter equations for the prediction step. in:

[0065]

[0066] Then, the updated Calculated as:

[0067]

[0068] Using these equations, the current state of a single object can be effectively tracked from continuous observations.

[0069] If tracking is performed from moving sensors, as is the case with autonomous vehicles, the system must compensate for the sensor movement in a process known as self-compensation. The system can perform object tracking in a fixed world coordinate system, which requires transforming observations from the moving vehicle's coordinate system to the fixed coordinate system used for prediction and update steps. However, situation interpretation and planning typically occur within the vehicle coordinate system, thus requiring the system to transform the updated state estimate back to the vehicle coordinate system.

[0070] Since tracking preferably occurs directly within the vehicle coordinate system, the system needs to convert the state estimate into the current vehicle coordinate system for measurement. Updates can then be performed directly within the vehicle coordinate system.

[0071] Multi-object tracking

[0072] The surrounding environment of an autonomous vehicle may contain more than one dynamic object tracked using a multi-object tracking (MOT) algorithm. Therefore, the sensors will report a set of detections for different objects. For example, a radar sensor will report multiple detected objects based on radar reflection, or multiple objects detected in a camera image. Typical object detection mechanisms using radar reflection, camera images, or lidar sensors operate based on a single image frame; that is, there is no explicit one-to-one correspondence between object detection i at time t and the state vector of object i estimated based on previous readings. Generally, this direct correspondence cannot be established when objects enter and leave the relevant sensor range, and object detection mechanisms may produce false positives or omissions due to occlusion or inadequate simple detection methods.

[0073] {o t Let k be the time at time t. t Object detection set, where o t、i The detection i represents time t. The state estimation of a single object depends on the precise correlation between the object detection and the tracked object in the update step of the Kalman equation. For example, consider object j at time t. state and They are different because the data association sequences are different.

[0074] Furthermore, the data association sequences of different tracked objects must be consistent. If an object detection can only be associated with one tracked object, then the data association sequences of different objects must be incoherent—a specific detection. t、i It cannot appear in the associated sequence of two different objects.

[0075] The consistency of this data association is guaranteed in the multi-object tracking algorithm prior to the update step. The ability to calculate the probability that each object detection i generates a predicted state for any object j is... Therefore, a set of k that is most likely to be consistent can be selected. t+δt Observation results {o t+δt The association with the currently tracked object is determined, and these observations are then carried over to the update step.

[0076] Multihypothesis Tracking

[0077] It is possible that at any given time step, there are several equally probable data associations. This ambiguity is particularly evident in cluttered scenarios with many obstructions, such as urban traffic in towns.

[0078] Understandably, it is unlikely that sequentially correlating observations to state estimates will correct errors in data correlations. Therefore, choosing the wrong data correlations can lead to the risk of misestimating the current state of the environment, thus making it impossible to plan safe movements.

[0079] This risk can be mitigated by removing the restriction of working only with a single consistent data set. The Multiple Hypothesis Tracking (MHT) algorithm maintains a list of the most likely consistent M data associations at each time point and processes each association independently within a branch process. In this way, data associations with similarly high probabilities and the resulting estimates of the dynamic environment's state are preserved until further observations that resolve the ambiguity are collected.

[0080] In the Multiple Hypothesis Tracking (MHT) algorithm, the consistent data associations maintained across all tracked objects are called "global hypotheses." The sequence of data associations for a single tracked object and the resulting state estimate are called "local hypotheses." Therefore, a global hypothesis comprises a set of consistent local hypotheses. A single local hypothesis can appear in multiple global hypotheses, provided consistency is not violated. The number of maintained global hypotheses is typically in the hundreds, while the number of local hypotheses is on the same order of magnitude.

[0081] To reduce memory usage, local hypotheses that are not part of any global hypothesis are removed from memory.

[0082] The difficulties of using this method

[0083] While the methods outlined above are theoretically attractive, they have some drawbacks, which this disclosure aims to address:

[0084] Typically, the state vector x is chosen based on prior knowledge of the kinematics of the tracked object. As mentioned earlier, for example, position, velocity, acceleration, heading angle, and steering rate are usually chosen as vehicle state variables. While this state definition is intuitive, it means that a nonlinear motion model with approximations must be introduced into the update equations. Therefore, for example, the x-axis position at time t+δt must be calculated as... It is nonlinear in its state variables. In fact, the most common extension of the classic Kalman filter, the so-called extended or unscented filter, is designed to allow for nonlinear motion and observation models. Furthermore, motion models often have parameters that are generally not available to the observer. For example, in the vehicle model, velocity v and steering rate... By steering angle α and wheelbase The discrepancy between the actual motion of the object and the motion model used is typically explained by the noise term Q, also known as process noise. Of course, model misrepresentation requires a larger noise term to explain it, thus making the estimation results more uncertain.

[0085] Furthermore, this state vector is defined primarily to accurately represent the vehicle's current motion state, rather than representing information derived from past observations and related to predicting the future. For example, in Figure 1A In the scenario, two vehicles are observed, and their complete kinematic state vectors represent their current states at the highway entrance and exit ramps. However, these kinematic states do not contain information to reduce future uncertainty. For both vehicles, the probability of entering or leaving the highway is the same; therefore, as... Figure 1B As shown, observers driving behind these two cars are forced to confront four equally probable future scenarios for the scene in front of them.

[0086] Since it is usually impossible to observe drivers' intentions, obtaining all or at least part of the past trajectories of traffic participants can reduce the uncertainty in predicting their future behavior and facilitate planning. The following... Figure 1C and 1D This was explained. If past patterns are available, the uncertainty surrounding the future evolution of the relevant scenarios essentially disappears.

[0087] A naive approach to trajectory tracking is to simply maintain a list of the latest L estimates of the motion state of a dynamic object. If the tracking frequency is 10 Hz, then an L = 10ΔT state vector is needed to maintain such a list over ΔT seconds. Therefore, the memory footprint is linearly related to L for each tracked object in multi-object tracking or each tracked local hypothesis in multi-hypothesis tracking. See Granstrom. The researchers (https: / / arxiv.org / abs / 1912.08718, Table 1) discuss the computational requirements in the latest implementations of the multiple hypothesis tracking (MHT) tracker.

[0088] Furthermore, if tracking is to be performed within the vehicle coordinate system, then a naive list of L-historical states of the trajectory would require computationally intensive self-compensation for each point in time.

[0089] This disclosure includes an implementation that uses the trajectory of an object over time step L as its state. However, instead of using a list, the system uses a trajectory parameterization representation with a small memory footprint independent of L, and estimates the parameters of the relevant representation using corresponding linear motion and observation models without approximating the Kalman update equation. This small memory footprint allows the relevant trajectory tracking to be used in conjunction with multi-target tracking algorithms in real-time systems running on embedded hardware.

[0090] Furthermore, the self-compensation in this representation is computationally extremely economical, as only parameter transformation is required, and therefore the computational workload does not depend on the length of the tracked trajectory.

[0091] Whether tracking a single or multiple objects, or whether tracking a single or multiple data association hypotheses, the core of every tracking algorithm following the sequential Bayesian data assimilation paradigm is similar. We need state representations, timely motion models representing the evolving states, and observation models so that we can evaluate the observation probabilities estimated for a given state. The following section details the proposed mathematical representation of a d-dimensional trajectory based on n+1 basis functions, along with the corresponding motion and observation models.

[0092] The relevant system has a time of Φ i Considering the n+1 trajectory basis functions of (t), these basis functions are arranged in an n+1 dimensional vector Φ(t)=[Φ0(t),Φ1(t),Φ2(t),......Φ n (t)] in.

[0093] The relevant system defines an (n+1)×d-dimensional control point matrix P. Each row in P corresponds to a control point in the d-dimensional dimension. The control point is a parameter of a trajectory; we use P to find the position of the parameterized trajectory at time t as c(t) = Φ. T Let P(t) be a vector of length t, where c(t) is a d-dimensional vector. The superscript T usually indicates a transformation. Clearly, P fully specifies c(t) for all t, therefore, if we want to track c(t), we only need to estimate and track P.

[0094] The self-motion compensation of the trajectory is achieved by simply changing the control points, which can be changed as trivially as any fixed point in space as the sensor moves and rotates.

[0095] The motion model of P is obtained as a time-invariant linear transformation that can be directly calculated from the basis functions.

[0096] The observation model is obtained as follows: Dynamic object sensors can typically obtain position information and time derivative information such as velocity. Due to our chosen representation, the j-th time derivative of the trajectory is used as our control point d. j / dt j c(t) = H j The linear transformation of P is given, where H j =d j / dt j Φ T (t).

[0097] Figure 2 This document illustrates examples of processes for generating vehicle control signals according to some implementations. For example, process 200 is implemented using one or more electronic devices implementing a software platform. In some examples, process 200 is implemented using one or more electronic devices on an autonomous vehicle (e.g., the vehicle itself). In some implementations, process 200 is implemented using a client-server system, and process blocks of process 200 are divided in any manner between the server and one or more client devices. Therefore, although parts of process 200 are described herein as being implemented by specific devices, it is understood that process 200 is not so limited. In process 200, some process blocks may be selectively combined, the order of some process blocks may be selectively changed, and some process blocks may be selectively omitted. In some examples, additional steps may be implemented in conjunction with process 200. Therefore, the operations shown in the figures (described in more detail later) are exemplary in nature and should not be considered limiting.

[0098] In process block 202, estimates of parametric trajectory representations of a set of objects are obtained (e.g., by one or more electronic devices). These objects may be traffic participants in the same surrounding environment as the relevant vehicle, and may include vehicles, pedestrians, cyclists, drones, animals, etc.

[0099] The d-dimensional parameterized trajectory representation includes n+1 time-based basis functions and n+1 d-dimensional control points. The n+1 time-based basis functions τ are arranged in an n+1-dimensional vector:

[0100] Φ(τ)=[φ0(τ) φ1(τ) φ2(τ) … φ n (τ)]

[0101] The system represents a trajectory of length ΔT at intervals τ∈[0,1], which can always be achieved by redistributing dτ=dt / ΔT for any ΔT.

[0102] Then, at any time t, in d dimensions with a time span of ΔT, a point on the object's past trajectory is given by a linear combination of n+1 basis functions, and the n+1 current control points are arranged in d dimensions as an (n+1)xd matrix P. t Each row corresponds to a control point:

[0103] c(τ)=Φ T (τ)P t

[0104] Here, P is a point along the track being tracked, where τ = 0 corresponds to t - ΔT, and τ = 1 corresponds to the current time t. t The control points are curve parameters that change over time as the object moves. They transform in the same way as points in space when the coordinate system moves, hence the name "control point."

[0105] An example of such basis functions is the Bernstein polynomial.

[0106]

[0107] In this case, curve c(τ) is a known Bézier curve, and control point P t There is a particularly intuitive interpretation. We will provide a running example with reference to this specific case. However, those skilled in the art will understand that all statements also apply to basis functions in general. In particular, basis functions can be optimized to accurately characterize empirically measured trajectories.

[0108] In order to use this trajectory representation in Kalman filtering and to specify the distribution of trajectory parameters, the vector... Introducing a state representation used at time t. t It is P t The matrix of control points (n+1)×d is formed by simply concatenating the terms, with the first two terms corresponding to p. o The last two terms correspond to p1, and so on. It can be assumed that the distribution of this state is through... The model is based on a multivariate Gaussian distribution, with a mean of μ. t The covariance is ∑ t Because of P t and x t For the equivalence of the two, the following explanation uses the most convenient notation available at the time.

[0109] Figure 3AThis section demonstrates an example using two-dimensional (i.e., n=3 and d=2) solid Bézier curves. The parameters of the solid Bézier curve are determined by four control points: p0, p1, p2, and p3. Each control point is a two-dimensional point represented by two numerical values. The state vector x of this type of trajectory... t The size is (n+1)d = 8. The diagram also indicates probability-related issues. Annotated points and corresponding ellipses represent x. t The average value μ corresponding to the control point item t And a 95% confidence interval. The solid line represents the average estimate of the trajectory. The dashed line corresponds to the sample trajectory. Where, x t From the average value of μ t The covariance is ∑ t Sampling is performed from a multivariate Gaussian distribution.

[0110] Then, process block 206 performs a data assimilation step by updating the local hypothesis set based on data received by one or more sensors of the autonomous system. Self-compensation is implemented in process block 206a. For parameterized trajectory representation,

[0111] Only the control points, i.e., the state vectors, need to be transformed. This requires significantly less computation than transforming the kinematic state vector list. Since this is a linear transformation, we can directly apply it to the state density parameters. Assuming the reference frame moves by a d-dimensional vector Δo and rotates through a d×d matrix R, the relevant system can first transform the covariance matrix of our state density. This covariance matrix is ​​only affected by the rotation R:

[0112]

[0113] The mean of the state density is first converted to homogeneous coordinates; that is, we introduce an extra dimension to the control points, which is a constant and equal to 1. The homogenized mean vector is calculated as follows:

[0114] μ h =[μ1,…μ d ,1,μ d+1 ,…μ 2d ,1,…,μ nd ,…μ (n+1)d ,1]

[0115] Then, we introduce a d×(d+1) matrix.

[0116] T = [R - RΔo]

[0117] And can convert the average vector to

[0118]

[0119] As mentioned earlier, I n It is an n×n identity matrix.

[0120] In process block 206b, a specific motion model is used to predict the trajectory extension of the trajectory control points.

[0121] Suppose that at time t, the system has m>n+1 trajectory samples c(τ). i ) Along the trajectory at different times τ i , i∈{1,...,m} at τ0=0 and τ m The sampling is performed between 1 and 1. These samples can be arranged into the number of rows of an m x d matrix C. This forms an m x (n+1) matrix B such that row i of B corresponds to Φ(τ). i Then, the control points can be estimated as a least-squares fit of the trajectory sample sampling:

[0122] P t = (B T B) -1 B T C

[0123] Now, we focus on the motion model of the control points, that is, we are interested in their movement along the trajectory. For this purpose, we can construct another mx(n+1) matrix B' such that row i of B' corresponds to Φ(τ). i +δ t / ΔT). The system can then obtain the converted control point:

[0124]

[0125] For a pair of control points, the linear motion model F(δt) depends only on the choice of basis functions. Correspondingly, for the state vector, the relevant system can be found to:

[0126]

[0127] Among them, I d It is the identity matrix in d dimensions.

[0128] In particular, the endpoint of the movement trajectory will be c(1) = Φ T (1)P t+δt . Figure 3B This demonstrates a running example of a cubic Bézier curve.

[0129] and

[0130] Matrix P t That is, the parameterized trajectories of control points p0, p1, p2, and p3 at time t are transmitted to matrix P. t+δtThat is, control points p′0, p′1, p′2, and p′3 at time t+δ t The trajectory is parameterized. Based on the structure, up to time t, the new trajectory estimate completely follows the old estimate.

[0131] Then, the prediction step in the Kalman filter equation is written as

[0132]

[0133] In process block 206d, the probability of the current sensor reading is calculated based on the predicted trajectory extension set using local assumptions. Typical object detection devices provide measurements of object position, velocity, and possible acceleration. These kinematic measurements are readily available. The derivative of i = th is:

[0134]

[0135] For trajectory tracking, natural selection considers the nearest point, i.e., the last trajectory point when τ = 1. Based on the perceptual information available for each object, we can construct an observation matrix H.

[0136] For example, if the system tracks the trajectory of an object in two dimensions, and the sensor provides the position and corresponding velocity, then an observation vector ot = [x, y, v] is formed. x υ y We use row Φ(1) and Construct a 2×(n+1) matrix H o And obtain the matrix H required for the Kalman equation update step from the above description:

[0137]

[0138] Thus, the system has all the necessary parts to track the trajectory ΔT of an object within a certain time range, and can always update the trajectory with the most recent observation.

[0139] In process block 206d, the probability of the current sensor reading calculated in the previous step is used to construct the global hypothesis with the highest probability in M.

[0140] In process block 206e, all unused local hypotheses among the global hypotheses with the highest probability in M ​​are removed from memory.

[0141] Then, process 206d feeds the global hypothesis with the highest probability in M ​​and the corresponding set of local hypotheses back to process block 208 for further processing.

[0142] In process block 208, the system determines vehicle control signals based on the updated trajectory of the object. In some embodiments, the system may determine the intent associated with the object based on the updated trajectory of the object, and accordingly determine the vehicle control signals. See reference... Figure 1A As illustrated by -D, historical trajectories can be used, for example, to determine the intentions of traffic participants (e.g., leaving a highway, entering a highway, crossing a road, turning, etc.). Based on the intentions associated with the object, the system can determine vehicle control signals to avoid collisions with the relevant object. These control signals can be provided or transmitted to the vehicle for vehicle control (e.g., maintaining speed, accelerating, decelerating, changing direction, etc.).

[0143] In some implementations, the system can use the updated trajectory for different downstream analyses. For example, the system can input the updated trajectory into a machine learning model to understand the situation. The machine learning model can be configured, for example, to receive object trajectories and identify object intentions, identify anomalous behavior, predict future trajectories, etc. Compact parameterized representations are uniquely suited for artificial intelligence (AI) algorithms that understand situations in autonomous vehicles because they summarize the past behavior of traffic participants in the context of traffic scenarios. Due to the compactness of Bessel representations, compact machine learning models can be trained computationally efficiently, and the trained models can provide rapid analysis. The machine learning models described herein include any computer algorithm that is automatically improved through experience and the use of data. The machine learning models can include monitored models, unmonitored models, semi-monitored models, self-monitored models, etc. Examples of machine learning models include, but are not limited to: linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, neural networks, k-means, random forests, dimensionality reduction algorithms, gradient boosting algorithms, etc.

[0144] In some implementations, the system, as described above, stores multiple hypotheses for an object. Each hypothesis for the object includes a Bézier curve representation (e.g., control points that set parameters for the curve). Therefore, an updated Bézier curve representation can be stored as one of multiple hypotheses associated with the object.

[0145] In some implementations, the system evaluates the updated trajectory of an object to determine whether the object's behavior is abnormal. Since Bézier curves are smooth curves that well characterize typical driving trajectories (e.g., comfortably and smoothly characterizing trajectories with limited agility), the actual measured deviation of the object's state can be used to detect abnormal behavior in other traffic participants. In some implementations, the system determines the deviation between the expected curve (e.g., obtained in process block 206b) and the actually observed trajectory and compares the deviation to a predetermined threshold. If the threshold is exceeded, the system can determine that the object is exhibiting abnormal behavior (e.g., reckless driving). Based on the identified anomaly, the system can generate a control signal (e.g., to move away from the abnormal object).

[0146] In addition to the techniques described with reference to process 200, which use Bézier curves to represent the trajectories of traffic participants other than the vehicle itself, these techniques can also be applied to represent the trajectory of a vehicle tracking itself using Bézier curves. Furthermore, while the techniques described with reference to process 200 involve the use of Bézier curves, it should be understood that the Bézier curve representation can be replaced by any linear combination of basis functions (e.g., first basis functions, second basis functions, third basis functions, etc.). In some implementations, machine learning models such as neural networks or Gaussian processes can be used to compute the basis functions.

[0147] The operations described in this article can be selected by... Figure 4 The component implementation is described in the text. Figure 4 An example of a computing device according to one embodiment is shown. Device 400 may be a computer host connected to a network. Device 400 may be a client computer or a server. Figure 4 As shown, device 400 can be any suitable microprocessor-based device type, such as a personal computer, workstation, server, or portable computing device (portable electronic device) such as a mobile phone or tablet computer. The device may include, for example, one or more processors 410, an input device 420, an output device 430, a storage device 440, and a communication device 460. Input device 420 and output device 430 generally correspond to the aforementioned device and can be connected to or integrated into a computer.

[0148] Input device 420 may be any device suitable for providing input, such as a touch screen, keyboard or auxiliary keyboard, mouse or voice recognition device. Output device 430 may be any suitable device suitable for providing output, such as a touch screen, haptic device or speaker.

[0149] Memory 440 can be any suitable means of providing storage, such as electrical memory, magnetic memory, or optical memory, including random access memory (RAM), cache, hard disk, or removable storage disk. Communication means 460 can include any suitable means of transmitting and receiving signals over a network, such as a network interface chip or network interface device. Computer components can be connected in any suitable manner, such as via a physical bus or wirelessly.

[0150] The software 450, which can be stored in memory 440 and executed by processor 410, may include, for example, programs that embody the functions of this disclosure (the functions being embodied, for example, by the aforementioned means).

[0151] Software 450 may also be stored and / or transmitted in any non-transitory computer-readable storage medium so as to be used, or used in conjunction with, an instruction execution system, apparatus, or device, as described above, to invoke instructions associated with the software from the instruction execution system, apparatus, or device, and to execute the instructions. Within the scope of this disclosure, a computer-readable storage medium may be any medium, such as memory 440, which may contain or store a program used, or used in conjunction with, an instruction execution system, apparatus, or device.

[0152] The software 450 can also be transmitted in any transmission medium for use by or in conjunction with an instruction execution system, apparatus, or appliance as described above, to acquire and execute system instructions associated with the software. In the context of this disclosure, the transmission medium can be any medium used for program communication, import, or transmission by or in conjunction with an instruction execution system, apparatus, or appliance. Transmission via a readable medium may include, but is not limited to, electronic transmission media, magnetic transmission media, optical transmission media, electromagnetic or infrared transmission media, and wired or wireless transmission media.

[0153] Device 400 can connect to a network of any type applicable to any interconnected communication system. The network can implement any applicable communication protocol and can be secured by any applicable security protocol. The network may include any applicable setup, such as a wireless network connection, T1 or T3 line, cable network, digital subscriber line (DSL), or telephone line, enabling network signal transmission and reception.

[0154] The device 400 can implement any operating system suitable for operation on a network. The software 450 can be written in any applicable programming language such as C, C++, Java, or Python. In different embodiments, the application software embodying the functions of this disclosure can be used in various configurations, such as client / server setups or as a web-based application or web service via a web browser.

[0155] Although the present disclosure and its embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to be included within the framework of the present disclosure and its embodiments as defined by the claims.

[0156] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form of the published patent application. In view of the foregoing teachings, many modifications and variations are possible. The selection and description of the relevant embodiments are for the best interpretation of the principles of these techniques and their practical application. This allows those skilled in the art to best utilize these techniques and various modified embodiments suitable for specific applications.

Claims

1. A method for generating control signals for controlling a vehicle, the method comprising: Obtain a parametric representation of the trajectory of a single object in the same environment as the vehicle; Within the framework of a multi-object and multi-hypothesis tracker, the parametric representation of the trajectory of a single object is updated based on data received by one or more sensors of the vehicle. as well as Control signals for controlling the vehicle are generated based on the updated object trajectory. The updating of the parameterized representation includes: Based on the obtained parametric representation and motion model, predict the expected parametric representation; The expected parametric representation is compared with data received by one or more sensors of the vehicle; and The parameterized representation is updated based on the comparison. The motion model is a linear model configured to shift the obtained parameterized representation forward by one time period.

2. The method according to claim 1, wherein, The control signal is generated based on the updated trajectories of the object and at least one other object in the same environment as the vehicle.

3. The method according to claim 1 or 2, further comprising: Provide control signals to the vehicle to control its movement.

4. The method according to claim 1 or 2, further comprising: The updated trajectory determines the object-related intent, wherein the control signal is determined based on the intent.

5. The method according to claim 4, wherein, The intent includes leaving the road, entering the road, changing lanes, crossing the road, turning, or any combination thereof.

6. The method according to claim 1 or 2, further comprising: The updated trajectory is input into a trained machine learning model to obtain an output, wherein the control signal is determined based on the output of the trained machine learning model.

7. The method according to claim 6, wherein, The machine learning model is a neural network.

8. The method according to claim 1 or 2, wherein, Obtaining a parametric representation of the trajectory involves retrieving multiple control points from memory.

9. The method according to claim 8, further comprising: The parametric representation obtained based on vehicle motion is transformed into a new coordinate system.

10. The method according to claim 9, wherein, The parametric representation obtained by the transformation includes transforming multiple control points of the parametric representation to a new coordinate system.

11. The method according to claim 1 or 2, wherein, Predicting the expected parametric representation involves determining multiple control points for the expected parametric representation.

12. The method according to claim 11, wherein, Determining multiple control points for the desired parametric representation includes obtaining the mean and / or covariance of multiple control points for the desired parametric representation.

13. The method according to claim 1 or 2, wherein, The parameterized representation is updated based on the Kalman filter algorithm.

14. The method according to claim 1 or 2, further comprising: The comparison is used to determine whether the relevant object is abnormal.

15. The method according to claim 1 or 2, wherein, The data mentioned is the first data, and the updated parameterized representation is the first parameterized curve representation, which also includes: Based on second data received from one or more sensors of the vehicle, the obtained trajectory parameterization representation is updated to obtain a second updated parameterization representation; and The parameterized representations updated for the first and second times are stored as object-related hypotheses.

16. The method according to claim 1 or 2, wherein, The objects referred to are traffic participants.

17. A vehicle comprising: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the one or more programs include the following instructions: Obtain a parametric representation of the trajectory of a single object in the same environment as the vehicle; Within the framework of a multi-object and multi-hypothesis tracker, the parametric representation of the trajectory of a single object is updated based on data received by one or more sensors of the vehicle; and Control signals for controlling the vehicle are generated based on the updated object trajectory. The updating of the parameterized representation includes: Based on the obtained parametric representation and motion model, predict the expected parametric representation; The expected parametric representation is compared with data received by one or more sensors of the vehicle; and The parameterized representation is updated based on the comparison. The motion model is a linear model configured to shift the obtained parameterized representation forward by one time period.

18. A system for generating control signals for controlling a vehicle, comprising: One or more processors; Memory; as well as One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, wherein the one or more programs include the following instructions: Obtain a parametric representation of the trajectory of a single object in the same environment as the vehicle; Within the framework of a multi-object and multi-hypothesis tracker, the parametric representation of the trajectory of a single object is updated based on data received by one or more sensors of the vehicle; and Control signals for controlling the vehicle are generated based on the updated object trajectory. The updating of the parameterized representation includes: Based on the obtained parametric representation and motion model, predict the expected parametric representation; The expected parametric representation is compared with data received by one or more sensors of the vehicle; and The parameterized representation is updated based on the comparison. The motion model is a linear model configured to shift the obtained parameterized representation forward by one time period.

Images