Methods for managing trajectory segments in a particle filter estimation framework

Abstract

This invention relates to a method for managing trajectory segments in a particle filter estimation framework, comprising the steps of: depending on a previous trajectory segment (T p,t The list of ) is used to perform trajectory segment prediction, thereby determining the remaining trajectory segments and new trajectory segments (T) new ); for initializing the new trajectory segment (T) new The new measurement results are sampled to determine the number of estimated new trajectory segments (N). eti ); depends on the previous trajectory segment (T) p,t The list determines the number of remaining trajectory segments (N). p,t ); depends on the number of new trajectory segments estimated (N) eti ), the number of remaining trajectory segments (N) p,t ) and memory limits (N max ) to determine the number of new trajectory segments to be sampled (N) new ) and the number of updated surviving trajectory segments (N) p,t+1 ); depends on the determined number (N) of the updated surviving trajectory segments. p,t+1 ), from the remaining trajectory segment (T) p,t The list of updated surviving trajectory segments (T) p,t+1 Sampling is performed; and the number (N) depends on the determined number of new trajectory segments. new ), from uncorrelated measurement results, for the new trajectory segment (T) new Sampling is performed.

Description

Technical Field

[0001] This invention relates to a method and control unit for managing tracklets in a particle filter estimation framework, as well as a control method. Existing technology

[0002] One of the essential functions of at least partially autonomous vehicles is estimating their dynamic world, which includes various traffic participants within complex driving scenarios. To achieve this within a particle filter estimation framework, dynamic grid maps are one possible way to combine and estimate multi-sensory information at an intermediate level. Therefore, individual particle filter-based estimators (so-called trajectory segments) are used to stabilize the accuracy of position estimation for dynamic cells in the dynamic grid map and to estimate the shape of objects.

[0003] Theoretically, the number of track segments may grow beyond the available memory limit. While track segments should be copied and removed based on verification metrics such as the probability of existence, the maximum number of track segments should be fixed within the memory limit.

[0004] Therefore, an improved method is needed for managing trajectory segments within a particle filter estimation framework. Summary of the Invention

[0005] According to one aspect, a method for managing trajectory segments in a particle filter estimation framework includes the following steps: performing trajectory segment prediction based on a list of previous trajectory segments to determine persistent trajectory segments and new trajectory segments; sampling new measurements used to initialize new trajectory segments to determine the number of estimated new trajectory segments; determining the number of persistent trajectory segments based on a list of previous trajectory segments; determining the number of new trajectory segments to sample and the number of updated persistent trajectory segments based on the number of estimated new trajectory segments, the number of persistent trajectory segments, and a memory bound; sampling updated persistent trajectory segments from the list of persistent trajectory segments based on the determined number of updated persistent trajectory segments, and sampling new trajectory segments from unassociated measurements based on the determined number of new trajectory segments.

[0006] Preferably, the particle filter estimation framework is configured to perform particle filter-based dynamic grid map estimation. More preferably, the dynamic grid map estimation includes multi-channel measurement processing. Multi-channel measurement processing is a multi-layer particle filter-based tracking solution—MLPT.

[0007] As used herein, the term "raster map" describes a map in which the surrounding space is subdivided into rectangular raster cells. Preferably, each raster cell is an independent building block of that space that stores attributes such as occupancy probability. The occupancy probability thus indicates the occupancy value of the raster cell.

[0008] Preferably, each trajectory segment is described by dual motions (static and dynamic).

[0009] For particle filter-based dynamic grid map estimation, the following steps are included. In the first step, raw sensor data (also known as raw measurements) of the target object is determined through multiple sensor interfaces. This raw sensor data is preprocessed and transformed into a compact, particularly medium-level, data structure. Additionally, the compact sensor data or compact measurement results are determined depending on the determined raw sensor data. Therefore, each sensor is described by its own measurement model. For example, both LiDAR point clouds and stereo vision images are transformed into a more compact Stixel-based representation. Stixel models the surrounding environment as vertically oriented rectangles that can incorporate attributes such as position, depth, size, and semantic information of the target object (specifically, individual object portions of the target object). In different examples, the radar position of the target object is compressed into a set of pre-filtered target points, which include the target object's Cartesian position and velocity.

[0010] In the second step, compact sensor data is integrated into separate evidence grid channels. Preferably, the evidence grid channel is defined as an independent 2D grid representation that accumulates a set of sensor data (in other words, sensor observations) from one or more sensors. For example, two grid channels are identified: a placeholder grid channel and a semantic grid channel. More preferably, the additional grid channel uses the gradient or height attribute of the target object.

[0011] For example, the measurement occupancy grid channels are determined by accumulating distance measurements from all available sensors over a given fixed time interval. New measurements are then incorporated into the occupancy grid using Dempster-Shafer evidence theory. Thus, each grid cell is described by a belief mass of whether it is occupied, free, or unknown, and each grid cell can be transformed into a regular occupancy probability using a Binney transformation.

[0012] For example, the measurement semantic raster channel combines the semantic information provided by both the Stixel transform and the compressed radar position. Thus, the projected object labels are determined by the measurement semantic raster channel. Each projected object label identifies a given object category, such as pedestrian, bicycle, or vehicle, and has an associated confidence score. Due to memory constraints, after integrating the object labels into the semantic channel, only the top object label is retained cell-by-cell based on its highest cumulative score, rather than storing a complete histogram of all cumulative object labels cell-by-cell.

[0013] Preferably, the two measurement grid channels are described with the same size and resolution. Furthermore, the two channels are aligned in time. The same time interval is used to integrate the input measurements and the spatial data, where a given region in world coordinates is projected onto the same cell index in both grids.

[0014] In the third step, a hybrid particle filter-based estimation is performed depending on the evidence grid channels. The hybrid filter-based estimation includes self-localizing trajectory segments and multi-channel grid estimation. In the particle filter-based estimation, at each specific time point, a probability distribution is approximated by a set of individual particles, where each particle represents an object hypothesis about the grid cell state and has an assigned weight based on how well the particle matches the measurement result.

[0015] In existing grid-based particle filters, the grid cell state is approximated using a set of samples. Particles are not permanently associated with a grid cell state but propagate according to their own motion model and are subsequently reassigned to new destination grid cells. Then, for all particles, weights proportional to the occupancy belief of that new destination grid cell are determined. When an occupancy of the destination grid cell is sensed, higher weights are assigned to those particles. Typically, particles predicted to be in the middle of a larger target object can be assigned to any of the occupied cells and still be rewarded with a high weight. In other words, particles receive their weights without knowing their own position relative to the assumed object. Preferably, the determination of weights also depends on additional knowledge about the shape of the target object. It is assumed that a given dynamic grid cell is part of a larger target object hypothesis. Therefore, in addition to its position and velocity, the dynamic grid cell is described by its relative position to the object landmark. The object landmark is initialized by selecting a random set of points from the same target object shape. To account for changes in the geometry of the target object, the relative distance from the grid cell to its selected landmark must also be updated recursively at each measurement iteration.

[0016] For each newly measured grid cell, a fixed set of particles is determined. This set of particles will represent a trajectory segment—an estimator based on individual particle filters. Therefore, instead of maintaining and updating a large set of particles for the entire grid map, multiple smaller, independent groups of particles are organized into trajectory segments. Finally, different trajectory segments, as part of the same target object assumption, are described by different combinations of random landmarks selected from the same target object contour, which indicates the contour of the target object. However, all particles included in a trajectory segment are initialized using the same fixed constellation of landmarks pre-selected to describe the state of the trajectory segment.

[0017] Multichannel raster estimation involves integrating structured measurements into the raster channels. For each raster cell location at each time point, an appearance vector is described, which combines the original measurements from the occupancy and semantic channels. Therefore, this appearance vector is entirely described by the quality for occupied cells, the quality for idle cells, and the semantic label.

[0018] Once new measurements are received, the following steps recursively estimate the dynamic state of each trajectory segment and each grid cell:

[0019] By considering the elapsed time and the estimated state of the particles in previous particle filter cycles, the new positions of these particles in the grid cells are predicted. A uniform motion model is assumed, where modeling error is accounted for by perturbing each propagated sample with a random noise component.

[0020] Preferably, the landmark prediction is implicitly accomplished through particle prediction, since the landmark is conditioned on the particle state and follows the particle's motion.

[0021] In another step, each predicted sample is assigned a new importance weight. This step incorporates information from the measurement results into the particle distribution by assigning weights to particles that are proportional to the likelihood matching the observation. For a given measurement result at a specific time and the predicted state from the preceding context, the measurement model comprises three components: measurement unit likelihood, landmark-based likelihood, and semantic likelihood.

[0022] The measurement unit likelihood is based on the positional error, which is the distance between the measurement result and the nearest particle, while the landmark likelihood depends on the distance between the particle landmark and the target object contour, representing a metric used for shape alignment. Furthermore, given the semantics of the particles, the semantic likelihood is defined by a dissimilarity metric.

[0023] To update the particle landmarks previously defined according to the Rao-Blackwellisation process, 2×2 Kalman filters (one for each landmark) were used. The state estimated by each Kalman filter is a 2D position. Furthermore, by using the Dempster-Shafer combination rule, the associated measured mass is used to update the occupied mass and the idle mass of each particle.

[0024] Based on the updated particles, the grid cell state and trajectory segment state are estimated. These estimates are weighted averages of the particle states. To calculate the grid cell state, all trajectory segments projected into the same cell are used, regardless of which trajectory segment they belong to. However, the trajectory segment state is estimated based on its corresponding particle, regardless of which cell it is projected into.

[0025] A general-purpose stochastic resampling algorithm with linear complexity is used to resample particles after normalizing the particle weights for each trajectory segment. The algorithm selects a new set of particles from the previous set by considering their importance weights and thus replacing particles with lower weights. Preferably, a selective resampling strategy is implemented, where particle resampling is triggered only when particle diversity falls below a predetermined threshold.

[0026] Preferably, at the end of each grid map particle filter iteration t At that time, the final list of trajectory segments H t Including the remaining trajectory segments A subset of the original trajectory, and the newly initialized trajectory segment (also known as the new trajectory segment). Another subset.

[0027] Therefore, this list of trajectory segments is defined by the following equation:

[0028] .

[0029] List of Remaining Track Segments H t,p Through time t Transmission to time t This was obtained by resampling from all remaining trajectory segments (+1). New trajectory segment list. H h,b By measuring each grid cell c A pre-determined set of initial weights Obtained through sampling.

[0030] As long as the number of track segments has not reached the maximum memory capacity, a new track segment is appended to the existing list of new track segments. However, when the maximum memory capacity is reached, the method ensures that, through sampling, the list of new track segments will have a balanced ratio between new and existing track segments. In these extreme cases, the maximum allowed number of new track segments is a parameter of the system and is set to be less than 20% of the total available space in the list.

[0031] Preferably, the trajectory segment occupancy quality The site mass of its particles was determined. The weighted average, where It is the normalized particle weight.

[0032]

[0033] Track segment occupancy mass is used to determine the extent to which the measured cell is covered by the track segment below it.

[0034] Preferably, the intensity of the grid cell is based on cell quality. D c Determined as the placeholder quality for all its trajectory segments The sum of .

[0035]

[0036] N h,c This indicates the number of trajectory segments falling into grid cell c. The quality-based intensity is interpreted as the expected number of target objects within that cell. It also provides a quantitative value regarding the degree to which the cell is covered by trajectory segments.

[0037] Preferably, the initial weights Indicates new trajectory segment The probability that a new trajectory segment needs to be initialized in a given cell c. This weight must be proportional to the need to initialize the new trajectory segment. Therefore, the initial weight is determined as follows:

[0038]

[0039] It is the measured occupancy mass of unit c, and D c It is strength based on mass.

[0040] Assuming the system memory is finite, what is the maximum value that can be created for a trajectory segment? N max Additionally, the maximum number of newly accepted trajectory segments is... N new_max . N new_maxThis is the maximum number of new track segments used to allocate new track segments when the memory is full (or in other words, there aren't many empty slots available). This means that even when the memory is full, some of the old track segments will be allocated a constant number of times. N new_max The new trajectory segment is replaced. The goal is to target [the target] within the established memory constraints. t The next particle filter iteration determines the new number of remaining trajectory segments. N p_updated And the new number of new trajectory segments N new .

[0041] The estimated number of trajectory segments to be initialized without any memory constraints. N eti By satisfying the conditions The number of new trajectory segments is determined by counting all grid cells. Without memory limitations, new trajectory segments are initialized in all grid cells. However, with the limits defined above, the number of new trajectory segments... N new It is determined by the following equation:

[0042]

[0043] N es It is the number of empty slots available for adding new trajectory segments, and it is defined as:

[0044]

[0045] in N p,i It represents the total number of trajectory segments used in the current particle filter iteration.

[0046] The number of updated residual trajectory segments N p,t+1 It is determined by the following equation:

[0047] .

[0048] If the number of remaining trajectory segments is close to the maximum allowed number of trajectory segments, it is preferable to allocate a fixed number, such as 100, to new trajectory segments and allocate the remaining number, such as 900 = 1000(max) - 100(new), to updated remaining trajectory segments.

[0049] If the number of remaining trajectory segments is much smaller than the maximum allowed number (e.g., N max =1000 Np,t If the space is 90, then that available space can be utilized. However, at the same time, considering factors such as... N eti , N es and N new_max Other parameters, such as these. This is indicated by the following constraints:

[0050] .

[0051] Therefore, a tracking solution for trajectory segments is provided that automatically adapts to the estimated number of trajectory segments based on the available memory on the computer device. In other words, memory complexity is limited to a fixed capacity, which is given as a parameter.

[0052] Therefore, a hierarchical memory structure for particles, trajectory segments, objects, and grid maps is provided.

[0053] In a preferred embodiment, the method includes the following steps: verifying trajectory segments based on a verification metric to identify invalid trajectory segments and removing them from a list of previous trajectory segments.

[0054] In a preferred embodiment, the verification metrics include the age of the trajectory segment and / or the probability of the trajectory segment's existence.

[0055] In a preferred embodiment, the method includes the following steps. Sampling the updated surviving trajectory segments includes sampling the index of the new trajectory segment from the list of previous trajectory segments, and sampling the new trajectory segments includes sampling the new measurement index used to initialize the new trajectory segments.

[0056] In a preferred embodiment, the method includes the step of performing trajectory segment prediction, which includes determining trajectory segment weights for retained trajectory segments and new trajectory segments.

[0057] Therefore, a hierarchical propagation of the object's existence probability is provided. The object's existence probability is given as the sum of the existence probabilities of all its trajectory segments. The existence probability of a trajectory segment is described by the sum of the weights of all particles. Changes in particle weights are reflected in changes in the trajectory segment's existence probability, and thus in the object's existence probability.

[0058] In a preferred embodiment, the trajectory segment is an estimator based on an individual particle filter that indicates a group of particles.

[0059] In a preferred embodiment, determining the number of new trajectory segments includes: determining the number of empty slots, which is the number of empty slots minus the number of remaining trajectory segments from the memory limit; and selecting a higher number from the maximum number of new trajectory segments and the number of empty slots, and selecting a lower number from the higher number and the estimated number of new trajectory segments.

[0060] In a preferred embodiment, determining the number of updated surviving track segments includes selecting a lower number from the difference between the number of surviving track segments and the memory limit and the number of new track segments.

[0061] Therefore, one of the challenges is to design a scalable (new sensors can be added) and flexible (the estimation solution can be easily decoupled or adapted to different sensor combinations) dynamic environment estimation component. Simultaneously, by utilizing remaining sensors to generate the correct model, environmental perception should be able to handle situations where part of the sensor system fails. Finally, we need an estimation solution that can automatically "adapt" to available computational resources and memory.

[0062] According to one aspect, as described herein, the control unit is configured to perform a method for managing trajectory segments within a particle filter estimation framework.

[0063] In a preferred embodiment, as described herein, a computer program including instructions is provided that, when executed by a computer, causes the computer to perform a method for measuring boundary performance.

[0064] In a preferred embodiment, as described herein, a computer-readable data carrier is provided having a computer program stored thereon.

[0065] According to one aspect, as described herein, a method for classifying objects in image data includes the steps of: receiving image data using a tracking method performed by a particle filter estimation framework and classifying objects in the image data, wherein the tracking method includes a method for managing trajectory segments.

[0066] In image recognition, objects in an image are detected and classified. For example, in the field of autonomous driving, objects are classified into categories such as streets, traffic participants, and the environment. For multiple subsequent images, a tracking method is used to re-detect objects from different images. Preferably, this tracking method is based on trajectory segments, which represent an estimator based on individual particle filters.

[0067] Therefore, the improved methods for managing trajectory segments led to improved methods for classifying objects in image data.

[0068] According to one aspect, a control method for a robot that is at least partially autonomous includes the steps of: receiving sensor data from the robot; performing image recognition on the received sensor data, including performing a method for managing trajectory segments in a particle filter estimation framework as described herein; and controlling the robot that is at least partially autonomous depending on the sensor recognition performed.

[0069] Preferably, the sensor data pertains to objects in the environment of at least partially autonomous robots.

[0070] Preferably, the sensor data includes lidar data, radar data, thermal data, ultrasonic data and / or image data, especially camera data.

[0071] Preferably, the at least partially autonomous robot includes at least partially autonomous vehicles. Alternatively, the at least partially autonomous robot can be any other mobile robot, such as those that move by flying, swimming, diving, or walking. In one example, the at least partially autonomous robot could be a lawnmower or a cleaning robot.

[0072] The subject matter of the invention will be explained in more detail below with reference to preferred exemplary embodiments, which are illustrated in the accompanying drawings, in which:

[0073] Figure 1 A schematic diagram illustrating the relationship between objects, trajectory segments, and particles is shown.

[0074] Figure 2 A schematic diagram is shown for a method of managing trajectory segments in a particle filter estimation framework;

[0075] Figure 3 A schematic diagram of the particle filter estimation framework is shown; and

[0076] Figure 4 Another schematic diagram is shown for a method of managing trajectory segments in a particle filter estimation framework.

[0077] The list of reference numerals used in the figures and their meanings are provided in summary form. In principle, the same reference numerals are used for the same parts in the figures.

[0078] Figure 1A schematic diagram illustrating the relationship between objects, trajectory segments, and particles is shown. For example, a traffic scene is observed using different sensors, resulting in a grid map M consisting of multiple grid cells C. At least one object O is identified in the grid map M. Particles P are grouped into trajectory segments T. Each trajectory segment T includes a trajectory segment state, which is updated using an individual particle filter by employing its own particles P. Multiple particles P are used to determine the object O by clustering the particles P.

[0079] Figure 2 A schematic diagram of a method for managing trajectory segments within a particle filter estimation framework is shown.

[0080] At a certain time point t, in a list of multiple previous trajectory segments h t,p In the middle, multiple remaining trajectory segments T p,t Available. In the next time step t+1, in the list h of multiple updated surviving trajectory segments. t+1,p Multiple surviving trajectory segments T were identified and updated in the middle. p,t+1 Residual trajectory segment T p,t+1 It is only from the list of remaining trajectory segments h t,p Resampled from the previous list of trajectory segments h, or selectively extracted from the previous list of trajectory segments h. t,p The particles P in the remaining trajectory segment are resampled.

[0081] In addition, several new trajectory segments T new New trajectory segment list h t+1,b It is based on the measured occupancy mass m c(Occ) Intensity D based on grid cell quality C and initial weight w init And it is certain.

[0082] Figure 3 A schematic diagram of a particle filter estimation framework 10 is shown. The particle filter estimation framework 10 includes multiple sensors 20 that provide sensor data for, for example, a traffic scene. In this case, a set of radar sensors 21, stereo vision sensors 22, and laser sensors 23 provide sensor data from the vehicle environment within the traffic scene. For example, the sensor data covers the front view, rear view, left view, and right view of the vehicle. The sensor data from the stereo vision sensors 22 and the laser sensors 23 is provided to a semantic stixel unit 30, which provides a more compact Stixel-based representation of the sensor data.

[0083] Preprocessed sensor data is provided to grid channel 40, which includes separate evidence grid channels, in this case, placeholder grid channel 41 and semantic grid channel 42. Grid channel 40, fed with sensor data, is based on tracking using a multilayer particle filter, which is performed by estimation unit 50. Estimation unit 50 provides estimated dynamic trajectory segments 60, which indicate the dynamic positions of objects around the vehicle.

[0084] Figure 4 Another schematic diagram is shown for a method of managing trajectory segments in a particle filter estimation framework. In the first step S10, depending on the previous trajectory segment T... p,t The list is used to perform trajectory segment prediction, thereby determining the remaining trajectory segments and the new trajectory segment T. new In the second step S20, the new trajectory segment T is initialized. new The new measurement results are sampled to determine the estimated number of new trajectory segments N. eti In the third step S30, depending on the previous trajectory segment T... p,t The list determines the number N of remaining trajectory segments. p,t In the fourth step S40, the number N of the estimated new trajectory segments depends on... eti The number of remaining trajectory segments N p,t and memory boundary N max Determine the number N of new trajectory segments to be sampled new And the number N of the updated surviving trajectory segments p,t+1 In step S50, the determined number N depends on the updated surviving trajectory segments. p,t+1 From the remaining trajectory segment T p,t The list of updated surviving trajectory segments T p,t+1 Sampling is performed. In step S60, the number N depends on the determined number of new trajectory segments. new From uncorrelated measurement results, the new trajectory segment T new Perform sampling.

Claims

1. A method for managing trajectory segments in a particle filter estimation framework for performing a tracking method that classifies objects in image data, comprising the following steps: Depends on the previous trajectory segment (T) p,t The list of ) is used to perform trajectory segment prediction (S10), thereby determining the remaining trajectory segments and new trajectory segments (T) new ), where the trajectory segment is an estimator based on an individual particle filter that indicates a group of particles; For initializing the new trajectory segment (T) new The new measurement results are sampled (S20) to determine the number of estimated new trajectory segments (N). eti ); Depends on the previous trajectory segment (T) p,t The list determines the number (N) of the remaining trajectory segments (S30). p,t ); Depends on the number of estimated new trajectory segments (N) eti ), the number of remaining trajectory segments (N) p,t ) and memory limits (N max (S40) to determine the number (N) of new trajectory segments to be sampled. new ) and the number of updated surviving trajectory segments (N) p,t+1 ); Depends on the determined number (N) of the updated surviving trajectory segments. p,t+1 ), from the remaining trajectory segment (T) p,t The list of updated surviving trajectory segments (T) p,t+1 Sampling is performed (S50); as well as Depends on the determined number (N) of the new trajectory segments new ), from uncorrelated measurement results, for the new trajectory segment (T) new Sampling is performed (S60).

2. The method according to claim 1, comprising the following steps: The validity of remaining trajectory segments depends on the verification metric used to identify invalid trajectory segments. as well as From the previous trajectory segment (T) p,t Remove the invalid trajectory segment from the list.

3. The method according to claim 2, wherein: Validation metrics include the age of the trajectory segment and / or the probability of its existence.

4. The method according to any one of the preceding claims, wherein, Sampling the updated surviving trajectory segment (S50) includes: sampling from the previous trajectory segment (T) p,t The new trajectory segment index is sampled from the list; and For the new trajectory segment (T) new Sampling (S60) includes: sampling the segments used to initialize the new trajectory segment (T). new The new measurement index is sampled.

5. The method according to any one of the preceding claims, wherein, Performing trajectory segment prediction includes: determining the remaining trajectory segments and new trajectory segments (T... new The trajectory segment weights.

6. The method according to any one of the preceding claims, wherein, Determine (S40) the number (N) of new trajectory segments. new )include: Determine the number of empty slots (N) es ), number of empty slots (N) es That is, from the memory limit (N) max Subtract the number of remaining trajectory segments (N) from the original data. p,i );as well as The maximum number of new trajectory segments (N) newmax ) and the number of empty slots (N) es Choose a higher number from the estimated number of new trajectory segments (N). eti Choose the lower quantity.

7. The method according to any one of the preceding claims, wherein, Determine (S30) the number (N) of the remaining trajectory segments to be updated. p,i+1 )include: The number of remaining trajectory segments (N) p,i ) and memory limits (N max ) and the number of new trajectory segments (N) new Choose the lower quantity from the difference between the two.

8. A control unit configured to perform the method according to any one of claims 1 to 7.

9. A method for classifying objects in image data, comprising the following steps: Receive image data; as well as Objects in image data are classified using a tracking method performed by a particle filter estimation framework, wherein the tracking method includes a method for managing trajectory segments according to any one of claims 1 to 7.

10. A control method for a robot that is at least partially autonomous, comprising the following steps: Receive sensor data from at least partially autonomous robots; Perform sensor identification on the received sensor data, including performing the method of any one of claims 1 to 7; as well as The robot is controlled at least partially autonomously, depending on the sensor recognition performed.

Images