Grid detection and tracking framework to build environment semantic occupancy and velocity field method and system

Abstract

The present application relates to a kind of grid detection tracking framework constructs environment semantic occupancy and velocity field method and system, comprising: by a variety of vehicle-mounted heterogeneous sensors, the complete semantic segmentation map of environment is obtained by the deep learning of neural network;Grid level tracking is carried out on the complete semantic segmentation map, the fine-grained semantic bird's eye view generated by the upstream module is given, the occupancy and speed of each grid are modeled, and the particle is distributed to the grid, and the particle is updated;Grid level post-fusion is carried out, and fine-grained semantic and speed information are fused, and the attribute of the semantic, occupancy and speed of each grid in the environment is comprehensively understood, and enhanced drivable space cognition is formed.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving, and in particular to a method and system for constructing the semantic occupancy and velocity field of an environment using computer vision and probabilistic reasoning theory to realize a grid-level detection and tracking framework. Background Technology

[0002] Occupied grid maps (OGMs) are an important form of environmental representation in robotics and autonomous driving. They describe the occupancy rate of various objects in the environment using raw sensor data, allowing vehicles to avoid collisions with obstacles. The main advantages of grid maps are their complete representation of perceived data and the scene, their ability to analyze perception range and uncertainty, and their common use as a powerful multi-sensor fusion framework. Unlike indoor robots, autonomous vehicles typically operate in highly dynamic traffic environments, requiring the ability to discern the speed of obstacles. This has led to the development of Dynamic Occupied Grid Maps (DOGMs). DOGMs primarily use Monte Carlo methods to model grid velocities with random particles and perform particle selection through Bayesian inference to ultimately obtain the grid velocity. However, for large datasets, this requires a very large number of particles, resulting in long runtimes and low velocity accuracy.

[0003] Scene flow is a technique for representing the changes of each point in an image or point cloud between two consecutive frames in 3D space. The main challenge lies in estimating motion in standard 3D space using image data, while point cloud methods face the difficulty of obtaining annotated real-world data. Previous studies have used simulated or artificially generated data. Scene flow computation is slow and cannot meet the requirements of real-time autonomous driving. Joint segmentation and motion prediction is a novel task that jointly estimates semantic and motion features at the raster level. It can be viewed as a rasterized scene flow technique based on bird's-eye view or a grid-based detection technique. Subsequent research, Pillar-Motion, introduces self-supervised correction and optimization using point cloud learning with image optical flow. Velocity estimation in this type of method is generally based on end-to-end prediction of preceding frames, rather than establishing a tracking relationship through the correlation between consecutive frames. This discontinuity between velocity and semantic temporal sequence affects the continuous understanding of the raster environment.

[0004] Semantic segmentation of point clouds refers to assigning a specific semantic label to each point in the point cloud. It can also be extended to instance segmentation and panoramic segmentation, and is an important method for achieving complete environmental understanding of LiDAR point clouds. In engineering, frame-by-frame point cloud annotation is very costly. The algorithmic challenge lies in the unordered nature of point cloud data, the sparseness of outdoor scenes, and the irregular density that is distance-dependent. Bird's-eye view semantic segmentation of images is used to segment moving objects and static elements from a bird's-eye view perspective. Compared to traditional image view segmentation tasks, the annotations from object detection can be repeatedly used to train dynamic object segmentation, while high-precision maps can be directly used to train static element segmentation. Correspondingly, the added difficulty is the need for a view fusion module to fuse information from multiple cameras into the bird's-eye view space. Both sensor and related technologies can achieve good single-frame results in semantic bird's-eye view images, but they lack velocity estimation. This lack of necessary velocity field information for environmental cognition and risk assessment means they cannot be directly used in drivable area recognition modules. Summary of the Invention

[0005] To address the aforementioned problems, the purpose of this invention is to provide a method and system for constructing environmental semantic occupancy and velocity field in a grid detection and tracking framework, which can improve the continuity and accuracy of individual grid attributes.

[0006] To achieve the above objectives, in a first aspect, the technical solution adopted by the present invention is as follows: a grid detection and tracking framework for constructing environmental semantic occupancy and velocity field, comprising: acquiring a complete semantic segmentation map of the environment through deep learning of a neural network using multiple vehicle-mounted heterogeneous sensors; performing grid-level tracking on the complete semantic segmentation map, given a fine-grained semantic bird's-eye view generated by an upstream module, modeling the occupancy and velocity of each grid, and assigning particles to the grids for particle updates; performing grid-level post-fusion to fuse the fine-grained semantic and velocity information, and comprehensively understanding the semantic, occupancy, and velocity attributes of each grid in the environment to form an enhanced drivable space cognition.

[0007] Furthermore, various vehicle-mounted heterogeneous sensors include: a velocity prior formed by a grid of point clouds from millimeter-wave radar through a sensor inverse mapper, supplementing the velocity auxiliary information of the grid tracking filter during observation; dense point cloud information output by a high-line-count lidar for a complete understanding of the scene without blind spots; and the installation of multiple vehicle-mounted cameras for surround view and depth estimation, followed by segmentation of objects and the ground on the bird's-eye view through perspective transformation to form a semantic grid for a complete understanding of the 360° scene.

[0008] Furthermore, raster-level tracking is performed on the fully semantic segmentation map, including:

[0009] Rotate the original information to align with the geodetic coordinate system; grids and particles do not require rotation.

[0010] Set the observation grid, and set the Boolean value of the region of interest (ROI) of the grid area containing common obstacles in target detection to true;

[0011] In the observation grid, select the grid within the region of interest. For observation elements with a Boolean value of true, evenly distribute the newly generated particles to the grids observed each time. If the corresponding grid of a particle has left the region of interest after prediction and update, make those particles disappear, change the particle weight to zero, sort the weights of the remaining particles, and finally ensure that the number of particles that disappear and the number of newly generated particles are equal to the set hyperparameter.

[0012] A dynamic grid map is constructed to predict and update particles and grids, achieving grid-level tracking.

[0013] Furthermore, a dynamic raster map is constructed, including:

[0014] Particle prediction involves using a suitable kinematic model to predict the position and velocity of the current particle in the next frame, including calculating a new grid cell index for each particle after prediction.

[0015] Particles are assigned to grids, and all particles are sorted according to the grid cell index. Each particle is checked to see if it is the first or last particle in a group with the same index. If so, its index is written into the corresponding grid cell, and the grid occupancy status is predicted and updated.

[0016] For persistent particles, the likelihood function of the particle relative to the observation is calculated, and the particle weights are adjusted according to the likelihood function. Finally, the weights are normalized without changing the particle's state variables. For a single particle, the new weight is equal to the old weight multiplied by the likelihood function and then normalized.

[0017] The output grid velocity information is based on the macroscopic statistical state variables of the particles, including the weighted sum of the particle velocities.

[0018] Furthermore, the grid occupancy status is predicted and updated, including:

[0019] The occupation status of the assigned particles is predicted and calculated, which is divided into occupied mass and idle mass, in order to calculate the occupation probability.

[0020] The grid is updated by combining the currently predicted grid occupancy quality with the quality observed by the sensor inverse mapper in the next frame, based on DS evidence theory.

[0021] Furthermore, post-fusion at the raster level includes:

[0022] Semantic segmentation under top view implemented by different sensors is set as a grid-level detection method, and different detection methods will be connected to the grid-level tracker respectively;

[0023] In the output layer of the tracker, a covariance cross-fusion method is used to calculate the occupancy and velocity of each grid cell. The optimization objective of the covariance cross-fusion method is to find an optimal weight to minimize the performance index. The selection of the performance index is represented by the trace of the covariance, which is the covariance after fusion.

[0024] Furthermore, it fosters enhanced awareness of drivable spaces, including:

[0025] The system densely samples the farthest reachable point in all 360° directions and connects the farthest reachable points to form a panoramic view of the drivable space. Simultaneously, it samples in the current frame's occupied grid and in the predicted occupied grid for the next second based on the tracking speed.

[0026] The final drivable area is obtained by taking the intersection of the current and future drivable areas;

[0027] After generating the drivable area, a virtual repulsive field is established for each grid, and a virtual kinetic energy field is established for the semantics represented by the grid to analyze the probability of potential collisions.

[0028] Secondly, the technical solution adopted by this invention is as follows: a grid detection and tracking framework for constructing an environmental semantic occupancy and velocity field system, comprising: a first processing module, which acquires a complete semantic segmentation map of the environment through deep learning of a neural network using multiple vehicle-mounted heterogeneous sensors; a second processing module, which performs grid-level tracking on the complete semantic segmentation map, and, given a fine-grained semantic bird's-eye view generated by an upstream module, models the occupancy and velocity of each grid, assigns particles to the grid, and updates the particles; and a third processing module, which performs grid-level post-fusion, integrates fine-grained semantic and velocity information, and comprehensively understands the semantic, occupancy, and velocity attributes of each grid in the environment to form an enhanced drivable space cognition.

[0029] Thirdly, the technical solution adopted by the present invention is: a computer-readable storage medium for storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0030] Fourthly, the technical solution adopted by the present invention is: a computing device comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods described above.

[0031] The present invention has the following advantages due to the adoption of the above technical solutions:

[0032] 1. The grid-based detection and tracking framework of this invention can replace object-level detection and tracking methods (DATMO).

[0033] 2. This invention can utilize the fusion of multiple sensors to jointly estimate the semantics, occupancy rate and velocity of the grid, while analyzing uncertainties, thereby improving the continuity and accuracy of each individual attribute of the grid.

[0034] 3. This invention can focus on the key regions of interest in autonomous driving, guide the particle generation model, reduce inference computation and latency, and improve the overall pipeline operation speed to real-time operation. Attached Figure Description

[0035] Figure 1 This is a flowchart of the semantic dynamic grid map in an embodiment of the present invention;

[0036] Figure 2 This is a flowchart illustrating the implementation of grid-level tracking in an embodiment of the present invention;

[0037] Figure 3 This is a schematic diagram of covariance cross-tracking level fusion in an embodiment of the present invention;

[0038] Figure 4 This is a schematic diagram of semantic dynamic grid map used for driving environment cognition in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0040] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0041] To address the problem of discontinuous velocity and semantic temporal sequence in existing technologies, which affects the continuous understanding of grid environments, this invention provides a grid detection and tracking framework for constructing environmental semantic occupancy and velocity fields, thereby achieving the estimation of environmental semantic occupancy and velocity fields. It combines emerging neural network-based joint segmentation and prediction methods with reasoning-based grid-level tracking methods to realize a grid detection and tracking pipeline for forming temporally continuous, grid-based environmental semantics, occupancy, and velocity estimation. It also provides a post-fusion method for multi-sensor semantic maps, an evaluation method for semantic dynamic maps, and an environmental risk cognition and collision detection method based on this environmental expression.

[0042] In one embodiment of the present invention, a method for constructing environmental semantic occupancy and velocity field using a grid detection and tracking framework is provided. In this embodiment, as... Figure 1 As shown, the method includes the following steps:

[0043] 1) A complete semantic segmentation map of the environment is obtained through deep learning of neural networks using a variety of heterogeneous vehicle sensors;

[0044] In this embodiment, grid-level detection is performed. Corresponding algorithms are designed for various heterogeneous vehicle-mounted sensors, and a complete semantic segmentation map of the environment is obtained through deep learning of neural networks. Different types of sensors, including LiDAR, cameras, and millimeter-wave radar, can be used in combination or individually. The purpose of complete semantic segmentation of the environment is to distinguish dynamic objects from the static environment at the smallest separable unit level of perception and to provide prior knowledge of the region of interest for grid-based tracking.

[0045] 2) Perform grid-level tracking on the fully semantic segmentation map. Given the fine-grained semantic bird's-eye view generated by the upstream module, model the occupancy and velocity of each grid, assign particles to the grid, and update the particles.

[0046] In this embodiment, a probabilistic reasoning-based tracking method is employed to probabilistically model the occupancy and velocity of each grid cell. The probabilistic reasoning is implemented using a particle filtering-based filter. Particles are assigned to the grid cells using a top-down particle allocation method proposed by Nuss et al., and the particles are then updated.

[0047] 3) Perform grid-level fusion to integrate fine-grained semantic and velocity information. Through a comprehensive understanding of the semantics, occupancy, and velocity attributes of each grid in the environment, an enhanced perception of drivable space is formed.

[0048] In this embodiment, the main focus is on the post-fusion of grid-level detections from different sensors, which achieves the fusion of fine-grained semantic and velocity information. By comprehensively understanding the semantic, occupancy, and velocity attributes of each grid in the environment, an enhanced awareness of drivable space is formed.

[0049] In step 1) above, the various vehicle-mounted heterogeneous sensors include:

[0050] 1.1) The point cloud of the millimeter-wave radar forms a velocity prior of the grid through the sensor inverse mapper, which supplements the velocity auxiliary information of the grid tracking filter in the observation;

[0051] Specifically, millimeter-wave radar data is a sparse two-dimensional point cloud containing Doppler measurements, which cannot express the overall semantic state of the environment on its own. Therefore, the millimeter-wave radar point cloud forms a velocity prior of a grid through a sensor inverse mapper, which supplements the velocity auxiliary information in the observation of the grid tracking filter.

[0052] 1.2) Dense point cloud information is output by a high-line-count lidar to achieve a complete understanding of the scene without blind spots;

[0053] Specifically, high-line-count LiDAR outputs dense point cloud information, enabling a complete understanding of the scene without blind spots. The original point cloud includes information such as 3D coordinates and reflectivity. There are two methods for processing dense point clouds. One is the semantic segmentation method of point clouds, such as using the advanced semantic segmentation network Cylinder3D++ to classify the point cloud into different semantic categories, and then using a sensor inverse mapper to form a semantic raster. The second method is to use the MotionNet method to pixelate the point cloud and then directly perform semantic segmentation to form a semantic raster.

[0054] 1.3) Install vehicle-mounted multi-cameras for surround view and perform depth estimation. By switching perspectives, segment objects and the ground on the bird's-eye view to form a semantic grid for a complete understanding of the 360° scene.

[0055] Specifically, vehicle-mounted multi-camera systems can achieve a complete understanding of a 360° scene through surround-view installation. Since the images lack depth information, depth estimation is required first. Then, a semantic grid is formed on the bird's-eye view by segmenting objects and the ground using viewpoint transformation technology. Viewpoint transformation technology can be a network method or a pseudo-LiDAR method.

[0056] In step 2) above, if Figure 2 As shown, raster-level tracking on a fully semantic segmentation map includes the following steps:

[0057] 2.1) Coordinate transformation process: Rotate the original information to align with the geodetic coordinate system; no rotation operation is required for the grid and particles.

[0058] Specifically, the center point of the raster map is fixed on the vehicle coordinate system. When the vehicle translates, the raster and particles will compensate for the vehicle's motion in the opposite direction. Practice shows that when the vehicle rotates during operation, the raster and particles will rotate in the opposite direction to compensate for the motion, which will bring a large error. Based on the requirement of constructing the driving space before decision-making, we choose to rotate the original information to align with the geodetic coordinate system, so that the raster and particles do not need to be rotated.

[0059] 2.2) Set the observation grid, and set the Boolean value of the region of interest for the grid area of ​​common obstacles in target detection to true;

[0060] Specifically, the map size and resolution are first set, and then corresponding video memory space is allocated. Observation grids occupy video memory space contiguously in an array, with each observation grid being an element. State variables include 2D coordinate position, idle mass, occupied mass, likelihood probability, and Boolean value for the region of interest. For the raw point cloud input, the occupied and idle masses are calculated using a sensor inverse mapper, followed by the likelihood probability. For other semantic grid inputs, occupancy analysis is performed on different categories based on the confidence probability of semantic segmentation. The ground category is determined as an unoccupied area, while other obstacles are defined as having varying degrees of occupancy. In particular, the Boolean value for the region of interest of grid regions whose semantic category is a common obstacle in object detection is set to true to facilitate subsequent particle allocation.

[0061] 2.3) Particle generation and distribution: Select the grid within the region of interest in the observation grid. For observation elements with a Boolean value of true, distribute the newly generated particles evenly to the grids observed each time. If the corresponding grid of a particle has left the region of interest after prediction and update, make these particles disappear, change the particle weight to zero, sort the weights of the remaining particles, and finally ensure that the number of particles that disappear and the number of newly generated particles are equal, which is a set hyperparameter.

[0062] In the top-down framework of this invention, the occupancy rate of the final state grid is calculated by Dempster / Shafer evidence theory. Similar to the implementation of the binary Bayesian filtering framework, the significance of particle filtering is to solve the velocity of the grid within the region of interest. Therefore, grids within the region of interest are selected from the observed grids, i.e., observation elements with a Boolean value of true, and newly generated particles are evenly distributed to the grids observed each time. In the resampling step, if the grid corresponding to a particle has left the region of interest after prediction and update, these particles disappear, and their weights become zero. Then, the weights of the remaining particles are sorted. The number of particles that disappear and the number of newly generated particles are set to be equal as hyperparameters to keep the total number of particles and the occupied memory and GPU resources stable during inference. Due to the strong continuity in autonomous driving scenarios, experiments have shown that a disappearance and regeneration of 10% of the total number of particles can solve the velocity estimation of most newly entered objects within the observation range.

[0063] 2.4) Construct a dynamic grid map to predict and update particles and grids, and achieve grid-level tracking.

[0064] The process of constructing a dynamic raster map includes the following steps:

[0065] 2.4.1) Predict the position and velocity of the current particle in the next frame by using the existing particles according to a suitable kinematic model, including calculating the new grid cell index for each particle after prediction.

[0066] Specifically, an appropriate number of random particles should be pre-sampled in a separate step, the kinematic model should follow a first-order linear motion model, and an appropriate detection probability should be considered, i.e., for the particle group in the (k+1)th frame. In summary, the weights are updated as follows:

[0067]

[0068] In the formula, p S It is the probability that a certain particle will still exist in the next frame of observation, and is generally set as a prior constant; This represents the updated weight of the i-th particle; This represents the weight of the i-th particle before the update; The state vector of a particle is generally expressed by a four-dimensional vector consisting of its planar position and planar velocity; v represents the total number of particles; and i represents the cycle symbol.

[0069] 2.4.2) Assign particles to grids, sort all particles according to grid cell index, and check whether each particle is the first or last particle in a group with the same index; if so, write its index into the corresponding grid cell, and predict and update the grid occupancy status.

[0070] The prediction and updating of grid occupancy status includes the following steps:

[0071] 2.4.2.1) Based on the allocated particle pairs, the occupancy status is predicted and calculated. The occupancy status is divided into occupied mass and idle mass, so as to calculate the occupancy probability.

[0072] 2.4.2.2) The grid is updated by combining the currently predicted grid occupancy quality with the quality observed by the sensor inverse mapper according to DS evidence theory.

[0073] Specifically, all particles are sorted according to their weights and the index of the grid they belong to. Each grid cell can store two particle indices. These represent the first and last indices of the group of particles predicted to be in the grid cell. For assignment, each particle checks whether it is the first or last particle in a group with the same index. If so, it writes its index into the corresponding grid cell. Since each grid cell can have at most one first or last particle, grid occupancy state prediction and updating is a crucial step. First, the occupancy status is calculated based on the particles assigned in the previous steps. The occupancy status is divided into occupied quality and idle quality, and the occupancy probability is calculated. The predicted occupancy quality is calculated as follows:

[0074]

[0075] In the formula, Indicates the quality of the raster c; O k+1 This indicates the occupied state of frame k+1; This represents the total number of persistent particles in grid c; This represents the weight of particle i in grid c.

[0076] The predicted idle quality is:

[0077]

[0078] In the formula, Indicates the quality of the raster c; F k+1 This represents the idle state of frame k+1; α represents the attenuation factor, which is a function of time T; T represents the time frame. This indicates the quality of the k-th frame.

[0079] The relationship between the occupancy probability of a grid cell and its occupancy quality and its free quality is as follows:

[0080]

[0081] In the formula, This represents the occupancy probability of grid c.

[0082] Raster updates require combining the currently predicted raster occupancy quality with the quality of new observations based on DS evidence theory:

[0083]

[0084] In the formula, This indicates the occupancy quality of the observation in frame k+1 of raster c.

[0085] The filter's state update divides the quality occupied by the next frame into two parts: occupied by a persistent preceding frame object and occupied by a new object, represented as:

[0086]

[0087] In the formula, This represents the sum of the weights of particles that persist in grid c throughout frame k+1. This represents the sum of weights of newly generated particles in grid c at frame k+1.

[0088] The newborn mass and the sustained occupancy mass should be related to the newborn detection probability:

[0089]

[0090] In the formula, p B This represents the probability of a new object being detected, and is generally set to a constant.

[0091] Calculate the new mass and the occupied mass of the persistent object based on this relationship:

[0092]

[0093]

[0094] 2.4.3) Update the persistent particles, calculate the likelihood function of the particles relative to the observations, and based on the likelihood function... Adjust the weights of the particles, and finally normalize the weights without changing the state variables of the particles; for a single particle, the new weight is equal to the old weight multiplied by the likelihood function and then normalized.

[0095] Specifically, the weights are normalized without changing the particle's state variables:

[0096]

[0097] For a single particle, the new weight is equal to the old weight multiplied by the likelihood function and then normalized:

[0098]

[0099] In the formula, This represents the state variable of particle i in grid c of frame k+1. This represents the state variable of particle i in the current frame grid c. This represents the weight of particle i in grid c. The likelihood function representing the observed state, z k+1 This represents the observations in the (k+1)th frame.

[0100] 2.4.4) Output the velocity information of the grid based on the macroscopic statistical state variables of the particles, including the weighted sum of the particle velocities;

[0101] Specifically, assuming that grid c contains a certain number of particles, the velocity of the grid is the weighted sum of the velocities of the particles. Taking the grid velocity in the x-direction as an example:

[0102]

[0103]

[0104] In the formula, p represents a particle that exists uniformly. This indicates the velocity of grid c in the x-direction; This represents the sum of the weights of all particles in grid c; This represents the velocity of particle i in grid c; This represents the weight of particle i in grid c.

[0105] Variance of a directional velocity estimate The calculation is as follows:

[0106]

[0107] Covariance of velocity estimates in two directions The calculation is as follows:

[0108]

[0109] In the formula, Cross-correlation coefficients representing velocities in the x and y directions; This represents the velocity of particle i, which exists consistently in grid c, in the x-direction at time k+1.

[0110] In step 3) above, if Figure 3 As shown, raster-level post-fusion includes the following steps:

[0111] 3.1.1) Semantic segmentation under top view implemented by different sensors is set as a grid-level detection method, and different detection methods will be connected to the grid-level tracker respectively;

[0112] The tracing component can be loaded by multiple CPU threads and computed in parallel by the GPU, occupying multiple blocks of video memory without affecting the overall speed.

[0113] 3.1.2) In the output layer of the tracker, the covariance cross fusion method is used to calculate the occupancy and velocity of each grid cell; the optimization goal of the covariance cross method is to find an optimal weight to minimize the performance index. The covariance trace is used to characterize the selection of the performance index, which is the covariance after fusion.

[0114] Specifically, each observed grid has its own velocity mean and covariance. The optimization objective of the covariance cross method is to find an optimal weight ω1 to minimize the performance index. The following formula uses the trace of the covariance to represent the selection of the performance index; that is, the fused covariance should be:

[0115]

[0116] In the formula, tr represents the trace operator of the matrix (the sum of the diagonals); P CI Let P1 represent the updated covariance matrix; ω1 represents the prior weight of the first observation; ω2 represents the prior weight of the first observation; P1 represents the initial value of the covariance of the first observation; P2 represents the initial value of the covariance of the second observation.

[0117] The final result of multi-sensor analysis is the mean and covariance of the fused velocity values ​​from each sensor, as well as the corresponding occupancy probability.

[0118] In step 3) above, if Figure 4 As shown, enhanced drivable space cognition is formed. Vector-based drivable space can be realized through semantic dynamic grid maps, including the following steps:

[0119] 3.2.1) Densely sample the farthest reachable point in all 360° directions, with the angular resolution as small as possible, and connect the farthest reachable points to form a panoramic perception of the drivable space; at the same time, sample in the occupied grid map of the current frame, and also sample in the predicted occupied grid map of the next second based on the tracking speed prediction.

[0120] 3.2.2) The final drivable area is obtained by taking the intersection of the current and future drivable areas;

[0121] 3.2.3) After generating the drivable area, the safety field theory is applied to the information of the grid medium to establish a virtual repulsive field for each grid. A virtual kinetic energy field is also established for the semantics represented by the grid to analyze the probability of potential collisions. Collision detection is based on considering the kinetic energy transformation caused by collisions with obstacles to deduce the probability of collisions between the vehicle and different objects occupying the grid and the corresponding trajectory planning strategies.

[0122] Step 3) above also includes an accuracy evaluation method for the semantic, occupancy, and velocity attributes of the dynamic semantic raster. The accuracy evaluation of the dynamic semantic raster adopts the evaluation method designed by MotionNet. For motion prediction, performance is evaluated by dividing each raster cell into 3 groups with different velocities: static, slow (≤5m / s), and fast (>5m / s). In each group, this embodiment calculates the average L2 distance between the estimated displacement and the actual displacement. For the accuracy of the classification task, performance is measured by two metrics: (1) overall cell classification accuracy (OA), i.e., the average accuracy of all cells; and (2) average class accuracy (MCA), i.e., the average accuracy of all classes.

[0123] In one embodiment of the present invention, a grid detection and tracking framework is provided to construct an environmental semantic occupancy and velocity field system, comprising:

[0124] The first processing module uses a variety of heterogeneous vehicle sensors to obtain a complete semantic segmentation map of the environment through deep learning of neural networks.

[0125] The second processing module performs grid-level tracking on the fully semantic segmentation map. Given the fine-grained semantic bird's-eye view generated by the upstream module, it models the occupancy and velocity of each grid, assigns particles to the grid, and updates the particles.

[0126] The third processing module performs grid-level post-fusion, integrating fine-grained semantic and velocity information. By comprehensively understanding the semantics, occupancy, and velocity attributes of each grid in the environment, it forms an enhanced perception of drivable space.

[0127] The first processing module mentioned above includes various heterogeneous vehicle sensors, including:

[0128] The point cloud of millimeter-wave radar forms a velocity prior of a grid through a sensor inverse mapper, which supplements the velocity auxiliary information of the grid tracking filter in the observation;

[0129] Dense point cloud information is output by high-line-count LiDAR to achieve a complete understanding of the scene without blind spots;

[0130] By installing multiple onboard cameras for surround view and performing depth estimation, and by transforming the viewpoint to segment objects and the ground on the bird's-eye view to form a semantic grid, a complete understanding of the 360° scene can be achieved.

[0131] The second processing module described above performs raster-level tracking on the fully semantic segmentation map, including:

[0132] Rotate the original information to align with the geodetic coordinate system; grids and particles do not require rotation.

[0133] Set the observation grid, and set the Boolean value of the region of interest (ROI) of the grid area containing common obstacles in target detection to true;

[0134] In the observation grid, select the grid within the region of interest. For observation elements with a Boolean value of true, evenly distribute the newly generated particles to the grids observed each time. If the corresponding grid of a particle has left the region of interest after prediction and update, make those particles disappear, change the particle weight to zero, sort the weights of the remaining particles, and finally ensure that the number of particles that disappear and the number of newly generated particles are equal to the set hyperparameter.

[0135] A dynamic grid map is constructed to predict and update particles and grids, achieving grid-level tracking.

[0136] The construction of a dynamic raster map includes:

[0137] Particle prediction involves using a suitable kinematic model to predict the position and velocity of the current particle in the next frame, including calculating a new grid cell index for each particle after prediction.

[0138] Particles are assigned to grids, and all particles are sorted according to the grid cell index. Each particle is checked to see if it is the first or last particle in a group with the same index. If so, its index is written into the corresponding grid cell, and the grid occupancy status is predicted and updated.

[0139] For persistent particles, the likelihood function of the particle relative to the observation is calculated, and the particle weights are adjusted according to the likelihood function. Finally, the weights are normalized without changing the particle's state variables. For a single particle, the new weight is equal to the old weight multiplied by the likelihood function and then normalized.

[0140] The output grid velocity information is based on the macroscopic statistical state variables of the particles, including the weighted sum of the particle velocities.

[0141] Preferably, predicting and updating the grid occupancy state includes:

[0142] The occupation status of the assigned particles is predicted and calculated, which is divided into occupied mass and idle mass, in order to calculate the occupation probability.

[0143] The grid is updated by combining the currently predicted grid occupancy quality with the quality observed by the sensor inverse mapper according to DS evidence theory.

[0144] In the third processing module mentioned above, the raster-level post-fusion includes:

[0145] Semantic segmentation under top view implemented by different sensors is set as a grid-level detection method, and different detection methods will be connected to the grid-level tracker respectively;

[0146] In the output layer of the tracker, a covariance cross-fusion method is used to calculate the occupancy and velocity of each grid cell. The optimization objective of the covariance cross-fusion method is to find an optimal weight to minimize the performance index. The selection of the performance index is represented by the trace of the covariance, which is the covariance after fusion.

[0147] The third processing module mentioned above forms an enhanced perception of the drivable space, including:

[0148] The system densely samples the farthest reachable point in all 360° directions and connects the farthest reachable points to form a panoramic view of the drivable space. Simultaneously, it samples in the current frame's occupied grid and in the predicted occupied grid for the next second based on the tracking speed.

[0149] The final drivable area is obtained by taking the intersection of the current and future drivable areas;

[0150] After generating the drivable area, a virtual repulsive field is established for each grid, and a virtual kinetic energy field is established for the semantics represented by the grid to analyze the probability of potential collisions.

[0151] The system provided in this embodiment is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0152] In one embodiment of the present invention, a computing device structure is provided. This computing device can be a terminal, which may include: a processor, a communication interface, memory, a display screen, and an input device. The processor, communication interface, and memory communicate with each other via a communication bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. When executed by the processor, the computer program implements a grid detection and tracking framework construction environmental semantic occupancy and velocity field method. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals. Wireless communication can be achieved through Wi-Fi, a management network, NFC (Near Field Communication), or other technologies. The display screen can be a liquid crystal display or an e-ink display. The input device can be a touch layer covering the display screen, or buttons, a trackball, or a touchpad mounted on the computing device casing, or an external keyboard, touchpad, or mouse. The processor can call logical instructions stored in the memory.

[0153] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0154] In one embodiment of the present invention, a computer program product is provided, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to perform the methods provided in the above-described method embodiments.

[0155] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions that cause a computer to perform the methods provided in the above embodiments.

[0156] The computer-readable storage medium provided in the above embodiments has a similar implementation principle and technical effect to the above method embodiments, and will not be described again here.

[0157] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0158] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0159] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A grid detection tracking framework constructs environment semantic occupancy and velocity field method, characterized in that, Comprise: A complete semantic segmentation map of the environment is obtained by deep learning of a neural network through a plurality of vehicle-mounted heterogeneous sensors; Grid-level tracking is performed on the complete semantic segmentation map, a fine-grained semantic bird's-eye view generated by an upstream module is given, occupancy and velocity of each grid are modeled, and particles are assigned to the grid for particle updating; Grid-level post-fusion is performed to fuse fine-grained semantic and velocity information, and an enhanced drivable space cognition is formed through comprehensive understanding of the semantic, occupancy and velocity attributes of each grid in the environment; Grid-level tracking is performed on the complete semantic segmentation map, comprising: Rotating the original information to align with the geodetic coordinate system, and the grid and the particle do not need to be rotated; Setting an observation grid, and setting a Boolean value of a grid area of a common obstacle in target detection to true as a region of interest; Selecting a grid in the observation grid within the region of interest, and the observation element with the Boolean value of true is evenly assigned to the grid of each observation to newly generate particles; if the corresponding grid of the particle after prediction and updating has left the region of interest, the particle is made to disappear, the particle weight becomes zero, the weights of the remaining particles are sorted, and the number of finally disappeared particles and the number of newly generated particles are equal to the set hyperparameter; A dynamic grid map is constructed to predict and update the particles and the grid to realize grid-level tracking; The dynamic grid map is constructed, comprising: Predicting the particles, predicting the position and velocity of the current particle in the next frame according to a suitable kinematic model, and calculating the new grid cell index of each particle after prediction; Assigning the particles to the grid, sorting all particles according to the grid cell index, and checking whether each particle is the first or last particle of the group with the same index; if so, the index is written into the corresponding grid cell to predict and update the grid occupancy state; Updating the particles that exist continuously, calculating the likelihood function of the particles relative to the observation, adjusting the weight of the particle according to the likelihood function, and finally normalizing the weight without changing the state variable of the particle; for a single particle, the new weight is equal to the old weight multiplied by the likelihood function and then normalized; Outputting the velocity information of the grid according to the macroscopic statistical state variable of the particle, including the weighted sum of the velocity of the particle; The grid-level post-fusion comprises: Setting the semantic segmentation under the bird's-eye view realized by different sensors as a grid-level detection method, and different detection methods will access the grid-level tracker respectively; A covariance cross fusion method is used in the output layer of the tracker to calculate the occupancy rate and the velocity of each grid; the optimization target of the covariance cross method is to solve an optimal weight to minimize the performance index, and the trace of the covariance is selected to represent the performance index.

2. The grid detection and tracking framework architecture for building environment semantic occupancy and velocity field method of claim 1, wherein, The plurality of vehicle-mounted heterogeneous sensors comprise: The point cloud of the millimeter wave radar is mapped to the grid by a sensor inverse mapper to form a velocity prior of the grid, and the velocity auxiliary information of the grid tracking filter in observation is supplemented; The high-line-number laser radar outputs dense point cloud information to completely understand the scene without blind area; The installation of the vehicle-mounted multi-camera for the surround view, the depth estimation, the segmentation of the object and the ground on the bird's eye view through the perspective conversion, and the formation of the semantic grid to fully understand the 360° scene.

3. The grid detection and tracking framework architecture for building environment semantic occupancy and velocity field method of claim 1, wherein, The prediction and update of the grid occupancy state, including: The prediction calculation of the occupancy state according to the assigned particles, the occupancy state being divided into the occupancy quality and the free quality to calculate the occupancy probability; The combination of the current predicted grid occupancy quality and the quality obtained by the sensor inverse mapper observation of the next frame according to the D-S evidence theory to update the grid.

4. The grid detection and tracking framework architecture for building environment semantic occupancy and velocity field method of claim 1, wherein, The formation of the enhanced drivable space cognition, including: The dense sampling of the farthest reachable point in the 360° direction to connect the farthest reachable points to form the panoramic drivable space cognition; meanwhile, the sampling in the current frame of the occupancy grid map and the sampling in the predicted occupancy grid map of the next second according to the tracking speed; The intersection of the present and future drivable areas to obtain the final drivable area; After the generation of the drivable area, the establishment of the virtual repulsive field of each grid, the establishment of the virtual kinetic field for the semantics represented by the grid, and the analysis of the potential collision possibility.

5. A grid detection and tracking framework system for building environment semantic occupancy and velocity field, for implementing the grid detection and tracking framework method for building environment semantic occupancy and velocity field according to any one of claims 1 to 4, characterized in that, Including: A first processing module that obtains a complete semantic segmentation map of an environment through a plurality of vehicle-mounted heterogeneous sensors and a deep learning of a neural network; A second processing module that performs grid-level tracking on the complete semantic segmentation map, models the occupancy and speed of each grid, and updates particles in the grid; A third processing module that performs grid-level post-fusion, fuses the fine-grained semantics and speed information, comprehensively understands the attributes of the semantics, occupancy and speed of each grid in the environment, and forms an enhanced drivable space cognition.

6. A computer-readable storage medium storing one or more programs, the one or more programs comprising instructions that when executed by a computer cause the computer to perform a method comprising: The one or more programs include instructions that, when executed by a computing device, cause the computing device to perform any of the methods of claims 1-4.

7. A computing device, comprising: Including: One or more processors, memories, and one or more programs, wherein the one or more programs are stored in the memories and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods of claims 1-4.

Images