Mapping a vehicle environment

Abstract

The present disclosure relates to mapping a vehicle environment. A computer-implemented method and apparatus of mapping a vehicle environment of a vehicle is disclosed. The method comprises determining an occupancy grid representing the vehicle environment, the occupancy grid comprising occupancy probability information of a first set of object detection results, wherein the occupancy probability information is determined from first localization information obtained from a first sensor system, the first localization information indicating one or more locations of the first set of object detection results relative to the vehicle; obtaining semantic information and second localization information associated with the semantic information from one or more semantic information sources, the semantic information comprising object classification information of a second set of object detection results, the second localization system indicating one or more locations of the second set of object detection results relative to the vehicle; and combining the object classification information of the second set of object detection results with the occupancy probability information of the occupancy grid to generate a classified occupancy grid.

Description

Technical Field

[0001] This disclosure relates to the vehicle environment of a mapped vehicle. Background Technology

[0002] With the development of fully autonomous and / or ADAS (Advanced Driver Assistance Systems) vehicles, numerous techniques have been developed to reliably estimate the vehicle environment based on large amounts of data from one or more sensors. One widely used approach involves using one or more sensors to detect objects or obstacles in the vehicle environment, then converting the raw data from the sensors into an occupancy grid containing cells associated with corresponding occupancy probabilities. The occupancy probability of each cell is calculated using a suitable sensor model (e.g., an inverse sensor model), which converts data related to the detected objects and additional information into occupancy probabilities.

[0003] Semantic information related to object classification in the vehicle environment can also be obtained. For example, a camera can acquire an image of the area in front of the vehicle. This image can then undergo segmentation analysis to determine the category of one or more objects in the field of view of the camera system. Information received from a mapping application can also be combined to provide further semantic information. However, the localization of such classified objects relative to the vehicle can be highly uncertain. Summary of the Invention

[0004] A first aspect of this disclosure provides a computer-implemented method for mapping a vehicle environment, the method comprising the steps of: determining an occupancy grid representing the vehicle environment, the occupancy grid including occupancy probability information of a first set of object detection results, wherein the occupancy probability information is determined based on first positioning information obtained from a first sensor system, the first positioning information indicating one or more positions of the first set of object detection results relative to the vehicle; obtaining semantic information and second positioning information associated with the semantic information from one or more semantic information sources, the semantic information including object classification information of a second set of object detection results, the second positioning system indicating one or more positions of the second set of object detection results relative to the vehicle; and combining the object classification information of the second set of object detection results with the occupancy probability information of the occupancy grid to generate a categorized occupancy grid.

[0005] The method may further include the step of: using the semantic information and the second location information to determine a semantic grid representing the vehicle environment, the semantic grid comprising a grid of evidence values ​​for one or more object classification types.

[0006] The steps of determining the semantic grid may include: projecting the object detection results in the second set of object detection results onto a grid representing the vehicle environment; determining the object space region of the object detection results based on the uncertainty value of the second localization information of the object detection results in the second set of object detection results; and assigning cells of one or more classification grids to be occupied by the object detection results in the second set of object detection results based on the object space region, each classification being related to a different classification type.

[0007] The method may further include the following steps: calculating the confidence value of a cell by applying a pignistic transformation to the evidence values ​​of one or more cells of the semantic grid; and combining the confidence values ​​of one or more cells of the semantic grid with the occupancy probability values ​​of one or more corresponding cells of the occupancy grid.

[0008] The step of combining the confidence values ​​of one or more cells of the semantic grid with the occupancy probability values ​​of one or more corresponding cells of the occupied grid may include: multiplying the confidence values ​​of the one or more cells of the semantic grid with the occupancy probability values ​​of one or more corresponding cells of the occupied grid.

[0009] The step of combining the occupancy probability information with the semantic evidence information may include the following steps: selecting object detection results with object classification from the second group of object detection results; using the second positioning information to identify the temporary position of the object detection results in the second group of object detection results; identifying the occupied area of ​​the occupied grid corresponding to the object detection results in the first group of object detection results as close to the temporary position of the object detection results in the second group of object detection results; and assigning the object classification of the object detection results in the second group of object detection results to the occupied area of ​​the occupied grid corresponding to the object detection results in the first group of object detection results.

[0010] The step of identifying the occupied region of an occupied grid may include the following steps: comparing the distance between the temporary position of the object detection result in the second set of object detection results and the occupied region of the occupied grid with a threshold distance; and if the distance is less than or equal to the threshold distance, classifying and assigning the object to a unit of the occupied region of the occupied grid.

[0011] The steps for determining a semantic raster may include: accumulating semantic evidence values ​​in each cell of the semantic raster using an algorithm that utilizes the Dempster-Shafer framework.

[0012] The first sensor system may include an active positioning sensor system, and the one or more semantic information sources may include a passive positioning sensor system.

[0013] The first sensor system may be a radar system, a lidar system, and / or an ultrasonic system.

[0014] The one or more semantic information sources may include a camera system configured to capture one or more images of the vehicle environment and extract the semantic information and the second positioning information from the captured one or more images.

[0015] A second aspect of this disclosure provides a computer-readable medium for storing computer-readable instructions that, when executed by a processor, cause the computer to perform the method of the first aspect.

[0016] A third aspect of this disclosure provides a computing device including a memory and a controller, the controller being configured to: determine an occupancy grid representing a vehicle environment, the occupancy grid including occupancy probability information of a first set of object detection results, wherein the occupancy probability information is determined based on first positioning information obtained from a first sensor system, the first positioning information indicating one or more positions of the first set of object detection results relative to the vehicle; obtain semantic information from one or more semantic information sources and second positioning information associated with the semantic information, the semantic information including object classification information of a second set of object detection results, the second positioning information being obtained from a second sensor system, the second positioning system indicating one or more positions of the second set of object detection results relative to the vehicle; and combine the object classification information of the second set of object detection results with the occupancy probability information of the occupancy grid to generate a categorized occupancy grid. Attached Figure Description

[0017] To fully understand this disclosure, exemplary aspects will be described with reference to the accompanying drawings, in which:

[0018] Figure 1 It is a two-dimensional representation of the vehicle;

[0019] Figure 2 This is a schematic block diagram of a computing device according to embodiments of the present disclosure;

[0020] Figure 3 This is a schematic block diagram illustrating the workflow of components of a computing device according to embodiments of the present disclosure;

[0021] Figure 4A This is an example diagram of a grid cell;

[0022] Figure 4B This is an example diagram of a semantic grid;

[0023] Figure 4C This is an example diagram showing how categories occupy grid cells;

[0024] Figure 5 This is a flowchart illustrating operations performed by a computing device according to embodiments of the present disclosure; and

[0025] Figure 6 This is a flowchart illustrating operations performed by a computing device according to embodiments of the present disclosure. Detailed Implementation

[0026] The aspects of this disclosure described herein provide methods and apparatus in which semantic information obtained from one or more semantic information sources can be combined with an occupancy grid to obtain a categorized occupancy grid. Such a categorized occupancy grid indicates the area occupied by one or more objects that have been appropriately categorized. Thus, semantic information obtained from one or more semantic information sources can be used to enhance the high-accuracy localization of objects in a vehicle environment provided by occupancy grid information.

[0027] The sensors used to obtain information for forming the occupation grid can be part of a lidar system, radar system, and / or ultrasonic system. Each system within these systems is configured to provide high positioning accuracy. Appropriate sensor fusion models can be used to combine data from the individual systems. However, these systems themselves do not provide classification of detected objects. Example object classifications include traffic signs, guardrails, other vehicles, streetlights, and so on.

[0028] In the example described in this paper, semantic information is obtained from a semantic sensor system. Semantic information includes object classification information; in other words, information describing the classification of objects detected by the semantic sensor system.

[0029] In various implementations, a semantic sensor system may include a camera system operable to capture images of the vehicle's environment. The captured images may undergo segmentation analysis to classify and assign objects to those detected in the captured images. Machine learning techniques provide the capability to identify objects with high confidence levels.

[0030] Camera systems can also be used to determine the location of detected objects relative to a vehicle. When used to estimate location information, a camera system can be called a passive positioning system. This contrasts with active positioning systems such as radar, lidar, and / or ultrasonic systems, which rely on the principle of emitting signals and measuring the scattered or reflected echo signals. A disadvantage of passive positioning systems is that their accuracy tends to decrease in adverse environmental conditions such as darkness, heavy fog, and heavy rain.

[0031] The implementation described herein provides a combination of high accuracy and reliability of object classification determined by semantic sensor systems (e.g., camera systems) and active localization systems (e.g., radar, lidar, and / or ultrasonic systems).

[0032] Figure 1 A vehicle 10 equipped with a device 20 according to one or more embodiments of the present disclosure is shown, the device 20 being used to map the environment of the vehicle 10. The device 20 may have various components mounted at different locations on the vehicle 10, but for simplicity, the device 20 is depicted as a single component mounted on the vehicle. The device 20 includes one or more sensors for sensing objects and / or free space in the environment surrounding the vehicle 10. For example, the device 20 includes one or more radar sensors, lidar sensors, ultrasonic sensors, and / or imaging sensors (e.g., stereo camera systems or RGB cameras) for detecting objects in the environment surrounding the vehicle 10. The device 20 is an environmental perception system configured to perform functions such as object tracking and free space estimation. Such an environmental perception system can be configured to map the environment of the vehicle 10 according to the methods disclosed herein.

[0033] In device 20, sensors are mounted on vehicle 10, enabling them to sense objects within a field of view F of the vehicle's surroundings, particularly sensing distances from vehicle 10 to objects in multiple directions within the field of view. For example, sensors may be mounted at the front corner of vehicle 10. The field of view is depicted as extending in the forward direction of vehicle 10, but it can extend in any direction of the vehicle. Furthermore, the field of view can be a partial view of the vehicle 10's surroundings, or it can be a panoramic view of the entire surroundings. It should be understood that the number, position, and orientation of the sensors in device 20 can be selected to achieve the desired field of view.

[0034] Figure 2 This is a schematic block diagram illustrating the functional components of a computing device 200 used in an exemplary embodiment of this disclosure. The computing device 200 is an electronic device for generating a classification grid as described herein. The computing device 200 may be part of or in communication with an automated vehicle control system 205 responsible for controlling fully or partially automated vehicles.

[0035] The computing device 200 includes a sensor system 210, which may include one or more of a radar system, a lidar system, and / or an ultrasonic system.

[0036] Sensor system 210 may be a radar sensor system comprising one or more radar antennas configured to transmit radar signals, such as modulated radar signals (e.g., chirped signals). The signals may be acquired or detected at one or more antennas and are commonly referred to as echo signals. Echo signals may be generated by the reflection of the transmitted radar signals onto an obstacle. The one or more antennas may be provided individually or as an antenna array, wherein at least one of the one or more antennas transmits radar signals and at least one of the one or more antennas detects echo signals. The detected or acquired echo signals represent the change in amplitude / energy of an electromagnetic field over time.

[0037] Alternatively or additionally, sensor system 210 may include one or more lidar or ultrasonic sensors, which are similarly configured to emit signals to the vehicle's surroundings and detect reflected signals from the surroundings and thus detect objects in the surroundings. Since sensor system 210 is used to determine occupancy grids, it may be referred to as an occupancy sensor system.

[0038] The computing device includes a camera system 215. The camera system 215 may include an RGB camera configured to capture images of the environment surrounding the vehicle. For example, the camera may be mounted on the vehicle and facing forward to capture images of the road and objects on and near the road (e.g., other vehicles, road signs, guardrails, streetlights, etc.). The camera system 215 can be used to extract semantic information from the captured images using a segmentation algorithm.

[0039] Camera system 215 may include a stereo camera system, and the distance from the stereo camera system to objects within the field of view of the stereo camera can be determined via a stereo algorithm. The distance information obtained from the stereo camera system can be combined with detection data obtained from sensor system 210 to determine the occupancy grid.

[0040] As used herein, semantic information refers to information describing objects in the vehicle environment, such as object classification. Example object classifications could include other vehicles, road signs, guardrails, streetlights, and so on. Semantic information can be obtained by analyzing images captured by the camera system 215, for example, using segmentation algorithms. Furthermore, semantic information can be obtained by comparing the vehicle's position with map data.

[0041] Camera system 215 may include a traffic sign recognition (TSR) module 220. TSR module 220 is configured to provide information about any detected traffic signs observed by the cameras of camera system 215 of the vehicle. Information from traffic signs may include speed limits, lane information, place names, and any other information displayed on the traffic sign. TSR module 220 can use any type of image recognition software known in the art and can incorporate various aspects of machine learning techniques, such as the use of neural networks. In this way, the information extracted from traffic signs by the TSR module is highly accurate. Additionally, TSR module 220 can estimate the position of traffic signs by analyzing images acquired by camera system 215. However, the accuracy of sign position estimation using TSR module 220 in conjunction with camera system 215 may not be as accurate as object localization using radar, lidar, or ultrasonic systems.

[0042] The computing device 200 includes a high-definition (HD) mapping module 225. The HD mapping module 225 acquires HD map data mapping the vehicle's environment. The HD map data can be obtained from a cloud server or a locally stored cache. The HD map data includes semantic information about objects in a specific area. For example, the locations of categorized objects such as road signs, intersections, and service stations can be included in the HD map data. The HD map data can be stored locally in a map database. However, the HD mapping module 225 can access the HD map data stored on a remote server 201 via the network interface 230 of the computing device 200.

[0043] The computing device 200 includes a vehicle positioning (VP) module 235. The vehicle positioning system 235 can use a Global Navigation Satellite System (GNSS), such as GPS, Galileo, GLONASS, etc. The vehicle positioning module 235 can combine information received from GNSS with information obtained from an Inertial Navigation System (INS). Thus, the vehicle positioning module 235 is configured to provide real-time indication of the vehicle's position. The HD mapping module 225 can use the vehicle's position to display the vehicle's location on an HD map.

[0044] While HD map data offers high accuracy, the vehicle positioning module 235 introduces a degree of uncertainty, making it difficult to estimate the position of classified objects relative to the vehicle. For example, non-military applications of GPS often exhibit uncertainties on the order of 10 meters. Tall buildings, tunnels, and cloudy weather also reduce positioning accuracy. Embodiments of this disclosure allow for the combination of semantic information from semantic sources such as HD maps with occupancy information from high-accuracy lidar, radar, and / or ultrasonic systems, which can help solve positioning problems.

[0045] The computing device 200 may include a user interface 240. The user interface 240 may include a display for showing information to a user. For example, the display may show the user real-time location information. The user interface may include an input device, such as a keypad. Input and output means may be provided via a touchscreen interface. Additionally, audio input / output may be provided via one or more microphones and speakers.

[0046] The computing device 200 includes a memory 245. The memory 245 may include volatile and non-volatile storage devices. The memory 245 may have an operating system 246 installed therein. The memory 245 may have computer-readable code stored therein, the execution of which can provide the functionality of the aspects of this disclosure described herein. In addition to the memory 245, it should be understood that one or more other components of the computing device 200 may include dedicated memory resources.

[0047] The computing device 200 includes a controller 250. The controller 250 includes one or more processing elements for executing computer-readable code 247. The controller 250 includes an occupancy grid determiner 251, a semantic grid determiner 252, and a classification occupancy grid determiner 253. As will be explained, embodiments of this disclosure provide accumulation of occupancy data through the occupancy grid determiner 251, accumulation of semantic data through the semantic grid determiner 252, and a combination of occupancy grid data and semantic data through the classification occupancy grid determiner 253.

[0048] Figure 3 This is a schematic block diagram illustrating the functional components of the computing device 200 in more detail.

[0049] Occupancy grid determiner 251 includes occupancy sensor modeling module 260, which is configured to acquire detection data 261 (e.g., distance and rate of change of distance data) from occupancy sensor 265 and calculate the positions of one or more objects in the environment surrounding the vehicle based on the detection data 261. Occupancy sensor 265 may correspond to one or more sensors of sensor system 210. Occupancy sensor modeling module 260 can be provided for each sensor system 210 supported by the mapping system. For example, occupancy sensor modeling module 260 can be provided for radar systems, lidar systems, and ultrasonic systems, thereby enabling the processing of detection data from the respective sensor systems.

[0050] For implementations including radar, lidar, or ultrasonic sensors, the corresponding occupancy sensor modeling module 260 can acquire echo signals (e.g., detected at one or more antennas) and apply analog-to-digital (A / D) conversion to them. The occupancy sensor modeling module 260 can convert the delay between the transmitted signal and the detected echo signal into distance data indicating the distance between the object and the vehicle, and also calculate the object's orientation based on the echo signal (e.g., by comparing multiple signals from different antennas). The distance data can be obtained by correlating the echo signal with the transmitted radar signal to acquire the delay.

[0051] Based on detection data 261 obtained from the corresponding occupancy sensor 265 (e.g., from one or more of a lidar, radar, or ultrasonic system), the occupancy sensor modeling module 260 uses the detection data 261 to generate occupancy evidence 262. Depending on the specific sensor system 210, the occupancy evidence 262 may take the form of an occupancy grid with probability values ​​indicating an area occupied by one or more objects in the vehicle environment. More specifically, the occupancy sensor modeling module 260 includes a sensor model that takes the acquired detection data 261 obtained from the occupancy sensor 265 as input and generates the occupancy grid from that data (it should be understood that any sensor model can be used—e.g., any physical model, deterministic model, or inverse sensor model). For example, the occupancy sensor modeling module 260 may receive multiple detection results from the occupancy sensor 265 of the sensor system 210, each detection result including a distance value and azimuth angle (indicating the distance and direction of the detected object or free space relative to the vehicle). The detection results can then be converted into a set of occupancy probabilities for the cells in the occupancy grid. Occupancy grids can be obtained for each sensor system 210, such as radar-based occupancy grids, lidar-based occupancy grids, and ultrasound-based occupancy grids.

[0052] Occupation evidence 262 from each sensor system 210 can then be provided to occupation evidence accumulator 263. Occupation evidence accumulator 263 calculates an occupation grid in the Earth reference frame based on the occupation evidence 262 obtained from the occupation sensors 265 of the various supporting sensor systems 210. Accumulation algorithms can be applied to radar-based occupation grids, lidar-based occupation grids, and ultrasound-based occupation grids. Furthermore, occupation evidence 262 can be accumulated along with historical occupation data. Example accumulation algorithms include algorithms based on Dempster-Shafer evidence theory, recursive Bayesian estimation, central limit theorem, or any other accumulation algorithm known in the art.

[0053] Figure 4AAn exemplary occupancy grid 400 is shown. Occupancy grid 400 comprises a grid of cells. An occupancy grid coordinate system is used to define cell positions. The coordinate system can be a right-handed coordinate system, with the origin, for example, attached to a corner of the grid. The coordinates of the lower right cell are (0,0), and the coordinates of each adjacent cell increase by 1 along the direction of movement from the first cell. The occupancy grid can be defined in a Earth reference frame, a vehicle reference frame, or another reference frame. For example, the occupancy grid can be generated in a vehicle reference frame, so that the coordinates of the occupancy grid each correspond to fixed coordinates in the vehicle reference frame. Each cell can employ an occupancy value indicating whether the location is occupied by an object. For example, in the example shown, a black cell indicates that the cell is occupied by an object (and corresponds to the location where the sensor detects the object). On the other hand, a white cell indicates that the cell is not occupied by an object. Finally, a gray cell can indicate that the occupancy status of the cell is unknown, for example, because another object is located between the cell and the sensor, causing no signal to reach the location corresponding to the cell. The occupancy value of each cell is calculated based on detection data 261 obtained from the occupancy sensors 265 of the respective sensor systems 210. For example, based on the distance, azimuth, and uncertainty of the detection results, occupancy evidence for each cell is calculated, describing the probability that each cell is the origin of the detection results (i.e., the location containing the detection results). The occupancy grid can be a Bayesian occupancy grid, where the probability of cell occupancy is calculated based on evidence values ​​obtained from object and / or free space detection results; however, any type of occupancy grid can be used without departing from this disclosure. For example, the occupancy probability of a cell can be given by p(occ) = 0.5 + 0.5·p(c|d), where p(c|d) is the occupancy evidence value for cell c given object detection result d. Similarly, the free space probability can be given by p(free) = 0.5 + 0.5·p(c|f), where p(c|f) is the free space evidence value for cell c given free space detection result f. In some embodiments, the occupancy grid can include a grid based on the probability of each cell being occupied, using P(occ) and / or P(free) values. The probability value of an occupied cell can then be assigned to each cell based on the object and / or free space detection results. The occupancy grid can be rendered as, for example, a grayscale image or a similar image (0 for white / black, 1 for black / white) based on the probability values. Alternatively, if the occupancy probability is above an upper threshold, each cell can be assigned "occupied," or if the occupancy probability is below a lower threshold, each cell can be assigned "empty." In some implementations, each grid cell can be assigned only as "empty" or "occupied." In other implementations, when the thresholds differ, if the occupancy probability is between the upper and lower thresholds, the cell can also be assigned "unknown." It should be understood that other methods can be used to generate the occupancy grid based on the detection data 261.

[0054] Once the occupancy grid determiner 251 generates the occupancy grid 400, the occupancy grid 400 is output to the classification occupancy grid determiner 253, so that the occupancy data obtained therefrom can be combined with the semantic evidence obtained from the semantic grid determiner 252.

[0055] Figure 5 This is a flowchart illustrating the operations performed to determine one or more semantic grids and combine semantic information with the occupying grid 400.

[0056] In step 5.1, one or more semantic sensor sources 270 provide detection data 271 to the semantic information modeling module 273 of the semantic grid determiner 252. The semantic source 270 may include a camera system 215, which may include a TSR module 220.

[0057] Detection data 271 can be obtained by analyzing image data acquired from camera system 215. The detection data 271 related to objects observed in the camera images can be further analyzed to provide localization information for one or more object detection results. The depth information can then be combined with the camera's azimuth information to determine the localization information of the object detection results relative to the vehicle. Localization information can be obtained by inferring depth information using deep learning techniques. However, such methods are often computationally expensive and / or provide sparse depth maps. Furthermore, this passive system produces unreliable information under adverse environmental conditions such as darkness, heavy fog, and heavy rain.

[0058] In step 5.2, the detection data 271 can be projected onto a grid representing a planar view of the vehicle environment. This grid 405... Figure 4B As shown in the figure. In this example, the detected planar object 406 is projected onto the grid 405.

[0059] After projecting the detected object 406 onto the grid 405, in step 5.3, an uncertainty range can be applied to the positioning information according to an appropriate uncertainty model. The uncertainty level can depend on factors such as camera resolution or the estimated distance of the detected object 406 from the camera. The uncertainty level can vary linearly or exponentially with distance from the camera. Alternatively, a uniform uncertainty range can be applied. The uncertainty range can be represented as a region of object space, for example... Figure 4B Box 407 is shown in the diagram.

[0060] In addition to the location information determined based on detection data 271, a transient evidence value can be used to determine whether a specific object detection result belongs to a specific object category. Example categories may include guardrails, road signs, traffic cones, another vehicle, or any other object commonly found on the road.

[0061] In step 5.4, cells 408 of grid 405 can be assigned to objects belonging to a specific object category and assigned a transient evidence value for that category. Cells can be selected by rasterizing box 407. Temporary evidence values ​​can then be assigned to cells 408 of grid 405. For example, a uniform evidence value can be applied to each cell 408. Alternatively, the temporary evidence value can depend on the distance from sensor 270.

[0062] Individual grids 405 can be provided for each category. Such grids can be called category grids. Category grids can be considered semantic grids for a specific category type. For example, a category grid can be provided for road sign classification, where each cell 408 can be assigned a provisional evidence value between zero and 1 for the road sign. A separate category grid can be provided for guardrails, where each cell can be assigned a provisional evidence value for the guardrail. Additional category grids can be provided for different object categories, such as another vehicle, street light, pedestrian, etc. This provisional semantic evidence 272 is obtained by the semantic information modeling module 273.

[0063] Once one or more classification grids are generated, provisional evidence values ​​can be output to semantic evidence accumulator 275. Semantic evidence accumulation then occurs in step 5.5. Semantic evidence from different sensors can be fused using a fusion algorithm based on the Dempster-Shafer evidence model. Here, the different sensors can be different camera sensors.

[0064] The Dempster-Shafer evidence model allows each classification grid to model different hypotheses. According to the Dempster-Shafer evidence model, the semantic classification space for a particular classification is represented as a set of independent masses representing the degree of trust, which is constructed into a discriminative framework:

[0065] Ω = {set of classes}

[0066] Using Dempster-Shafer fusion rules New evidence can be integrated into the existing cell representation:

[0067]

[0068]

[0069]

[0070] In this example, different classes A, B, and C can be different classification types, such as traffic signs, other vehicles, guardrails, streetlights, etc. Quality indices 1 and 2 can be quality values ​​from different semantic sensor sources 270. Additionally or alternatively, different indices can indicate quality values ​​that change over time. In this way, the Dempster-Shafer framework allows for the accumulation of semantic evidence values.

[0071] Each cell in the classification grid has multiple qualities: one quality for the classification type of the grid, and another quality for the classification considered in combination with other possible classification types. Potential conflicts within cells can be resolved by using combined qualities from other classification types. Various evidence fusion rules can be applied to resolve conflicts. Information from different classification types can originate from the same sensor source 270 or from different sensor sources.

[0072] The resulting raster is a cumulative classification raster, which generates a trust value for each cell of the classification raster, indicating the level of trust that the cell is occupied by objects of a specific classification type. Classification rasters can be determined for each classification type of interest.

[0073] It is also possible to accumulate evidence over time. Temporal accumulation can be achieved using a temporal filtering algorithm based on the Dempster-Shafer combination rule. Accumulating semantic evidence over time allows temporary semantic evidence indicating that an object detection result belongs to a specific object category to be accumulated together with historical semantic evidence of that category.

[0074] This allows us to generate classification grids for each classification type.

[0075] In step 5.6, the evidence values ​​of each cumulative classification raster can be transformed using a pignistic transformation to obtain the confidence level of each semantic evidence value for each classification type raster. The equation for the pignistic transformation is as follows:

[0076]

[0077] In step 5.7, the cumulative classification raster with the highest confidence level can then be selected. For example, a classification raster with cumulative semantic evidence values ​​for traffic sign classification can be selected. The selected transformed classification raster can then be output to the classification occupancy raster determiner 253. The selected transformed classification raster can also be referred to as a semantic raster because it is the raster selected to provide semantic information to be combined with the occupancy raster 400.

[0078] In step 5.8, the classification occupancy grid determiner 253 combines the information contained in the occupancy grid with the information contained in the transformed semantic grid. The step of merging information from the occupancy grid with information from the transformed semantic grid can be performed using a joint probability rule.

[0079] For any given cell, the semantic confidence value obtained from that cell in the transformed semantic raster can be multiplied by the occupancy probability obtained from that cell in the occupied raster. The result of this multiplication is the probability that the cell is occupied and classified in a particular category.

[0080] p(A|occupancy)=p(occupancy)·pig(A)

[0081] Therefore, the classification occupancy grid, which includes the classification occupancy probability values, was determined.

[0082] Semantic information from other sources can also be incorporated into the categorical occupying grid. For example, HD map data that includes semantic data can be combined with the categorical occupying grid. This can be achieved by overlaying the HD map on top of the categorical occupying grid and fitting the categorized objects in the categorical occupying grid to the categorized objects in the HD map.

[0083] Figure 4C A classification occupies grid 410 according to another example of this disclosure. In this example, one or more of the semantic sensor sources 270 report the presence of object 411 in the vehicle environment. The likelihood of an object belonging to various possible categories can be determined and compared. The object category with the highest likelihood can be selected. For example, a camera using machine learning determines the presence of a traffic sign, and a TSR module extracts information from the sign.

[0084] In this example, the occupancy raster generated using radar, lidar, and / or ultrasound systems is generated in the manner described above. The resulting occupancy raster can then be converted into a list of objects using segmentation algorithms such as clustering algorithms, density-based noise applied spatial clustering (DBSCAN), K-means, expectation-maximization using a Gaussian mixture model (GMM), or any other suitable algorithm known in the art. Objects in the occupancy raster can be listed as outlines, clusters, or bounding boxes.

[0085] The reported semantic object (traffic sign 411 in this example) can then be associated with the corresponding occupying object 412. This association can be performed by measuring the distance between the semantic object and the occupying object and comparing the measured distance to a threshold. If the measured distance is less than or equal to the threshold distance, the reported object can be associated with the occupying object. Additionally or alternatively, a shape metric of the occupying object can be measured and compared to the shape of the reported object, for example, using a machine learning algorithm. For instance, since the shape of a traffic sign is substantially planar, a match can be determined with an occupying object having an elongated shape. The semantic information of the reported object can then be assigned to the occupying object 412.

[0086] Figure 6 This is a flowchart illustrating example operations performed by the computing device 200. Although Figure 5 The steps shown are presented in an exemplary order, but it should be understood that one or more of these steps may be executed in parallel by individual processor cores.

[0087] In step 6.1, an occupancy grid representing the vehicle environment is determined. The occupancy grid includes occupancy probability values ​​determined based on a set of object detection results. Occupancy information is determined based on positioning information obtained from a first sensor system such as radar, lidar, and / or ultrasonic systems. The positioning information indicates one or more positions of the first set of object detection results relative to the vehicle.

[0088] In step 6.2, semantic information is obtained from one or more semantic information sources. The semantic information sources can be one or more sensor systems, such as a camera system. The semantic information includes object classification information for a set of object detection results and positioning information indicating the location of that set of object detection results relative to one or more positions of the vehicle. The accuracy of the positioning information obtained from the semantic information sources is generally lower than that obtained from the sensors used to generate the occupying grid.

[0089] In step 6.3, the object classification information from the second set of object detection results is combined with the occupancy probability information of the occupancy grid to generate a categorized occupancy grid. This can be performed by generating a semantic grid including semantic evidence values ​​based on location information obtained from semantic information sources. Semantic grids can be formed by accumulating semantic information from multiple semantic sensor sources. More than one semantic grid can be generated, each focusing on a specific classification type. A semantic grid for a specific classification type can include grids of evidence values ​​generated using evidence values ​​from other classification types to resolve potential conflicts.

[0090] Therefore, the resulting combined occupancy grid achieves improved localization accuracy for object detection compared to individual semantic grids. Furthermore, the occupancy grid is further improved by incorporating object classifications from one or more sensor sources.

[0091] It should be understood that the method described in this paper provides several improvements over existing object classification systems. A classification occupancy grid is provided, which combines information from other semantic sources. The steps of obtaining the semantic grid and combining the semantic grid with the occupancy grid can be largely independent of the occupancy grid processing algorithm. This allows the semantic layer to be easily applied to different occupancy grids with different sensor groups.

[0092] In the foregoing description, exemplary aspects have been described with reference to several exemplary embodiments. Therefore, this specification should be considered illustrative rather than restrictive. Similarly, the figures shown in the accompanying drawings, which emphasize the functionality and advantages of the exemplary embodiments, are presented for illustrative purposes only. The architecture of the exemplary embodiments is flexible and configurable enough that it can be utilized in ways other than those shown in the figures.

[0093] In one exemplary implementation, the software implementations presented herein may be provided as computer programs or software, such as one or more programs having instructions or sequences of instructions, which are included or stored in an article of manufacture such as a machine-accessible or machine-readable medium, an instruction store, or a computer-readable storage device, each of which may be non-transitory. Programs or instructions on non-transitory machine-accessible media, machine-readable media, instruction stores, or computer-readable storage devices can be used to program computer systems or other electronic devices. Machine or computer-readable media, instruction stores, and storage devices may include, but are not limited to, floppy disks, optical disks, and magneto-optical disks, or other types of media / machine-readable media / instruction stores / storage devices suitable for storing or transmitting electronic instructions. The techniques described herein are not limited to any particular software configuration. They may be applicable to any computing or processing environment. As used herein, the terms “computer-readable,” “machine-accessible medium,” “machine-readable medium,” “instruction store,” and “computer-readable storage device” shall include any medium capable of storing, encoding, or transmitting instructions or sequences of instructions for execution by a machine, computer, or computer processor and causing the machine / computer / computer processor to perform any of the methods described herein. Furthermore, in this art, software is often referred to as taking an action or causing a result, either in one form or another (e.g., program, procedure, process, application, module, unit, logic, etc.). Such expressions are simply a shorthand way of describing how the execution of software by a processing system causes the processor to perform actions to produce a result.

[0094] Some implementations can also be achieved by fabricating application-specific integrated circuits, field-programmable gate arrays, or by interconnecting appropriate networks of conventional component circuits.

[0095] Some implementations include a computer program product. The computer program product may be a storage medium, instruction store, or storage device that stores instructions thereon or therein that can be used to control or cause a computer or computer processor to perform any of the exemplary implementations described herein. The storage medium / instruction store / storage device may include, for example, but not limited to, optical discs, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory, flash memory cards, magnetic cards, optical cards, nanosystems, molecular memory integrated circuits, RAID, remote data storage / archiving / warehousing, and / or any other type of device suitable for storing instructions and / or data.

[0096] Some implementations stored on any of a computer-readable medium, instruction storage unit, or storage device include hardware for controlling the system and software for enabling the system or microprocessor to interact with a human user or other mechanisms using the results of the exemplary embodiments described herein. Such software may include, but is not limited to, device drivers, operating systems, and user applications. Finally, such a computer-readable medium or storage device also includes software for performing the exemplary aspects of this disclosure as described above.

[0097] Included in the system's programming and / or software are software modules used to implement the processes described herein. In some example embodiments herein, the module comprises software, but in other example embodiments herein, the module comprises hardware or a combination of hardware and software.

[0098] While various exemplary embodiments of this disclosure have been described above, it should be understood that they are presented as examples and not as limitations. It will be apparent to those skilled in the art that various changes in form and detail may be made therein. Therefore, this disclosure should not be limited to any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

[0099] Furthermore, the purpose of this abstract is to enable patent offices and the public, especially scientists, engineers, and practitioners in the field who are unfamiliar with patent or legal terminology or wording, to quickly determine the substance and nature of the technical disclosure of this application from a cursory examination. The abstract is not intended to limit the scope of the exemplary embodiments presented herein in any way. It should also be understood that any process recited in the claims need not be performed in the order presented.

[0100] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of what may be claimed, but rather as descriptions of specific features of particular implementations described herein. Some features described in the context of individual implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented individually or in any suitable sub-combination in multiple implementations. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in certain circumstances, and the claimed combination may be for sub-combinations or variations thereof.

[0101] In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0102] Some illustrative embodiments and implementations have now been described, and it is clear that the foregoing, presented by way of example, is illustrative and not restrictive. In particular, while many of the examples presented herein relate to specific combinations of device or software elements, these elements can be combined in other ways to achieve the same purpose. Actions, elements, and features discussed in connection with only one embodiment are not intended to exclude similar roles in other embodiments.

[0103] The apparatus described herein may be implemented in other specific forms without departing from its characteristics. The above embodiments are illustrative and not limiting of the described systems and methods. Therefore, the scope of the apparatus described herein is indicated by the appended claims rather than the foregoing description, and modifications falling within the meaning and scope of equivalents of the claims are included therein.

Claims

1. A computer implementation method for mapping a vehicle's environment, the computer implementation method comprising the following steps: An occupancy grid representing the vehicle environment is determined, the occupancy grid including occupancy probability information of a first set of object detection results, wherein the occupancy probability information is determined based on first positioning information obtained from a first sensor system, the first positioning information indicating one or more positions of the first set of object detection results relative to the vehicle; Semantic information and second location information associated with the semantic information are obtained from one or more semantic information sources, the semantic information including object classification information of a second set of object detection results, and the second location information indicating one or more positions of the second set of object detection results relative to the vehicle; The semantic information and the second location information are used to determine a semantic grid representing the vehicle environment, the semantic grid comprising grids of evidence values ​​for one or more object classification types; and The object classification information of the semantic grid is combined with the occupancy probability information of the occupancy grid to generate a classification occupancy grid.

2. The computer implementation method according to claim 1, wherein, The steps to determine a semantic raster include the following: The object detection results in the second set of object detection results are projected onto a grid representing the vehicle environment; Based on the uncertainty value of the second positioning information of the object detection results in the second set of object detection results, the object space region of the object detection results in the second set of object detection results is determined; and Based on the object space region, one or more classifying grid cells are assigned to be occupied by the object detection results in the second group of object detection results, and each classification is related to a different classification type.

3. The computer implementation method according to claim 1 or 2, further comprising the following steps: The confidence value of a cell is calculated by applying a pignistic transformation to the evidence values ​​of one or more cells of the semantic raster. as well as The confidence values ​​of one or more cells of the semantic grid are combined with the occupancy probability values ​​of one or more corresponding cells of the occupancy grid.

4. The computer implementation method according to claim 3, wherein, The step of combining the confidence values ​​of one or more cells of the semantic grid with the occupancy probability values ​​of one or more corresponding cells of the occupied grid includes: multiplying the confidence values ​​of the one or more cells of the semantic grid with the occupancy probability values ​​of one or more corresponding cells of the occupied grid.

5. The computer implementation method according to claim 1, wherein, Combining the object classification information of the semantic raster with the occupancy probability information of the occupied raster includes the following steps: Select the object detection results with object classification from the second group of object detection results; The second location information is used to identify the temporary location of the object detection result in the second set of object detection results; The occupied region of the occupied grid corresponding to the object detection result in the first group of object detection results is identified as the temporary position close to the object detection result in the second group of object detection results; and The object categories of the object detection results in the second group are assigned to the occupied areas of the occupied grid corresponding to the object detection results in the first group.

6. The computer implementation method according to claim 5, wherein, The steps to identify the occupied area of ​​an occupied grid include the following: The distance between the temporary position of the object detection result in the second group of object detection results and the occupied area of ​​the occupied grid is compared with a threshold distance; and If the distance is less than or equal to the threshold distance, the object is classified and assigned to the cell of the occupied region of the occupied grid.

7. The computer implementation method according to claim 1, wherein, The steps for determining a semantic raster include: accumulating semantic evidence values ​​in each cell of the semantic raster using an algorithm that utilizes the Dempster-Shafer framework.

8. The computer implementation method according to claim 1, wherein, The first sensor system includes an active positioning sensor system, and the one or more semantic information sources include a passive positioning sensor system.

9. The computer implementation method according to claim 1, wherein, The first sensor system is a radar system, a lidar system, and / or an ultrasonic system.

10. The computer implementation method according to claim 1, wherein, The one or more semantic information sources include a camera system configured to capture one or more images of the vehicle environment and extract the semantic information and the second positioning information from the captured one or more images.

11. A computer-readable medium storing computer-readable instructions, which, when executed by a processor, cause the computer to perform a computer-implemented method according to any one of claims 1 to 10.

12. A computing device, the computing device comprising a memory and a controller, the controller being configured to: An occupancy grid representing the vehicle environment is determined, the occupancy grid including occupancy probability information of a first set of object detection results, wherein, The occupancy probability information is determined based on first positioning information obtained from the first sensor system, the first positioning information indicating one or more positions of the first group of object detection results relative to the vehicle; Semantic information and second location information associated with the semantic information are obtained from one or more semantic information sources, the semantic information including object classification information of a second set of object detection results, and the second location information indicating one or more positions of the second set of object detection results relative to the vehicle; The semantic information and the second location information are used to determine a semantic grid representing the vehicle environment, the semantic grid comprising a grid of evidence values ​​for one or more object classification types; as well as The object classification information of the semantic grid is combined with the occupancy probability information of the occupancy grid to generate a classification occupancy grid.

Images