Visual and wireless combined three-dimensional mapping and advanced driver assistance system for autonomous vehicles

Abstract

The invention relates to visual and wireless joint three-dimensional mapping and advanced driver assistance systems for autonomous vehicles. A system for mapping an outdoor environment includes at least one map comprising an access point (AP) location map and a reflector map generated from a plurality of wireless signals collected by a plurality of automotive vehicles. As at least one of the plurality of automotive vehicles traverses a mapped area, a crowd-sourced dataset is collected from individual vehicles of the plurality of automotive vehicles and sourced from a plurality of perception sensors. A data packet is created from the crowd-sourced dataset, the data packet comprising a set of wireless positioning samples and a set of visual features, the data packet being communicated to a cloud-based database where cloud-based mapping is performed. A plurality of distance measurements yields a circular AP candidate location within a free space operating window of a vehicle operation of the plurality of automotive vehicles. Application of a plurality of reflectors defined at a plurality of planar reflective surfaces adds to the AP candidate location.

Description

Technical Field

[0001] This disclosure relates to a vehicle location mapping system using wireless technology. Background Technology

[0002] Wireless signals and visual features are used for vehicle localization and mapping, respectively. Localization using wireless signals typically requires accurate mapping of the wireless infrastructure before use. The Global Positioning System (GPS) operation using wireless signals for vehicles, including automobiles such as cars, trucks, vans, SUVs, autonomous vehicles, and electric vehicles, provides the wireless infrastructure but can be negatively affected by environmental conditions including buildings, structures, and reflective surfaces. If negative environmental conditions exist in the vehicle environment that reduce the accurate use of wireless signals, the precise position or orientation of the vehicle is required.

[0003] Multipath propagation is also considered to degrade the performance of wireless-based positioning systems. In wireless and radio communications, multipath propagation is a phenomenon where a signal reaches the receiving antenna via two or more paths. Causes of multipath include atmospheric conduction, ionospheric reflection and refraction, and reflection from bodies of water and ground objects such as mountains and buildings. When the same signal is received along more than one path, multiple signal path receptions cause interference and phase shifts in the received signal, and therefore, using the received signal can generate inaccurate vehicle locations. Destructive interference causes the signal to fluctuate in strength, which can cause the wireless signal to become too weak to be properly received in certain areas.

[0004] Therefore, while current vehicle positioning systems have achieved their intended purpose, a new and improved vehicle location mapping system is still needed. Summary of the Invention

[0005] According to several aspects, the system for mapping an outdoor environment includes at least one map comprising an access point location map identifying the locations of multiple access points (APs) and a reflector map generated from multiple visual features and multiple wireless signals collected from multiple vehicles. A crowdsourced dataset is collected from individual vehicles and derived from multiple sensing sensors as at least one of the multiple vehicles traverses the mapping area. A set of wireless positioning measurements includes: time of flight, angle of arrival, channel state information, and power delay profile. A data packet is created from the crowdsourced dataset, comprising a set of wireless positioning samples and a set of visual features, and this data packet is transmitted to a cloud database where cloud mapping is performed. Multiple distance measurements generate circular AP candidate locations within a free-space operating window of vehicle operation for at least one of the multiple vehicles, wherein the distance measurements, combined with the application of multiple reflectors defined at multiple planar reflective surfaces, improve the AP candidate locations.

[0006] In another aspect of this disclosure, the wireless positioning measurements include: time of flight, angle of arrival, channel state information, and power delay curve.

[0007] In another aspect of this disclosure, the collected sensing sensor data includes images from one or more cameras, images from one or more laser imaging detection and ranging (lidar) systems, and images from radar systems.

[0008] In another aspect of this disclosure, additional sensor data is collected, including data from GNSS, vehicle speed, vehicle yaw, and vehicle CAN bus data.

[0009] In another aspect of this disclosure, the AP location map and the reflector map respectively include candidate locations of access points (APs) and the media access control (MAC) identity corresponding to the APs.

[0010] In another aspect of this disclosure, the location of a potential signal reflector is identified by the AP location map and the reflector map, the potential signal reflector defining a surface from which wireless signals can be reflected.

[0011] In another aspect of this disclosure, at least one of a plurality of vehicles is equipped with a radio receiver that provides distance measurements to different APs, wherein the distance measurements are provided as either line-of-sight (LOS) measurements or non-line-of-sight (NLOS) measurements.

[0012] In another aspect of this disclosure, the AP location map and reflector map also contain semantic data that identifies walls, buildings, or other real-world objects.

[0013] In another aspect of this disclosure, at least one fused local map is created for an individual vehicle and an optimized global map of the wireless AP and the planar surface is created, wherein the AP location map and the reflector map may also be combined with data uploaded from one or more prior vehicle maps.

[0014] In another aspect of this disclosure, the cloud mapping process includes data uploaded by multiple vehicles, utilized visual features, and various wireless positioning procedures applied to create AP location maps and reflector maps.

[0015] According to several aspects, a system for mapping an outdoor environment includes at least one map generated from multiple wireless signals collected by multiple vehicles. An onboard processing unit of at least one of the multiple vehicles includes perception sensor data derived from at least one camera, lidar system, or radar system, as well as data from a GPS unit. A semantic feature detection module detects lane edges of a road. A 3D position detection module detects 3D positions of planar surfaces approaching the multiple vehicles. An image feature extraction module identifies objects including corners and descriptors including pixels related to a given vehicle position. The output of the image feature extraction module is transmitted to a 3D feature coordinate module, which determines 3D feature coordinates via a motion reconstruction structure of one of the multiple vehicles. A model generator receives the outputs from the 3D position detection module, the 3D feature coordinate module, and vehicle sensor data and distance data. An optimizer receives data from the model generator and calculates the position of one of the vehicles and any identified objects to be input into the at least one map.

[0016] In another aspect of this disclosure, the at least one map includes an access point location map that identifies the locations of multiple access points (APs) and a reflector map generated from multiple visual features and multiple wireless signals collected from the multiple vehicles.

[0017] In another aspect of this disclosure, a cloud database is provided where the access point (AP) location map and the reflector map are mapped in the cloud.

[0018] In another aspect of this disclosure, the optimizer is limited to one of a Kalman filter and a nonlinear least squares solver.

[0019] In another aspect of this disclosure, the loop closure detection module identifies whether an object or surface was previously identified and becomes identified a second or subsequent time.

[0020] In another aspect of this disclosure, the on-board processing unit also includes distance data derived from the angle of attack (AoA) sensor.

[0021] In another aspect of this disclosure, the on-board processing section also includes vehicle sensor data, including odometer information, inertial measurement unit (IMU), wheel speed sensor (WSS), and visual odometer (VO) data.

[0022] According to several aspects, a method for collecting data and mapping an outdoor environment includes: using one or more cameras or lidar systems to apply data processing steps for individual vehicles to detect reflective surfaces, such as via semantic segmentation; collecting the reflective surfaces as a dataset; fitting the reflective surfaces of the dataset to a planar model; creating one or more access point (AP) maps with estimated AP locations and planar surfaces; developing multiple planar surface maps; combining wireless AP distance information with planar surface detection to estimate the true AP locations; and applying particle filters to obtain the spatial distribution of AP locations and vehicle attitude.

[0023] In another aspect of this disclosure, the method further includes extracting visual features and matching and tracking said visual features for use in odometers and loop closure.

[0024] In another aspect of this disclosure, the method also includes collecting multiple maps created by multiple vehicles.

[0025] 1. A system for mapping an outdoor environment, comprising:

[0026] At least one map, comprising an access point location map identifying the locations of multiple access points (APs), and a reflector map generated from multiple visual features and multiple wireless signals collected from multiple vehicles;

[0027] The crowdsourced dataset is collected from individual vehicles among the plurality of vehicles as at least one of the plurality of vehicles passes through the mapped area and originates from multiple sensing sensors.

[0028] A group with wireless positioning measurement values;

[0029] A data packet created from the crowdsourced dataset, the data packet including a set of wireless location samples and a set of visual features, is transmitted to a cloud database where a cloud mapping process is performed; and

[0030] Multiple distance measurements are generated to produce a circular AP candidate position within a free-space operation window of at least one of the multiple vehicles, wherein the multiple distance measurements, in addition to the application of multiple reflectors defined at multiple planar reflective surfaces, improve the AP candidate position.

[0031] 2. The system for mapping outdoor environment according to Scheme 1, wherein the wireless positioning measurements include: time of flight, angle of arrival, channel state information, and power delay curve.

[0032] 3. The system for mapping an outdoor environment according to Scheme 2, wherein the crowdsourced dataset collected from the plurality of sensing sensors includes images from one or more cameras, images from one or more laser imaging detection and ranging (lidar) systems, and images from radar systems.

[0033] 4. The system for mapping an outdoor environment according to Scheme 3, wherein additional sensor data is collected, including data from GNSS, vehicle speed, vehicle yaw and vehicle CAN bus data.

[0034] 5. The system for mapping outdoor environments according to Scheme 1, wherein the AP location map and the reflector map respectively include candidate locations of access points (APs) and the media access control (MAC) identity corresponding to the APs.

[0035] 6. The system for mapping an outdoor environment according to Scheme 1, wherein the location of a potential signal reflector is identified by the AP location map and the reflector map, the potential signal reflector defining a surface from which wireless signals can be reflected.

[0036] 7. The system for mapping an outdoor environment according to claim 1, wherein at least one of the plurality of vehicles is equipped with a radio receiver that provides distance measurements to different APs among the APs, wherein the distance measurements are provided as either line-of-sight (LOS) measurements or non-line-of-sight (NLOS) measurements.

[0037] 8. The system for mapping an outdoor environment according to Scheme 1, wherein the AP location map and the reflector map further contain semantic data for identifying roads and intersections.

[0038] 9. The system for mapping an outdoor environment according to Scheme 1 further includes at least one fused local map created for the plurality of vehicles and an optimized global map of the wireless AP and the plurality of planar reflective surfaces, wherein the AP location map and the reflector map are also combined with data uploaded from one or more prior generated vehicle maps.

[0039] 10. The system for mapping an outdoor environment according to Scheme 1, wherein the cloud mapping process includes data uploaded from the plurality of vehicles, utilized visual features, and wireless positioning procedures applied to create the AP location map and the reflector map.

[0040] 11. A system for mapping an outdoor environment, comprising:

[0041] At least one map generated from multiple wireless signals collected from multiple vehicles;

[0042] At least one of the plurality of vehicles has an on-board processing unit that includes perception sensor data from at least one camera, lidar system or radar system and data from a GPS unit.

[0043] A semantic feature detection module for detecting lane edges on roads;

[0044] A 3D position detection module for detecting 3D positions of objects close to the planar surfaces of the plurality of vehicles;

[0045] The image feature extraction module identifies objects including those at corners and descriptors including pixels related to the location of a given vehicle.

[0046] The output of the image feature extraction module is transmitted to the 3D feature coordinate module, which determines the 3D feature coordinates via a motion recovery structure of one of the plurality of vehicles.

[0047] The model generator receives outputs from the 3D position detection module and the 3D feature coordinate module, as well as vehicle sensor data and distance data; and

[0048] An optimizer that receives data from the model generator calculates the location of one of the vehicles and any identified objects to input into the at least one map.

[0049] 12. The system for mapping an outdoor environment according to claim 11, wherein the at least one map includes an access point location map that identifies the locations of multiple access points (APs) and a reflector map generated from multiple visual features and multiple wireless signals collected from the multiple vehicles.

[0050] 13. The system for mapping the outdoor environment according to Scheme 12 further includes a cloud database where the cloud mapping of the access point (AP) location map and the reflector map is performed.

[0051] 14. The system for mapping an outdoor environment according to Scheme 11, wherein the optimizer is defined as one of a Kalman filter and a nonlinear least squares solver.

[0052] 15. The system for mapping an outdoor environment according to Scheme 11 further includes a loop closure detection module, which identifies whether an object or surface was previously identified and becomes identified a second or subsequent time.

[0053] 16. The system for mapping an outdoor environment according to Scheme 11, wherein the vehicle-mounted processing unit further includes distance data derived from an angle-of-attack (AoA) sensor.

[0054] 17. The system for mapping an outdoor environment according to Scheme 11, wherein the on-board processing section further includes vehicle sensor data, including odometer information, inertial measurement unit (IMU), wheel speed sensor (WSS), and visual odometer (VO) data.

[0055] 18. A method for mapping an outdoor environment, comprising:

[0056] Using one or more cameras or lidar systems to apply data processing steps to individual vehicles to detect reflective surfaces, such as via semantic segmentation;

[0057] Collect the reflective surfaces as a dataset;

[0058] Fit the reflective surface of the dataset to a planar model;

[0059] Create a map of one or more access points (APs) with estimated AP locations and planar surfaces;

[0060] Develop multiple planar surface maps;

[0061] Combining wireless AP distance information with planar surface detection to estimate the true AP location; and

[0062] Particle filters are applied to obtain the spatial distribution of AP location and vehicle attitude.

[0063] 19. The method according to claim 18 further includes extracting visual features and matching and tracking the visual features for use in odometers and loop closure.

[0064] 20. The method according to Scheme 18 further includes collecting multiple maps created by multiple vehicles.

[0065] Other application areas will become apparent from the description provided herein. It should be understood that the description and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0066] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0067] Figure 1 It is an illustration of a system and method for mapping an outdoor environment according to an exemplary aspect;

[0068] Figure 2 It is used for Figure 1 A planar view of the semi-circular candidate source surface of the system;

[0069] Figure 3It is by Figure 2 Modified floor plan to show potential AP locations;

[0070] Figure 4 It is a graph that identifies the power versus time relationship for first-, second-, and third-order reflections;

[0071] Figure 5 It demonstrates the Figure 1 A flowchart illustrating the steps involved in processing each vehicle using the system and methods described.

[0072] Figure 6 It demonstrates how to align maps generated from multiple vehicles to create maps for use in [the context of a project]. Figure 1 A flowchart of the steps for the final map of the system and methods;

[0073] Figure 7 It is a flowchart illustrating the steps of the offline mapping process in the cloud;

[0074] Figure 8 It is a plan view showing the mapping process performed on a single car vehicle;

[0075] Figure 9 It is a demonstration of targeting Figure 1 A flowchart of the cloud mapping process performed by the system and methods; and

[0076] Figure 10 It is a plan view showing three hypotheses used to infer the location of AP. Detailed Implementation

[0077] The following description is exemplary in nature and is not intended to limit this disclosure, application, or use.

[0078] refer to Figure 1The system and method for mapping an outdoor environment 10 use visual features and wireless signals to generate one or more maps, including an access point (AP) location map 12 and a reflector map 14. A crowdsourced dataset 16 is initially collected. Crowdsourced data 16 is collected from multiple individual vehicles, including a main vehicle 18 and multiple other vehicles 20a, 20b, 20c, 20d, 20e. Crowdsourced data 16 is data derived from various perception sensors as the main vehicle 18 and the multiple other vehicles 20a, 20b, 20c, 20d, 20e pass through a mapped area, where the mapped area is defined as any area within the driving path of one of the multiple vehicles. The collected perception sensor data includes images from one or more cameras 22, images from one or more laser imaging detection and ranging (lidar) systems 24, radar images, etc. Wireless positioning measurements are also collected, including: vehicle time-of-flight, vehicle angle of arrival, channel state information, power delay curves, etc. It can also collect other sensor data, such as vehicle global navigation satellite system (GNSS) data, vehicle speed, vehicle yaw, a set of vehicle CAN bus data, etc.

[0079] According to several aspects, at least one of the vehicles in the main vehicle 18 is equipped with a radio receiver 26, such as, but not limited to, WiFi Fine Time Measurement (FTM), 5G, etc. The environment in which the main vehicle 18 operates may impede the performance of the Global Positioning System (GPS) and hinder the identification of AP locations. The AP location map 12 and reflector map 14 therefore contain candidate locations of access points (APs) and their corresponding Media Access Control (MAC) IDs. The locations of potential signal reflectors are identified by the AP location map 12 and reflector map 14, which define surfaces from which wireless signals can be reflected. The AP location map 12 and reflector map 14 may further include image features developed by the system, such as Scale Invariant Feature Transform (SIFT) and their coordinates. The AP location map 12 and reflector map 14 further include other relevant semantic data for identifying, for example, walls, buildings, roads, intersections, etc. Radio receiver 26 can also provide distance measurements to different APs, but due to the aforementioned signal reflector, multiple distances may be reported, and the measurements may be provided as line-of-sight (LOS) or non-line-of-sight (NLOS) measurements, as discussed in more detail below in the accompanying drawings.

[0080] Data packet 28 is created from the crowdsourced dataset 16 and includes a set of wireless location samples 30 and a set of visual features 32, which are transmitted, for example, by a radio receiver 26 to a cloud database 34, where a cloud mapping process 36 is performed. The cloud mapping process 36 includes processing uploaded data from individual vehicles, collectively utilizing visual features, algorithms, and wireless location algorithms to generate an AP location map 12 and a reflector map 14. Local maps are fused to create optimized global maps of wireless access points and planar surfaces for the individual vehicles. The AP location map 12 and reflector map 14 may also be combined with data uploaded from a pre-existing or previous map 38 generated from one or more vehicles, which is created through ground surveys, aerial imagery, etc.

[0081] refer to Figure 2 And refer to again Figure 1 Distance measurements generate circular candidate locations 40 within the free-space operating window 42 of vehicle operation. For example, the direct line of sight from the host vehicle 18 to the candidate AP 44 may be obstructed by an obstacle 46. Improved AP candidate locations are given by adding the distance measurements to reflectors defined by multiple planar surfaces, such as the reflective surface 48 of building 50. A first reflection distance 52 from the reflective surface 48 to the candidate AP 44 is added to or combined with a second reflection distance 54 from the host vehicle 18 to the reflective surface 48 to generate a free-space distance 56 defining the free-space operating window 42. Multiple measurements are further refined over time by fusing them, for example, through triangulation, and may include ranges with reflections 58. Several other items, such as planar surfaces of signs, surfaces of parking lights, and visual features such as trees, can be identified as reflective surfaces and are stored in the AP location map 12 and the reflector map 14.

[0082] refer to Figure 3 And refer to again Figure 2 Individual surfaces among multiple planar surfaces (e.g., reflective surfaces) create semi-circular candidate source surfaces. The mapping process requires the use of a first-order reflection estimation source AP. Therefore, measurement model assumptions are made. Free-space distance 56 defines, for example, a distance (r) measurement from a sensor. X and Y coordinates are assigned to identify the extent of reflective surface 48. The curve representing the circular candidate location 40 provides the extent of the true, accurate location of the object or candidate AP 44 based on the reflection data.

[0083] refer to Figure 4 And refer to again Figure 3 Marker 60 shows the power 62 as it changes over time 64. It is assumed that the power in second-order and higher-order reflections is negligible, therefore the p3 value and higher values ​​are negligible and are ignored.

[0084] Continue to refer to Figure 3 and Figure 4 The first two important path lengths and corresponding power are obtained via power delay curves or other mechanisms, such as fine time measurement (FTM), where (r1p1) represents the LOS path length and power, and (r2p2) represents the first reflection, or (r1p1) represents the first reflection and (r2p2) is a higher-order reflection.

[0085] By sensing, a planar surface that can cause reflection is detected. The possible location of the emitter can be determined by defining the following equation 1:

[0086] Equation 1:

[0087]

[0088] Equation 1 defines a set of points that, after a distance r, terminate at the origin upon reflection from the line segment defined by endpoints p1 and p2.

[0089] LOS distance model loss is typically Consider the possibility of reflection using Equation 2 below:

[0090] Equation 2:

[0091] .

[0092] refer to Figure 5 Flowchart 66 identifies how data, such as reference data, is processed in the onboard processing section 68 of an individual vehicle within an automobile. Figure 1 The main vehicle 18 shown processes data either in onboard processing or cloud processing 70. Onboard processing segment 68 includes data sourced from reference... Figure 1 The data 72 is from one or more cameras 22, lidar systems 24, or radar-derived sensing sensors. The onboard processing unit 68 further includes vehicle sensor data 74, such as data from odometer information, inertial measurement unit (IMU) 76, wheel speed sensor (WSS) 78, visual odometer (VO) data, and data from GPS device 80. The onboard processing unit 68 further includes distance data 82 derived from, for example, angle-of-attack (AoA) sensors.

[0093] The sensor data 72 is transmitted to multiple modules, including: a semantic feature detection module 84 for detecting lane edges; a 3D position detection module 86 for detecting the 3D position of planar surfaces; and an image feature extraction module 88, which performs operations such as scale-invariant feature transformation (SIFT) programmed to identify objects such as corners and descriptors such as pixels at a given location. The output of the image feature extraction module 88 is transmitted to each of the following: a 3D feature coordinate module 90, which determines 3D feature coordinates via, for example, a motion recovery structure of the main vehicle 18; and a loop closure detection module 92, which identifies whether an object or surface was previously identified and becomes identified a second or subsequent time.

[0094] The outputs from the 3D position detection module 86, the 3D feature coordinate module 90, and the loop closure detection module 92, along with vehicle sensor data 74 and distance data 82, are transmitted to the model generator 94. Data from the model generator 94 is transmitted to the optimizer 96, which may be, for example, a Kalman filter or a nonlinear least squares solver. The optimizer 96 calculates the positions of vehicles such as the main vehicle 18 and any identified objects. The output from the optimizer 96 is transmitted to and generates a vehicle map 98. The output from the semantic feature detection module 84 is transmitted directly to the vehicle map 98 and added to the vehicle map 98 after vehicle pose recognition. Note that the model generator 94, optimizer 96, and vehicle map 98 can be processed either in the vehicle or in the cloud.

[0095] Sensor information is used to simultaneously estimate the master pose and the location of various features (SLAM). A semantic segmentation network is trained to identify planar reflective surfaces. Image features are estimated and mapped. After learning the vehicle pose, semantic features such as lane edges are added to the map.

[0096] refer to Figure 6 Map fusion flowchart 100 shows that maps constructed from individual vehicles (such as reference maps) Figure 5 The identified vehicle map 98, along with (n) additional maps 102, are from a reference. Figure 1Data from the identified previous map 38 is combined and, for example, stored in the cloud to create the final map 104. The previous map 38 contains more information about connections and other missing information. Data from all maps (including vehicle map 98, the plurality of additional maps 102, and the previous map 38) is passed through register module 106, where, for example, lane edges from all maps can be registered to correct any deviations in the location information of map features. Register module 106 aligns the map data to align all map data from the various maps. The aligned map output from register module 106 is then passed to fusion module 108, which uses a weighted average to fuse the map data and smooths the data before outputting the final map 104.

[0097] refer to Figure 7 And refer to again Figure 1 The offline mapping process flowchart 110 illustrates multiple processing steps that can be executed on the crowdsourced dataset 16 uploaded to the cloud processing group 114 via the cloud edge 112 through data packet 28. The dataset 116 uploaded by a single vehicle can be processed as follows: The sample sequence of the dataset 116 uploaded by a single vehicle is transmitted to the feature extraction and reconstruction module 118, which extracts data features and reconstructs 3D features in the order of the sample sequence. Multiple local point clouds generated separately by the feature extraction and reconstruction module 118 are transmitted to the point cloud storage module 120, which stores and tracks the multiple local point clouds. The output from the point cloud storage module 120 is directed to the segmentation module 122, which segments the identified planar surface of the sensed object. The sample sequence of the dataset 116 uploaded by a single vehicle is also transmitted in parallel to the sample localization module 124, which assigns the localization data to the identified image. A set of sample locations 126 is output from the sample localization module 124. Further output from the point cloud storage module 120 is directed and stored in the global point cloud 128. Data from the global point cloud 128 can be transmitted to the sample localization module 124 and the segmentation module 122 respectively. The output from the segmentation module 122 is used to create a database including multiple planar surface models 130.

[0098] While performing analysis and processing on the dataset 116 uploaded by individual vehicles, the crowdsourced AP mapping dataset 132 is also processed separately. Data from the set of sample locations 126 is fed to a particle filter 134, which operates to update and resample the data from the set of sample locations 126. A particle filter initialization module 136 receives the output of particle filter 134 and initializes the next or APn object. A set of AP locations 138 defines the final output of the cloud processing group 114 using the output from particle filter 134.

[0099] Note that image processing, including that performed by the feature extraction and reconstruction module 118, can also be performed via one or more vehicles, rather than within the cloud processing group 114. The extracted features and 3D reconstructed image data can then be uploaded via the cloud edge 112 along with data packets 28 from the vehicles.

[0100] refer to Figure 8 And refer to again Figure 1 and Figure 7 An example of the mapping process is as follows. The main vehicle 18 drives through a narrow street 140 that defines an urban canyon. The main vehicle 18 collects a series of sensor data along a timeline from time t0 to time tn. Each frame of the data may include camera images, wireless positioning measurements, GPS data, vehicle speed, yaw, etc. All collected data is uploaded to the cloud, such as reference... Figure 7 The cloud processing group 114 is described for use in offline mapping.

[0101] For cloud locations, there are two mapping processes. Process 1 defines a vehicle's data preprocessing. The goal is to process and integrate a vehicle's sensor measurement samples into three (3) databases: a global point cloud database, a planar surface database, and a sample location database.

[0102] Continue to refer to Figure 7 and Figure 8 The feature extraction and reconstruction module 118 loads a series of images (or lidar, radar) and uses 3D reconstruction algorithms (such as visual SLAM or structure of motion reconstruction (SFM)) to reconstruct 3D point clouds.

[0103] The point cloud storage module 120 uses a point storage algorithm to integrate the point cloud into the existing global point cloud data generated by other crowdsourced data.

[0104] The segmentation module 122 uses surface algorithms to identify valid planar surfaces from the point cloud. For example, surfaces 142, 144, 146, and 148 are detected. Surfaces 142, 144, 146, and 148 are saved to a database that includes multiple planar surface models 130.

[0105] The sample localization module 124 uses camera images from each frame of the data to determine the vehicle's precise location using a localization algorithm from visual SLAM or SFM. Once a 3D point is determined, the sample localization module 124 attaches wireless positioning measurement data (such as FTM, Channel State Information (CSI), Parallel Distributed Processing (PDP), etc.) to that 3D sample point. Each frame of the uploaded sequence of data is then processed, and the created 3D sample points are saved to a database defined by a set of sample locations 126.

[0106] These three databases will then be used as input to the crowdsourced AP mapping dataset 132 of process 2, which defines the crowdsourced AP mapping process, in which the particle filter initialization module 136 initializes the particle filter 134 to locate the specific AP (i.e., APx).

[0107] Particle filter 134 updates its data using wireless location measurement samples from the set of sample locations 126 database. This update process involves multiple iterations until a predetermined completion condition is met. Once completed, the final location of APx is saved to a set of AP locations 138 database.

[0108] Particle weights are calculated in particle filter 134. For particle gj, the weights are calculated as shown in Equation 3 below:

[0109] Equation 3:

[0110] .

[0111] For Equation 3, gj is one of the particles, si is one of the wireless samples, and pk is one of the paths from gj to si. The path can be a direct path (such as p0) or a reflected path (such as p4). PDP(qj,si,pk) is a function that obtains the corresponding power level from the wireless measurements for a given path pk between qj and s8.

[0112] Since the positions of si and gj are known, the lengths of their direct or reflected paths are also known. These path lengths can be converted into time of flight based on the known speed of light. The time-of-flight value can then be mapped to a power delay curve generated from wireless measurements of the position samples.

[0113] Features detected by the perception system can optionally be associated with mapped features. The state includes the attitude of the vehicle (such as the main vehicle 18), the coordinates of the reflector in the region of the main vehicle 18, and the AP position of the individual location in the hypothetical and image feature locations. The state model is developed based on Equations 4 through 8 below:

[0114] .

[0115] Observations include: 1) odometer information, inertial measurement unit (IMU) 76, wheel speed sensor (WSS) 78, visual odometer (VO), etc.; 2) image features; 3) GPS data; 4) reflector coordinates from sensing; and 5) (if available) distance, MAC address, and AoA measurement of the AP. An observation model is developed based on Equations 9 to 12 below:

[0116] .

[0117] By adding cyclic closure constraints, the above system represents a SLAM problem. Multiple AP locations can be estimated for each measurement because the source may be unknown if it is a LOS or NLOS source. AP information is associated with MAC addresses. Solutions can be obtained using Kalman filters, particle filters, or factor graph optimization.

[0118] refer to Figure 9 And refer to again Figures 1 to 8 Flowchart 150 provides a method for performing the cloud mapping process. For individual vehicle data processing step 152, one or more cameras 22, lidar systems 24, etc., are used to detect reflective surfaces, such as through semantic segmentation. Reflective surfaces are collected in sample sequence collection step 154. Data from sample sequence collection step 154 ​​is then fitted to a planar model in mapping step 156. Visual features are also extracted, matched, and tracked for odometer and loop closure. One or more AP maps with estimated AP locations and planar surfaces are then created in AP map generation step 158. Multiple planar surface maps are developed in planar surface map creation step 160. Wireless AP distance information is combined with planar surface detection to estimate the true AP location. (The text then refers to...) Figure 7 The particle filter 134 described is used to obtain the spatial distribution of the AP position and the attitude of the main vehicle. In the map collection step 162, maps created by other vehicles are collected.

[0119] In data fusion step 164, map data from AP map generation step 158, planar surface map creation step 160, and map collection step 162 are fused. Similarly, in fusion step 164, a fused local map is created using data collected from individual vehicles and used to create an optimized global wireless AP map 166 for wireless access points and a global planar surface map 168. The mapping creation process can occur in-vehicle on any of a plurality of vehicles including the primary vehicle 18, or in a reference vehicle. Figure 7 This occurred in the cloud processing group 114.

[0120] The following marginal likelihood function can be defined:

[0121]

[0122] in:

[0123]

[0124] and

[0125] It has variance σ at x 2 The probability of a zero-mean Gaussian

[0126] U(y,a,b) represents the probability of a and b being uniformly distributed.

[0127] |L| is the cardinality of L.

[0128] (wi, θ i) ) = m(li,x0,r i )

[0129] If cond is true, then It is 1, otherwise it is 0.

[0130] These are redundant parameters.

[0131] Given prior values ​​P(z), Ph(i), P(Ѱ), etc., the objective is to estimate the posterior value P(z,Hi,Ѱ / y).

[0132] refer to Figure 10 The following measurement model assumptions are provided, giving three possible assumptions for considering the inferred AP position z=(x,y) for the exemplary AP 172, and giving the measured value y =(r1,p1,r2,p2). Three vehicles are shown, including the main vehicle 18, a first vehicle 20a among the other vehicles, and a second vehicle 20b among the other vehicles. AP 172 is shown relative to these three vehicles and with respect to the exemplary reflector 174.

[0133] in:

[0134] H0 is defined by the first line segment 176: z represents the LOS position of AP.

[0135] H1 is defined by the second set of line segments 178' and 178"; z represents the first-order reflection position of AP.

[0136] Ha is defined by the third set of line segments 180', 180”, 180'”: z represents a higher-order reflection, or a reflection from an unknown reflector, or an outlier.

[0137] Assume the receiver power follows the inverse square law. Here, α = 1 represents the LOS signal, and α < 1 represents reflection. For the mapping, the state space consists of the following: the main vehicle attitude, the planar surface position, the AP 172 position, the reference power is p0, and the reflection loss is limited by α.

[0138] According to the first aspect, the cloud side (such as reference) Figure 7The limited cloud processing group 114 creates a crowdsourced hybrid map based on wireless signals and visual features collected from numerous vehicles. Low-end vehicles (defined as cars equipped only with a radio) can use the crowdsourced hybrid map for precise positioning. The crowdsourced hybrid map can also be used to correct multipath errors from wireless positioning signals.

[0139] According to the second aspect, visual positioning enhanced by wireless signal applications, cloud-side (such as reference) Figure 7 Limited cloud processing group 114) creates crowdsourced hybrid maps based on wireless signals and visual features collected from numerous vehicles. High-end vehicles, defined as car vehicles equipped with radios and cameras / visual features, can utilize crowdsourced hybrid maps to enhance the precise positioning process, including under certain conditions, such as, but not limited to, changing lighting conditions, visual features that cannot be tracked, and rapid vehicle movement.

[0140] According to the third aspect, when this system is used for positioning based on visual features on one or more individual vehicles using wireless signals and on-board computing, it does not involve cloud computing.

[0141] According to the fourth aspect, smartphone positioning using only wireless signals can be challenging, especially in urban canyons or multi-story parking structures. Smartphones can 1) compensate for multipath errors and improve positioning accuracy using both reflector and wireless access point (AP) models; and 2) when the smartphone's camera is active, the camera can assist in positioning by utilizing visual features in the point cloud.

[0142] The system and method disclosed herein for mapping an outdoor environment 10 utilizes crowdsourced vehicle sensor data to create maps of wireless access points and reflective surfaces. The system and method utilize visual feature algorithms (e.g., SLAM) to create a 3D model of the environment and extract planar surfaces that can cause multipath reflections. Based on the created planar surfaces, wireless reflection paths are modeled, and the precise location of the wireless access points (APs) is determined.

[0143] The system and method disclosed herein for mapping outdoor environments 10 offer several advantages. These advantages include providing a system for mapping outdoor environments using a combination of visual features and wireless signals. Visual features from cameras, lidar, or other sensors are used to identify multipath sources. The map can be combined with other maps via the cloud and subsequently used by lower-level vehicles (vehicles lacking advanced navigation systems) for functions such as positioning. During mapping, visual features are used to identify reflections and moving objects in the environment. Visual features are also used to assist in creating consistent maps and wireless measurements.

[0144] The description in this disclosure is merely exemplary in nature, and variations thereof that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such variations should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A system for mapping an outdoor environment, comprising: At least one map, comprising an access point location map that identifies the locations of multiple access points, and a reflector map generated from multiple visual features and multiple wireless signals collected from multiple vehicles. The crowdsourced dataset is collected from individual vehicles among the plurality of vehicles as at least one of the plurality of vehicles passes through the mapped area and originates from multiple sensing sensors. A group with wireless positioning measurement values; A data packet created from the crowdsourced dataset, the data packet including a set of wireless positioning samples and a set of visual features, is transmitted to a cloud database where a cloud mapping process is performed. and Multiple distance measurements between a planar reflective surface and access point candidate locations and vehicles generate a circular access point candidate location within a free-space operation window of at least one of the multiple vehicles, wherein a first reflection distance from the planar reflective surface to the access point candidate location is added to or combined with a second reflection distance from the vehicle to the planar reflective surface to generate a free-space distance defining the free-space operation window, thereby improving the access point candidate location.

2. The system for mapping outdoor environments according to claim 1, wherein, The wireless positioning measurements include: time of flight, angle of arrival, channel state information, and power delay curve.

3. The system for mapping outdoor environments according to claim 2, wherein, The crowdsourced dataset collected from the plurality of sensing sensors includes images from one or more cameras, images from one or more laser imaging detection and ranging systems, and images from radar systems.

4. The system for mapping outdoor environments according to claim 3, wherein, Additional sensor data was collected, including data from GNSS, vehicle speed, vehicle yaw, and vehicle CAN bus data.

5. The system for mapping outdoor environments according to claim 1, wherein, The access point location map and the reflector map respectively contain candidate locations of access points and the Media Access Control (MAC) identity corresponding to the access points.

6. The system for mapping outdoor environments according to claim 1, wherein, The location of a potential signal reflector is identified by the access point location map and the reflector map, and the potential signal reflector defines a surface from which wireless signals are reflected.

7. The system for mapping outdoor environments according to claim 1, wherein, At least one of the plurality of vehicles is equipped with a radio receiver that provides distance measurements to different access points among the access points, wherein the distance measurements are provided as either line-of-sight (LOS) measurements or non-line-of-sight (NLOS) measurements.

8. The system for mapping outdoor environments according to claim 1, wherein, The access point location map and the reflector map also contain semantic data that identifies roads and intersections.

9. The system for mapping an outdoor environment according to claim 1, further comprising at least one fused local map created for the plurality of vehicles and an optimized global map of the access points and the plurality of planar reflective surfaces, wherein, The access point location map and the reflector map are also combined with data uploaded from one or more previously generated vehicle maps.

10. The system for mapping outdoor environments according to claim 1, wherein, The cloud mapping process includes data uploaded from the plurality of vehicles, utilized visual features, and various wireless positioning procedures applied to create the access point location map and the reflector map.

11. A system for mapping an outdoor environment, comprising: At least one map is generated from multiple wireless signals collected from multiple vehicles, the at least one map including an access point location map that identifies the locations of multiple access points, and a reflector map generated from multiple visual features and multiple wireless signals collected from the multiple vehicles. At least one of the plurality of vehicles has an on-board processing unit that includes sensing sensor data from at least one camera, laser imaging detection and ranging system or radar system, and data from a GPS unit. A semantic feature detection module for detecting lane edges on roads; A 3D position detection module for detecting 3D positions of objects close to the planar surfaces of the plurality of vehicles; The image feature extraction module identifies objects including those at corners and descriptors including pixels related to the location of a given vehicle. The output of the image feature extraction module is transmitted to the 3D feature coordinate module, which determines the 3D feature coordinates via a motion recovery structure of one of the plurality of vehicles. The model generator receives outputs from the 3D position detection module and the 3D feature coordinate module, as well as vehicle sensor data and distance data. as well as An optimizer that receives data from the model generator, the optimizer calculating the location of one of the vehicles and any identified objects to input into the at least one map; Multiple distance measurements between a planar reflective surface and access point candidate locations and vehicles generate a circular access point candidate location within a free-space operation window of at least one of the multiple vehicles, wherein a first reflection distance from the planar reflective surface to the access point candidate location is added to or combined with a second reflection distance from the vehicle to the planar reflective surface to generate a free-space distance defining the free-space operation window, thereby improving the access point candidate location.

12. The system for mapping an outdoor environment according to claim 11 further includes a cloud database where the cloud mapping of the access point location map and the reflector map is performed.

13. The system for mapping an outdoor environment according to claim 11, wherein, The optimizer is limited to one of a Kalman filter and a nonlinear least squares solver.

14. The system for mapping an outdoor environment according to claim 11 further includes a loop closure detection module, which identifies whether an object or surface was previously identified and becomes identified a second or subsequent time.

15. The system for mapping an outdoor environment according to claim 11, wherein, The on-board processing unit also includes distance data derived from the angle-of-attack (AoA) sensor.

16. The system for mapping an outdoor environment according to claim 11, wherein, The on-board processing section also includes vehicle sensor data, including information from the odometer, inertial measurement unit (IMU), wheel speed sensor (WSS), and visual odometer (VO) data.

17. A method for mapping an outdoor environment, comprising: Acquire at least one map, which includes an access point location map that identifies the locations of multiple access points, and a reflector map generated from multiple visual features and multiple wireless signals collected from multiple vehicles. Acquire a crowdsourced dataset, which is collected from individual vehicles among the plurality of vehicles and originates from multiple sensing sensors when at least one of the plurality of vehicles passes through the mapped area; Acquire groups with wireless positioning measurements; A data packet created from the crowdsourced dataset is acquired, the data packet including a set of wireless positioning samples and a set of visual features, the data packet is transmitted to a cloud database, and a cloud mapping process is performed at the cloud database; and Multiple distance measurements are acquired between a planar reflective surface and access point candidate locations and vehicles, which generate a circular access point candidate location within a free-space operation window of at least one of the multiple vehicles. A first reflection distance from the planar reflective surface to the access point candidate location is added to or combined with a second reflection distance from the vehicle to the planar reflective surface to generate a free-space distance defining the free-space operation window, thereby improving the access point candidate location.

18. The method of claim 17, further comprising extracting visual features and matching and tracking the visual features for use in odometer and loop closure.

19. The method of claim 17, further comprising collecting multiple maps created by multiple vehicles.

Images