Sensor data alignment correction and environmental mapping related applications

Abstract

Generating a map associated with the environment may include collecting sensor data received from one or more vehicles and generating a set of links that align the sensor data. A mesh representation of the environment may be generated from the aligned sensor data. The system may determine proposed link additions, deletions, and / or modifications and receive corrections including instructions to add, delete, or modify links. In response to receiving the corrections, the system may realign a window of sensor data associated with the corrections. The corrections and / or their associated sensor data may be collected as training data for machine learning models, which may be trained to generate link modification suggestions and / or determine sensor data that may be associated with insufficient sensor data alignment.

Description

[Technical field]

[0001] This PCT international application claims the benefit of priority to U.S. Patent Application No. 16 / 697,117, filed November 26, 2019, which is incorporated herein by reference. [Background technology]

[0002] Robots and other devices, such as augmented reality devices, may collect sensor data to determine what objects are in the environment surrounding the device and where the device is in the environment relative to those objects. In some examples, the sensor data may be collected to generate a map of the environment and a path for the device through the environment. Inaccuracies in such maps may cause a device that relies on the map to misdetermine the device's location in the environment, may impede actions performed by the device, or may even endanger the safety of the user of the device or other users in the environment.

[0003] Furthermore, the sensor data and / or maps may be incredibly complex and / or coded in machine language and collected over large areas, such as entire cities, any of these factors making it difficult or even impossible for humans to detect errors in the sensor data and / or maps generated therefrom. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Application Serial No. 15 / 674,853 [Patent Document 2] U.S. Patent Application Serial No. 15 / 675,487 [Brief description of the drawings]

[0005] The detailed description will be made with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and the same reference numbers in different drawings indicate similar or identical items. [Figure 1] FIG. 1 illustrates an example scenario in which an autonomous vehicle configured with a positioning component may determine a pose of the autonomous vehicle relative to an environment based at least in part on sensor data and a map, and / or collect sensor data to determine an estimated trajectory and / or map of the vehicle. [Diagram 2] FIG. 2 illustrates a block diagram of an exemplary system including a positioning component, a mapping component, and a correction component. [Figure 3A] FIG. 3A shows an exemplary top-down representation of an environment. [Figure 3B] FIG. 3B illustrates two sets of sensor data associated with two different poses, links between the poses, and an example sensor data alignment. [Figure 4] FIG. 4 illustrates a flowchart of an example process for correcting sensor positioning, sensor data alignment, and / or trajectory, determining maps, and / or generating training data for training machine learning models to identify links to remove, add, or adjust. [Figure 5A] Figure 5A shows two types of link modification: Figure 5A shows a link deletion between two poses located in different tunnels. [Figure 5B] Figure 5B shows two types of link modification: Figure 5B shows a link addition between two poses on either side of the highway. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Generating the map associated with the environment may include collecting sensor data received from one or more vehicles. The vehicles may include autonomous, semi-autonomous, and / or manned vehicles configured with one or more sensors that capture sensor data associated with the environment. The one or more sensors may include a light detection and ranging (lidar) sensor, a radio detection and ranging (radar) sensor, a sound detection and ranging (sonar) sensor, an image capture device (e.g., an RGB camera, an intensity camera, an IR camera, a stereo camera, a depth camera, etc.), or other sensors. As a non-limiting example, the vehicle may be an autonomous vehicle configured with eight lidar sensors. Each of the lidar sensors may return data corresponding to a detected surface in the environment. The data may be represented as points (e.g., data points) having coordinates (e.g., Cartesian coordinates, polar coordinates, etc.) corresponding to at least a portion of the detected surface.

[0007] In some examples, one or more vehicles may pass through a portion of an environment multiple times from the same or similar viewpoints and / or from different viewpoints, and the sensor data captured by the one or more vehicles may be received and aggregated into a collection of sensor data. Thus, a collection of sensor data received from one or more vehicles may include a subset of sensor data associated with the same portion of the environment. For example, the collection of sensor data may include first sensor data associated with a portion of the environment and received from a vehicle, and second sensor data associated with the same portion received from the vehicle at a subsequent time, or second sensor data received from another vehicle. The first sensor data and the second sensor data may be associated with the same, similar, or different sensor poses. In other words, at the time the first sensor data is captured, the sensor(s) of the vehicle are located and oriented in a first pose (i.e., pose may include position and / or orientation), and the second sensor data is captured while the sensor(s) of the vehicle (or another vehicle) are located in the same, similar, or different pose. In other words, a portion of the environment may be seen from the same and different poses. In other words, at the time the first sensor data is captured, the sensor(s) of the vehicle are located and oriented in a first pose (i.e., pose may include position and / or orientation), and the second sensor data is captured while the sensor(s) of the vehicle (or another vehicle) are located in the same, similar, or different pose. In other words, a portion of the environment may be seen from the same and different poses. Thus, the second sensor data may be associated with the same portion of the environment as the first sensor data, but the second sensor data may include at least a portion of sensor data associated with an alternative or additional portion of that portion. For example, the first sensor data may include data points indicative of a surface of a first side of a building, and the second sensor data may include data points indicative of a surface of a first side and / or a second side of the same building, with the second sensor data being captured from a different pose than the first sensor data.

[0008] Determining the map of the environment using at least a portion of the collected sensor data may include determining a link between a first sensor data and a second sensor data of the collected sensor data, where the link may indicate that the first sensor data and the second sensor data are associated with the same portion of the environment and / or the link may indicate that the first pose is associated with the second pose. Determining the link may be based at least in part on determining that the first pose and the second pose satisfy a global parameter, such as, for example, a threshold distance between the poses, a similarity of the sensor data associated with the two poses, etc. Such a link may be a proposed link. Because the first pose may be associated with the first sensor data and the second pose may be associated with the second sensor data, the link may be implicitly (or explicitly, as described above) associated with the first sensor data and the second sensor data. In some examples, the first pose and / or the second pose may be a pose estimated by the autonomous vehicle (e.g., using a Bayes filter (such as a Kalman filter), bundle adjustment, maximum a posteriori estimation (MAP), and / or any simultaneous localization and mapping (SLAM) algorithm), as discussed in more detail in U.S. Patent Application Publication No. 2017 / 0139994, filed August 11, 2017, which is incorporated herein in its entirety.

[0009] In some examples, determining the map may additionally or alternatively include determining an alignment of the first sensor data to the second sensor data based at least in part on the link. For example, determining the alignment may be based at least in part on the existence of a proposed link between the first pose and the second pose. The link may cause the system discussed herein to determine a transformation of the pose(s) to align the sensor data associated therewith and / or attempt to associate the transformation(s) with the link once the transformation(s) are determined. In some examples, the technique may include displaying the proposed link prior to determining the alignment(s) and / or the map. Aligning the first sensor data and the second sensor data may include perturbing at least one of the poses associated with the first sensor data and the second sensor data to reduce an error associated with the alignment. In some examples, the error may include a total residual error between respective data points of the first sensor data and the second sensor data. In other words, the error may be an indication of how far from the distance the first sensor data is aligned with the second sensor data.

[0010] Perturbing the pose may include moving the position and / or orientation of the pose (e.g., changing the lateral and / or vertical changing coordinates associated with the pose, the yaw, roll, and / or pitch associated with the pose), which may be associated with the orientation of the sensor data associated therewith. In other words, a change in the position and / or orientation of the pose may cause a corresponding change in the sensor data associated therewith. In some examples, the pose may be the pose of a separate sensor, or the pose may be a reference frame of one or more sensors, such as the position and / or orientation of a vehicle to which the sensor is associated. In the latter example, the discrete sensor may be associated with the pose of the sensor relative to the pose of the vehicle. Once the error is minimized or otherwise sufficient (e.g., the error falls below an error threshold), by perturbing the pose / aligning the sensor data, a transformation associated with the alignment may be stored in association with the link.

[0011] The techniques discussed herein may include receiving a modification including instructions to add a link, remove a link, or modify a parameter associated with a link. For example, the modification may be received based at least in part on a user interaction with a user interface, and the parameter may include a covariance and / or a threshold distance. The techniques may include initiating a new sensor data alignment based at least in part on receiving the modification. When a link is added, the new alignment may include aligning sensor data that was not previously aligned, while when a link is removed, the new alignment may include separating sensor data that was previously aligned. The modification to a parameter, such as a covariance, may include additionally including or excluding pose and / or sensor data in the new alignment.

[0012] When a correction is received, any new sensor data alignment based on the correction may be limited to realigning sensor data associated with a window of sensor data (e.g., rather than realigning the entire region), although realigning the entire region associated with the correction may be contemplated. In some examples, determining the window may be based at least in part on a threshold distance (e.g., sensor data associated with movement ±2 meters from a position associated with a pose), a threshold time (e.g., sensor data associated with movement ±0.5 seconds from a time associated with a pose), vehicle speed, distance between linked poses, and / or the like. In at least some examples, such windowed reoptimization may include, but is not limited to, downweighting contributions from measurements as a function of distance from the proposed region, such that further measurements from the selected region are incrementally corrected, such as a Gaussian, linear, polynomial, or otherwise.

[0013] Determining the map may include generating a representation of the environment based at least in part on the sensor data alignment (e.g., using the transformation associated with the link in an example where each link has a transformation associated therewith that aligns the respective sensor data). In some examples, determining a map of a large area such as a city may include generating a representation such as a mesh and / or wireframe model of the environment based at least in part on thousands or millions of sensor data alignments. With the computational constraints of current cloud computing, the process of generating a map from sensor data alignments may take up to a week. In some examples, the vehicle may generate a lower resolution map using on-board computing resources and / or by distributing computation across one or more vehicles and / or computing resources.

[0014] The techniques discussed herein may determine sensor data alignment across the environment as part of generating the map (i.e., at map generation time) because the number of alignments may be computationally and time-costly. In some examples, when a correction is received, an option to redetermine the alignment of the associated sensor data may be made available and / or a redetermining may be automatically triggered to evaluate the effectiveness of the correction. Additionally or alternatively, the alignment may be calculated prior to generating the map. In at least one non-limiting example, the technique may include waiting to determine the sensor data alignment until the map generation operation, except for at least those sensor data alignments associated with any correction or corrections for which alignment is triggered by a user selection.

[0015] The techniques may additionally or alternatively include displaying the collected sensor data, the estimated trajectory data associated therewith, and / or the proposed links. The estimated trajectory data may include a series of past poses estimated by the vehicle associated with the sensor data captured by the vehicle. In a non-limiting example, the techniques may include displaying at least the estimated trajectory data, the proposed links, and / or a two-dimensional road representation (e.g., a top-down map). In some examples, the two-dimensional road representation may be used in conjunction with the trajectory data to determine the likelihood that a link may be added, removed, and / or parameters associated therewith may be modified. For example, the techniques may include determining that a total residual associated with the sensor data alignment meets or exceeds a threshold, training a machine learning model to determine the sensor data and / or areas that may be problematic (e.g., may require an added, removed, or adjusted link), and / or the like. In an additional or alternative example, the display may be generated based at least in part on determining that an error (e.g., a residual) associated with the sensor data alignment cannot be reduced below a threshold error.

[0016] The techniques discussed herein may additionally or alternatively include adding and / or removing links, modifying parameters associated with links, and / or adding any modifications received to the links and / or data associated therewith to a training dataset for training a machine learning (ML) model to indicate areas for potentially adding, removing, or modifying links. In some examples, the training data may include an indication of the type of modification (e.g., link addition, link removal, link parameter adjustment) and / or the data associated therewith (e.g., pose data, sensor data, link data).

[0017] The techniques discussed herein may reduce the time to generate a map (e.g., a three-dimensional representation) of an environment, reduce the computational resources and power to generate the map, and improve the accuracy of the map. These techniques may improve the safety of vehicles that use the map to calibrate and / or position sensors for improved accuracy.

[0018] (Example scenario) FIG. 1 illustrates an example scenario 100 including a vehicle 102. In some examples, the vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout its journey, and a driver (or passenger) is not expected to control the vehicle at any time. However, in other examples, the vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. It is contemplated that the techniques described herein may be applied to more than robotic control, such as autonomous vehicles. For example, the techniques described herein may be applied to search, manufacturing, augmented reality, and the like. Additionally, the vehicle 102 is depicted as a land vehicle, but may also be a spacecraft, a seaplane, and / or the like. In some examples, the vehicle 102 may be represented in a simulation as a simulated vehicle. For simplicity, the discussion herein does not distinguish between simulated and real-world vehicles. Thus, references to a "vehicle" may refer to a simulated vehicle and / or a real-world vehicle.

[0019] In accordance with the techniques described herein, the vehicle 102 may receive sensor data from sensors 104 of the vehicle 102. For example, the sensors 104 may include position sensors (e.g., Global Positioning System (GPS) sensors), inertial sensors (e.g., acceleration sensors, gyroscope sensors, etc.), magnetic field sensors (e.g., compasses), position / speed / acceleration sensors (e.g., speedometers, drive system sensors), depth position sensors (e.g., lidar sensors, radar sensors, sonar sensors, time-of-flight (ToF) cameras, depth cameras, ultrasonic and / or sonar sensors, and / or other depth-sensing sensors), image sensors (e.g., cameras), audio sensors (e.g., microphones), and / or environmental sensors (e.g., barometers, hygrometers, etc.).

[0020] The sensors 104 may generate sensor data, which may be received by a computing device 106 associated with the vehicle 102. However, in other examples, some or all of the sensors 104 and / or computing devices 106 may be located separately and / or remotely from the vehicle 102, and data capture, processing, command, and / or control may be communicated to and from the vehicle 102 by one or more remote computing devices over wired and / or wireless networks.

[0021] The computing device 106 may include a memory 108 that stores the perception component 110, the planning component 112, the positioning component 114, and / or the system controller 116. While depicted in FIG. 1 for illustrative purposes, it should be understood that the positioning component 114 may reside on a separate computing device (or otherwise) than any one or more of the other components. In some examples, the perception component 110 may be a primary perception component of other perception components, such as a secondary perception component that may be part of a collision avoidance component. In some examples, the perception component 110, the planning component 112, and / or the positioning component 114 may include a hardware and / or software pipeline that may include one or more GPUs, ML models, Kalman filters, and / or the like.

[0022] In general, the perception component 110 may determine what is in the environment surrounding the vehicle 102, and the planning component 112 may determine how to steer the vehicle 102 according to the information received from the perception component 110. For example, the planning component 112 may determine a trajectory 118 based at least in part on the perception data and / or other information, such as, for example, one or more maps, positioning data generated by the positioning component 114 (e.g., where the vehicle 102 is in the environment relative to the representation 120 of the environment and / or features detected by the perception component 110), etc. The trajectory 118 may include instructions to the controller 116 to operate the drive components of the vehicle 102 to achieve a steering angle and / or steering rate that may result in a vehicle position, vehicle speed, and / or vehicle acceleration. For example, the trajectory 118 may comprise a target direction, a target steering angle, a target steering rate, a target position, a target speed, and / or a target acceleration for the controller 116 to track. The perception component 110, the planning component 112, and / or the positioning component 114 may include one or more machine learning (ML) models and / or other computer-executable instructions.

[0023] In some examples, the perception component 110 may receive sensor data from the sensors 104 and determine data related to objects in the vicinity of the vehicle 102 (e.g., detected objects, instance segmentation, semantic segmentation, 2D and / or 3D bounding boxes, classifications associated with the track), route data specifying the vehicle's destination, global map data identifying roadway features (e.g., features detectable with different sensor modalities useful for localizing an autonomous vehicle), local map data identifying features detected in the vicinity of the vehicle (e.g., locations and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable with various sensor modalities), and the like. The object classifications determined by the perception component 110 may distinguish between different object types, such as, for example, passenger cars, pedestrians, bicyclists, delivery trucks, semi-trucks, traffic signs, and the like. The tracks may include historical, current, and / or predicted object positions, speeds, accelerations, and / or orientations. The data generated by the perception component 110 may be collectively referred to as perception data. Once the perception component 110 generates the sensory data, the perception component 110 can provide the sensory data to the planning component 112 .

[0024] The planning component 112 may use the perception data received from the perception component 110 and / or the positioning data received from the positioning component 114 to determine one or more trajectories, control the movement of the vehicle 102 to traverse a path or route, and / or otherwise control the operation of the vehicle 102, although any such operations may be performed by various other components (e.g., positioning may be performed by a positioning component, which may be based at least in part on the perception data). For example, the planning component 112 may determine a path for the vehicle 102 from a first position to a second position, generate multiple potential trajectories for controlling the movement of the vehicle 102 according to a receding horizon technique (e.g., one microsecond, half a second) substantially simultaneously and based at least in part on the perception data and / or simulated perception data (which may further include predictions regarding detected objects within such data), to control the vehicle to traverse the path (e.g., to avoid any of the detected objects), and select one of the potential trajectories as the trajectory 118 of the vehicle 102, which may be used to generate drive control signals that may be transmitted to drive components of the vehicle 102. FIG. 1 shows an example of such a trajectory 118 represented as arrows indicating direction, speed, and / or acceleration, but the trajectory itself may include instructions for the controller 116, which may then operate the drive system of the vehicle 102.

[0025] In some examples, the positioning component 114 may receive sensor data and / or sensory data and determine a pose of the vehicle 102 based at least in part on the sensor data, sensory data, and / or map. For example, the map may be stored in a memory of the computing device 106 and may include a representation 120 of the environment generated according to any of the techniques discussed herein. In some examples, the representation 120 may be a mesh (as shown in FIG. 1 ) or any other three-dimensional (or other dimensional) representation of the environment. In some examples, the representation 120 may include at least a small area, such as a block or a perimeter of the vehicle, a city, or more, so that the representation 120 may be encoded and / or compressed. In some examples, the representation 120 may be available to the vehicle 102 via a network interface. For example, the representation 120 may be streamed or otherwise transmitted to the vehicle 102 based at least in part on the path and / or position of the vehicle 102. In some examples, the vehicle 102 may determine the map or at least a portion of the map.

[0026] The positioning component 114 can determine the pose (e.g., position and / or orientation) of the vehicle based at least in part on the map (e.g., with respect to the representation 120) by matching the sensor data to the map and determining the relative pose of the vehicle. The representation 120 can include global and / or local coordinates such that determining the pose of the vehicle with respect to the map can be sufficient to determine the global pose (e.g., position and / or orientation with respect to a city-wide, country-wide, or earth-wide map) and / or local pose (e.g., the vehicle's position and / or orientation along a city block) of the vehicle. In some examples, the positioning component 114 can compare the sensor data to the representation 120 and determine a real-time pose of the vehicle based at least in part on the comparison. Additionally or alternatively, the positioning component 114 can calibrate the sensor 104 based at least in part on the representation 120, as discussed in more detail in U.S. Patent Application Publication No. 2013 / 0133994, filed Aug. 11, 2017, which is incorporated herein in its entirety. In some examples, calibrating one of the sensors 104 may include adjusting a pose associated with the sensor relative to the pose of the vehicle 102.

[0027] In some examples, the vehicle 102 may be a mapping vehicle and may or may not include a perception component 110 and / or a planning component 112, and the positioning component 114 may include a mapping component. Such mapping vehicles may include manually operated vehicles equipped with sensors that collect data to generate such maps.

[0028] In some examples, the positioning component 114 may store sensor data, such as sensor data 122, in a log associated with an estimated pose of the vehicle, such as estimated pose 124, when the sensor data 122 is received. In some examples, estimating the pose 124 of the vehicle may be based at least in part on sensor data, such as odometry measurements (e.g., wheel encoder output, IMU output), other sensor information (e.g., GPS position, compass output), perception data, and / or positioning data (e.g., an estimated pose based at least in part on SLAM techniques). In some examples, generating the log may include SLAM techniques and / or components thereof that may perform moving object detection, loop closure detection, and / or the like. In at least some such examples, such loop closure may be used to suggest links, as described in detail herein. The pose 124 may include a position 126 and / or an orientation 128 (e.g., an orientation that may include yaw, pitch, and / or roll). In some examples, multiple estimated poses may be aggregated into an estimated trajectory 130. The sensor data may be associated with one or more of the poses that make up the estimated trajectory 130. In at least one non-limiting example, the estimated trajectory 130 may be determined based at least in part on a SLAM technique that is based at least in part on the map / representation 120. The SLAM technique may include a Bayes filter (such as a Kalman filter), bundle adjustment, maximum a posteriori estimation (MAP), and / or any SLAM algorithm.

[0029] In some examples, the vehicle 102 may transmit the estimated trajectory 130 and associated sensor data to a remote computing device, which may include a distributed computing service, another vehicle, and / or the like. The remote computing device and / or the vehicle 102 may generate a map, which may include the representation 120 and / or updates to the representation 120. In some examples, determining the map may include determining a correction 132 to the estimated trajectory, as indicated by the corrected trajectory 134. Note that the estimated trajectory 130 may be associated with a larger error / smaller degree of alignment of the estimated trajectory with respect to the position and orientation of the roadway 136 compared to the corrected trajectory 134. In some examples, changes to the sensor data alignment may result in a change to the trajectory if the trajectory is estimated based at least in part on the resulting map. In some examples, the corrected trajectory 134 may be transmitted to the vehicle 102.

[0030] In some examples, the error and / or degree associated with the alignment may be used herein to identify regions and / or one or more poses of the estimated trajectory 130 as candidates for link modification (e.g., adding, removing, or adjusting). The degree of alignment and / or the proposed link modification may be the output of an ML model trained based at least in part on supervised techniques (e.g., using GPS data, if available, using data labeling, using perception and / or planning data).

[0031] In some examples, controller 116 may comprise software and / or hardware for operating drive components of vehicle 102 sufficient to track trajectory 118. For example, controller 116 may comprise one or more proportional-integral-derivative (PID) controllers.

[0032] (System example) FIG. 2 illustrates a block diagram of an example system 200 that implements the techniques discussed herein. In some examples, the example system 200 may include a vehicle 202, which may represent the vehicle 102 of FIG. 1. In some examples, the vehicle 202 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that can perform all safety-critical functions throughout the entire journey, and where a driver (or passenger) is not expected to control the vehicle at any time. However, in other examples, the vehicle 202 may be a fully or partially autonomous vehicle having other levels or classifications. Additionally, in some cases, the techniques described herein may be usable by non-autonomous vehicles.

[0033] Vehicle 202 may include vehicle computing device 204, sensors 206, emitters 208, network interfaces 210, and / or drive components 212. Vehicle computing device 204 may represent computing device 106, and sensor 206 may represent sensor 104. System 200 may additionally or alternatively include computing device 214.

[0034] In some examples, the sensors 206 may represent the sensors 104 and may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Unit (IMU), accelerometers, magnetometers, gyroscopes, etc.), image sensors (e.g., Red Green Blue (RGB), Infrared (IR), intensity, depth, time of flight cameras, etc.), microphones, wheel encoders, environmental sensors (e.g., thermometers, hygrometers, light sensors, pressure sensors, etc.), etc. The sensors 206 may include multiple instances of each of these or other types of sensors. For example, the radar sensors may include individual radar sensors located at corners, front, rear, sides, and / or top of the vehicle 202. As another example, the cameras may include multiple cameras positioned at various locations about the exterior and / or interior of the vehicle 202. The sensors 206 may provide input to the vehicle computing device 204 and / or the computing device 214.

[0035] The vehicle 202 may also include emitters 208 for emitting light and / or sound, as described above. The emitters 208 in this example may include interior audio and visual emitters for communicating with passengers of the vehicle 202. By way of example and not limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 208 in this example may also include exterior emitters. By way of example and not limitation, the exterior emitters in this example may include lights or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.) for signaling heading, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0036] Vehicle 202 may also include a network interface 210 that enables communication between vehicle 202 and one or more other local or remote computing devices. For example, network interface 210 may facilitate communication with other local computing devices on vehicle 202 and / or drive components 212. Network interface 210 may also additionally or alternatively enable the vehicle to communicate with other nearby computing devices (e.g., other vehicles, traffic signals, etc.). Network interface 210 may additionally or alternatively enable vehicle 202 to communicate with computing device 214. In some examples, computing device 214 may include one or more nodes of a distributed computing system (e.g., a cloud computing architecture).

[0037] Network interface 210 may include physical and / or logical interfaces for connecting vehicle computing device 204 to another computing device or network, such as network 216. For example, network interface 210 may enable Wi-Fi-based communications, such as frequencies defined by the IEEE 200.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or via an appropriate wired or wireless communication protocol that allows each computing device to interface with other computing devices. In some cases, vehicle computing device 204 and / or sensors 206 may transmit sensor data to computing device 214 over network 216 at a particular frequency, after a predetermined period of time, in near real-time, etc.

[0038] In some examples, the vehicle 202 may include one or more drive components 212. In some examples, the vehicle 202 may have a single drive component 212. In some examples, the drive component 212 may include one or more sensors for detecting conditions around the drive component 212 and / or the vehicle 202. By way of example and not limitation, the sensors of the drive component 212 may include one or more wheel encoders (e.g., rotational encoders) for sensing the rotation of the wheels of the drive component, inertial sensors (e.g., IMUs, accelerometers, gyroscopes, magnetometers, etc.) for measuring the orientation and acceleration of the drive component, cameras or other image sensors, ultrasonic sensors for acoustically detecting objects around the drive component, LIDAR sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive component 212. In some cases, the sensors on the drive component 212 may overlap or supplement corresponding systems (e.g., sensors 206) of the vehicle 202.

[0039] The drive components 212 may include many vehicle systems including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current utilized by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a brake system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking forces to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, and other electrical components). Additionally, the drive components 212 may include a drive components controller that may receive and pre-process data from sensors and control the operation of various vehicle systems. In some cases, the drive components controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components that perform various functions of the drive components 212. Additionally, the drive components 212 may also include one or more communication connections that enable each drive component to communicate with one or more other local or remote computing devices.

[0040] Vehicle computing device 204 may include a processor 218 and a memory 220 communicatively coupled to one or more processors 218. Memory 220 may represent memory 108. Computing device 214 may also include a processor 222 and / or a memory 224. Processor 218 and / or 222 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, processor 218 and / or 222 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs)), gate arrays (e.g., field programmable gate arrays (FPGAs)), and / or other devices or portions of devices that process electronic data and convert the electronic data into other electronic data that may be stored in registers and / or memory.

[0041] Memory 220 and / or memory 224 may be examples of non-transitory computer-readable media. Memory 220 and / or memory 224 may store an operating system and one or more software applications, instructions, programs, and / or data for implementing the methods and functions attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static RAM (SRAM), synchronous DRAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely examples in connection with the description herein.

[0042] In some examples, memory 220 and / or memory 224 may store positioning component 226, perception component 228, planning component 230, mapping component 332, map 234, and / or system controller 236. Positioning component 226 may represent positioning component 114, perception component 228 may represent perception component 110, and / or planning component 230 may represent planning component 112.

[0043] In at least one example, the positioning component 226 may include hardware and / or software to receive data from the sensors 206 to determine a position, velocity, and / or orientation (e.g., one or more of an x ​​position, a y position, a z position, a roll, a pitch, or a yaw) of the vehicle 202. For example, the positioning component 226 may include and / or request / receive a map 234 of the environment and may continually determine a position, velocity, and / or orientation of the autonomous vehicle within the map 234. In one example, the positioning component 226 may receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., using simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine a position, pose, and / or velocity of the autonomous vehicle. In some examples, as discussed herein, the positioning component 226 can provide data to various components of the vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory and / or for generating map data. In some examples, the positioning component 226 can provide the mapping component 232 with the pose (e.g., position and / or orientation) of the vehicle 202 relative to the environment (e.g., via position and / or orientation relative to the map 234) and / or sensor data associated therewith.

[0044] In some examples, perception component 228 may include a predictive system implemented in hardware and / or software. Perception component 228 may detect objects in the environment surrounding vehicle 202 (e.g., identify that an object is present), classify objects (e.g., determine an object type associated with a detected object), segment sensor data and / or other representations of the environment (e.g., identify a portion of the sensor data and / or representation of the environment as associated with a detected object and / or object type), determine characteristics associated with the object (e.g., a track identifying a current, predicted, and / or previous position, orientation, velocity, and / or acceleration associated with the object), and / or the like. Data determined by perception component 228 is referred to as perception data.

[0045] The planning component 230 may receive the position and / or orientation of the vehicle 202 from the positioning component 226 and / or the perception data from the perception component 228 and determine instructions to control the operation of the vehicle 202 based at least in part on any of this data. In some examples, determining the instructions may include determining the instructions based at least in part on a format associated with a system with which the instructions are associated (e.g., a first instruction for controlling the movement of the autonomous vehicle may be formatted in a first format of messages and / or signals (e.g., analog, digital, aerodynamic, kinematic) that the system controller 236 and / or the drive component 212 may analyze / execute, and a second instruction for the emitter 208 may be formatted according to a second format associated therewith).

[0046] The mapping component 232 may operate on the vehicle 202 and / or the computing device 214. In some examples, the mapping component 232 may be downstream (receives output from) the sensors 206, the perception component 228 in the pipeline, and / or the planning component 230. The positioning component 226 may be configured to pass all, a portion, or none of the output of the positioning component 226 to the mapping component 232. In some examples, the mapping component 232 may include components for determining to collect sensor data, perception data, and / or pose data from one or more vehicles, determining links, determining sensor data alignment, and / or determining a map including a representation of the environment based at least in part on the sensor data, links, and / or pose data.

[0047] Memory 220 and / or memory 224 may additionally or alternatively store a collision avoidance system, a ride management system, and / or the like. Although positioning component 226, perception component 228, planning component 230, map 234, and / or system controller 236 are illustrated as being stored in memory 220, any of these components may include processor-executable instructions, machine learning models (e.g., neural networks), and / or hardware, and all or a portion of any of these components may be stored in memory 224 or configured as part of computing device 214. In some examples, a mapping component operating on vehicle 202 may collect and / or encode sensor data (e.g., raw sensor data, sensor data alignment, sensor data with perceptual labels), pose data, and / or perception data for transmission to computing device 214. Mapping components operating on vehicle and / or computing device 214 may perform operations described herein to generate a map based at least in part on the link corrections.

[0048] In some examples, the computing device 214 (and / or 204) may include a tuning component 238. In some examples, the tuning component may include one or more ML models trained with training data as discussed herein. The tuning component may be trained to identify regions that may be candidates for pose, sensor data, and / or link corrections, and / or the tuning component may output link corrections. The tuning component may additionally or alternatively determine a degree of alignment of the estimated pose data to the roadway. Determining the pose and / or sensor data associated with the candidate corrections may be based at least in part on the degree of alignment associated therewith.

[0049] As described herein, the localization component 226, the perception component 228, the planning component 230, the tuning component 238, and / or other components of the system 200 may include one or more ML models. For example, the localization component 226, the perception component 228, the planning component 230, and / or the tuning component 238 may each include a different ML model pipeline. In some examples, the ML model may include a neural network. An exemplary neural network is a bio-inspired algorithm that passes input data through a series of connection layers to produce an output. Each layer in a neural network may include another neural network, or may include any number of layers (convolutional or not). As can be understood in the context of the present disclosure, the neural network may utilize machine learning, which can refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0050] Although described in the context of neural networks, any type of machine learning can be used consistent with the present disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least-angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative dichotomiser 3 (ID3), chi-squared automated interaction detection (CHAID), decision stump, conditional decision tree), Bayesian algorithms (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence (AODE), etc.), and / or ensemble algorithms (e.g., ensemble algorithms ... estimators), Bayesian belief networks (BNNs), Bayesian networks), clustering algorithms (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g., perceptrons, backpropagation, Hopfield networks, RBFNs (Radial Basis FunctionNetwork), deep learning algorithms (e.g., Deep Boltzmann Machines (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), stacked autoencoders), dimensionality reduction algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), ensemble algorithms (e.g., Boosting, Bootstrap Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), Random Forests), SVM (Support Vector Machines), supervised learning, unsupervised learning, semi-supervised learning, etc. Further example architectures include neural networks such as ResNet-50, ResNet-101, VGG, DenseNet, and PointNet.

[0051] Memory 220 may additionally or alternatively store one or more system controllers 236, which may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers 236 may communicate with and / or control corresponding systems of drive component 212 and / or other components of vehicle 202. System controller 236 may control operation of vehicle 202 based at least in part on instructions received from planning component 230.

[0052] 2 is depicted as a distributed system, it should be noted that in alternative examples, components of vehicle 202 may be associated with computing device 214 and / or components of computing device 214 may be associated with vehicle 202. That is, vehicle 202 may perform one or more functions associated with computing device 214, and vice versa.

[0053] (Example of sensor data alignment, trajectory, and linking) 3A illustrates an exemplary top-down representation of an environment 300. The exemplary environment 300 includes two roadways (302 and 304, respectively), a number of light posts 306, and the base of a fountain 308.

[0054] FIG. 3B illustrates data that may be part of the log data generated by the autonomous and / or mapping vehicles. In the illustrated example, the log data is overlaid on a representation of the environment 300 for reference, and the representation of the environment 300 is represented by a gray line. In some examples, the log data may include log data received from one or more autonomous vehicles and / or mapping vehicles. The log data may include, for example, at least a pose 310 (e.g., an estimated position and / or orientation of the vehicle) associated with sensor data 312, both of which may be generated by the first vehicle as it traverses the environment 300. The log data may additionally or alternatively include a pose 314 associated with sensor data 316, both of which may be generated by the first vehicle or the second vehicle as it traverses the environment 300. The sensor data 312 and / or the sensor data 316 may represent lidar data, depth camera data, and / or the like. In some examples, the pose 310 and / or the pose 314 may be part of a trajectory generated by the vehicle.

[0055] The log data may be transmitted to a computing device(s), such as computing device 214, which may determine a link 318 between pose 310 and pose 314 based at least in part on one or more global parameters of a SLAM, loop closure, and / or other mapping algorithm to determine that sensor data associated with two different poses are likely to contain a common view of the environment (e.g., at least some of the two sets of sensor data correspond to the same part of the environment and / or different perspectives of the same part of the environment). For example, the global parameters may include a threshold distance and / or a threshold similarity (e.g., a threshold degree of alignment), and determining the (proposed) link 318 may include determining that the distance between pose 310 and pose 314 is less than or equal to a threshold distance and / or degree of alignment determined between sensor data 312 and sensor data 316, or that the distance meets or exceeds the threshold degree of alignment. In some examples, determining the link 318 may include determining a pose(s) within a threshold parameter of the end of the link (e.g., within the covariance σ of the end of the link or within the covariance 3σ of the end of the link).

[0056] FIG. 3B may depict pre-alignment of sensor data or may depict alignment of sensor data since the sensor data are non-overlapping and / or discontinuous. Determining alignment may include determining sensor data associated with a pose(s). For example, determining sensor data may include determining sensor data that is within a window of the pose, where the window may be a span of distance or time around the pose (e.g., ±2 meters, ±0.5 seconds, although other windows are contemplated, need not be uniform in the positive and negative directions, and may be based at least in part on operational characteristics of the sensor, such as the speed or frame capture rate of the vehicle). Such alignment may include, for example, Iterative Closest Point (ICP) estimation as described herein, or other optimization. In the depicted example, the first sensor data and the second sensor data both include multiple lidar data collections, each associated with a different pose, or in other examples, may be associated with the same pose when the pose is determined at a lower frequency than the frequency at which the sensor data is captured. Determining the alignment may include perturbing an estimated pose associated with the sensor data to reduce error in the alignment of the first data to the second data. For example, aligning sensor data 312 and sensor data 316 may include perturbing pose 310 and / or pose 314 until sensor data 312 and sensor data 316 associated with light post 306 and fountain 308 overlap and / or are contiguous.

[0057] 3B also provides an example of a display that may be triggered by the techniques discussed herein in response to receiving an instruction to modify a link 318 (e.g., by adding a link 318 or modifying a parameter associated with the link 318). Based at least in part on receiving the instruction, alignment of sensor data associated with the link 318 may be triggered or made available for user initiation before a map of the environment is generated, such that alignment resulting from the modification may be evaluated. It will be understood that sensor data alignment of sensor data associated with unmodified links may be delayed until a map is generated, although such alignment may be determined at any other time (e.g., as part of determining the link in some examples).

[0058] (Example Processing) 4 illustrates a flow diagram of an example process 400 for training a machine learning model to generate a map of an environment, identify and / or modify portions of the map, and / or modify links during or prior to generation. In some examples, the example process 400 may be performed by at least a mapping component and / or a tuning component.

[0059] At operation 402, the example process 400 may include receiving sensor data associated with the environment and / or receiving an estimated trajectory according to any of the techniques discussed herein. For example, the sensor data may include lidar data, radar data, infrared image data, visual image data, sonar data, and / or the like. In some examples, the estimated trajectory may be associated with the sensor data. The sensor data and / or the estimated trajectory may be part of a collection of sensor data and / or estimated trajectories received from one or more vehicles. In some examples, the estimated trajectory may include one or more poses. The pose may indicate an estimated position and / or orientation of the vehicle, as estimated based at least in part on a SLAM algorithm, which may be based at least in part on a simple mapping of the environment performed by the vehicle. The pose may be associated with one or more frames of sensor data (e.g., serialized or time-stamped sensor data). For example, the pose may be associated with a frame of sensor data captured at or around the time the vehicle determined the estimated pose.

[0060] In operation 404, the example process 400 may include determining a link based at least in part on the global parameters, the sensor data, and / or the estimated trajectory according to any of the techniques discussed herein. For example, determining the link may include determining edges between nodes of the coefficient graph where the nodes are different poses, and attempting to match sensor data associated with the two different poses. In at least one non-limiting example, the algorithm may sample the trajectory with a frequency (e.g., every 5 meters, which may be based at least in part on the speed of the vehicle) and determine whether another trajectory satisfies the global parameters for the sample. For example, operation 404 may include determining a first pose based at least in part on sampling the trajectory and determining that a distance from the first pose to a second pose of the second trajectory is less than a threshold distance.

[0061] In some examples, the link may include parameters such as a transformation (e.g., determined at least in part based on aligning sensor data associated with two or more poses) and / or a covariance and / or a threshold distance associated with two or more poses. The covariance and / or threshold distance may be used as part of determining the alignment to identify sensor data that may be associated with the same element and / or constraint(s) for perturbing the pose. In some examples, determining the link may be based at least in part on global parameters, such as, for example, a match score threshold, a threshold distance between poses, etc.

[0062] At operation 406, the example process 400 may include determining a first alignment based at least in part on the sensor data and the link according to any of the techniques discussed herein. In some examples, determining the link may be part of determining the alignment. Determining the alignment may include perturbing an estimated pose associated with the sensor data to reduce an error in the alignment of the first data to the second data, as further discussed in U.S. Patent Application Publication No. 2013 / 0133994, filed Aug. 11, 2017, which is incorporated herein in its entirety. In some examples, determining the link and / or alignment may include an anxious search, an iterative closest points (ICP) algorithm, a graph-based SLAM, and / or any other SLAM algorithm. In some examples, the final perturbation of the pose(s) to achieve the alignment may be stored as a transformation associated with the link. In some examples, the transformation may be used to correct and / or refine the estimated pose and / or determine a corrected / refined trajectory. Additionally or alternatively, operation 406 may include determining a map. In an additional or alternative example, operation 406 may include determining an improved trajectory prior to determining the sensor data alignment.

[0063] At operation 408, an example of process 400 may include determining a degree of alignment associated with the first alignment. In some examples, the degree of alignment may include residuals calculated as part of the pose perturbation and / or additional errors associated with the alignment. In additional or alternative examples, an ML model or other algorithm may be trained to determine a degree of alignment associated with a section of a trajectory that includes multiple poses.

[0064] At operation 410, the example process 400 may include determining whether the degree of alignment and / or error meets or exceeds a threshold, according to any of the techniques discussed herein. If the degree of alignment and / or error meets or exceeds a threshold, operation 410 may transition to operation 412; otherwise, the example process 400 may transition to operation 402, 414, or 416.

[0065] At operation 412, the example process 400 may include triggering the display of a notification associated with the first alignment and / or the improved trajectory, according to any of the techniques discussed herein. For example, the example process 400 may include sending a warning message along with data associated with the first alignment and / or the improved trajectory, triggering the display of the first alignment and / or the improved trajectory, coding the first alignment and / or the improved trajectory based at least in part (e.g., color-coding the display of the improved trajectory), and / or the like. In some examples, if the degree of alignment does not meet a threshold and a link exists between the first pose and the second pose, operation 412 may include triggering the display of a suggestion and / or option to remove the link. In additional or alternative examples, if the degree of alignment meets or exceeds a threshold and a link does not exist, operation 412 may include triggering the display of a suggestion and / or option to add a link to the two poses associated with the alignment.

[0066] At operation 414, the example process 400 may include receiving a modification including instructions to add proposed links, remove links, and / or change parameters associated with the links in accordance with any of the techniques discussed herein. In some examples, a sensor data alignment may be determined in response to receiving the instructions, and / or an option to perform the sensor data alignment may be made available (e.g., via a user interface, via a publish-subscribe system to a tuning and / or mapping component).

[0067] FIG. 5A illustrates an example scenario 500 in which removing a link improves the accuracy of the resulting map. FIG. 5A illustrates two tunnels, tunnel 502 and tunnel 504. Because the tunnels may appear very similar in the sensor data, the link determination step may generate link 510 that associates pose 506 in tunnel 502 with pose 508 in tunnel 504. If the map is generated based at least in part on link 510, the sensor data may be sufficiently aligned that the resulting map may merge the two tunnels into a single tunnel. Overpasses and other uniform features may also cause this effect. Removing link 510 may cause the mapping component to skip sensor data associated with pose 506 in an attempt to align it with sensor data aligned to pose 508, resulting in two distinct tunnels in the resulting map. In some examples, the generation of links in this scenario may be permitted by global parameters associated with link generation and / or sensor data alignment.

[0068] FIG. 5B illustrates an example scenario 512 showing two sides (514 and 516, respectively) of a highway divided by a highway fence 518. In the example scenario 512, a pose 520 on highway side 514 was not linked to a pose 522 on highway side 516, despite a significant amount of overlapping sensor data associated with the two poses. Adding link 524 can cause the mapping component to determine a sensor data alignment of the sensor data associated with pose 520 and pose 522, respectively. A map generated from the sensor data alignment may include more detail as a result of the addition of link 542. In some examples, the failure to generate a link in this scenario may be based at least in part on a global parameter that limits the generation of the link.

[0069] In some examples, updating (realigning) the sensor data alignment based at least in part on a modification (e.g., addition, removal, change) to the link may include disassociating the sensor data and / or further pose downweight perturbations from the link, although in additional or alternative examples the perturbations may be weighted equally or otherwise. In some such examples, such realignment may include pose graph relaxation, nonlinear least squares optimization, bundle adjustment, or any other filtering / mapping algorithm (e.g., SLAM).

[0070] In operation 416, the example process 400 may include determining a second map, a second refined trajectory, and / or a second alignment based at least in part on the sensor data and the modification according to any of the techniques discussed herein. Determining the second alignment may include determining a new alignment or determining a realignment. In some examples, determining the second alignment may include determining a window of sensor data associated with the modification. For example, this may include determining sensor data associated with a link associated with two poses that are unassociated with each other, two newly associated poses, or two poses with modified parameters. The window may include a span of time and / or distance for the pose. In some examples, newly aligning / realigning the sensor data associated with the window of sensor data may include weighting the sensor data based at least in part on the proximity of the sensor data to the time and / or location associated with the linked pose. For example, the alignment may down-weight (e.g., apply lower scalar values ​​and increase covariance or other stiffness parameters) the further the sensor data is from the linked pose in time and / or distance in an attempt to align the sensor data. Such weighting may be a function of such distance, including, but not limited to, Gaussian, linear, polynomial, sigmoid, etc. In some examples, modifying a weight associated with a window of sensor data may include modifying a sampling rate associated with the window of sensor data (e.g., decreasing the sampling rate away from the associated time and / or location).

[0071] In some examples, when the correction(s) are received, any updated alignments are generated, and instructions to generate the map are received, the remaining sensor data alignment may be determined based at least in part on the respective links associated therewith (e.g., in some examples, the sensor data is associated with uncorrected links and / or corrected links that were not selected to create an updated alignment or were not configured to cause an automatic alignment). Determining the alignment may include pose graph relaxation, ICP, ankous search, and / or the like.

[0072] In some examples, determining the map may include determining a representation of the environment based at least in part on the sensor data alignment. For example, determining the map may include determining a polyline model by a marching squares algorithm, a dual contouring algorithm, etc., and / or determining a mesh model and / or a wireframe model based at least in part on a marching cubes algorithm, a dual contouring algorithm, ray tracing, etc.

[0073] At operation 418, the example process 400 may include transmitting the second map to the autonomous vehicle in accordance with any of the techniques discussed herein. In some examples, the autonomous vehicle may use the map to perform sensor calibration and to position the autonomous vehicle and / or sensors of the autonomous vehicle.

[0074] At operation 420, the example process 400 may include adding the correction and / or associated data to the training data set according to any of the techniques discussed herein. For example, upon acceptance of the correction (e.g., when an instruction to generate a map providing implicit acceptance is received or when explicit acceptance is received from a user), the correction and / or associated data may be added to the training data set. In some examples, the training data may include an indication of what type of correction was made, parameters associated with the link (e.g., the location of an end, a covariance associated therewith), sensory data associated with the link (e.g., a bridge, overpass, or other problematic structure was detected), at least a portion of the sensor data, and / or two or more poses, and / or characteristics of the environment (e.g., dimensions, locations, and / or number of objects in the environment, the number of lanes of the roadway, a curvature associated with the roadway, the proximity of a portion of the roadway to an intersection, the number of roadways intersecting the roadway).

[0075] At operation 422, the example process 400 may include training an ML model based at least in part on the training dataset according to any of the techniques discussed herein. The ML model may be trained to output a set of modified global parameters, regions and / or poses of the environment that may require modification, instructions to modify links, the likelihood that links will be created and / or deleted, determine the likelihood that the sensor data is likely to be associated with a degree of alignment that does not meet a threshold, and / or the like. In some examples, at runtime, the ML model may receive proposed alignment data associated with the region as input and may output a null output or modifications to the link(s) associated with the region. The proposed alignment data may include the sensor data, pose / trajectory data, and / or any links generated for those associated with the region. The ML model may iteratively receive input for each region and output any modifications to each region until proposed alignment data associated with the investigated environment is provided as input to the ML model.

[0076] The example process 400 may include more or fewer operations, and the operations depicted and discussed herein may be in any other order. The operations discussed herein may be accomplished as different processes and / or processed in parallel. In some examples, operations depicted as different may be merged into a single operation, and vice versa.

[0077] (Example clauses) A. A method comprising: receiving sensor data associated with an environment; determining a first map and a first trajectory associated with the vehicle based at least in part on the sensor data and the link, the first trajectory including a plurality of poses, the link including an indication that a first portion of the sensor data associated with a first pose of the first trajectory corresponds to a second portion of the sensor data associated with a second pose; receiving a correction including at least one of an instruction to add a proposed link or an instruction to remove a link; determining a second map and a second trajectory based at least in part on the sensor data and the correction; and transmitting the second map to an additional vehicle.

[0078] B. The method of claim A, wherein the sensor data includes first sensor data and second sensor data, the first sensor data being associated with a first sensor of a first system, and the second sensor data being associated with a second sensor data of a second system.

[0079] C. The method of any one of clauses A and B, wherein determining at least one of the second map or the second trajectory includes determining a window of sensor data associated with the correction, and determining the second map includes redetermining a portion of the first map associated with a portion of the environment corresponding to the window of sensor data.

[0080] D. The method of any one of clauses A-C, wherein determining the window of sensor data includes determining a subset of the sensor data that is within a threshold distance or time of a position or time associated with the modification.

[0081] E. The method of any one of clauses A to D, wherein determining the first and second portions is based at least in part on at least one of a covariance, a sampling rate, or a scalar associated with the link, and the modification further includes changing at least one of the covariance, the sampling rate, or the scalar.

[0082] F. The method of any one of clauses A-E, wherein the modification, the first portion, and the second portion are added to a training dataset based at least in part on receiving the modification, and the method further includes training a machine learning model based at least in part on the training dataset, wherein the machine learning model is trained to identify one or more modifications to add links or remove links.

[0083] G. A system having one or more processors and a memory storing processor-executable instructions, which when executed by the one or more processors, cause the system to perform operations including receiving sensor data associated with an environment; determining a first alignment based at least in part on the sensor data and a link, the link including an indication that a first portion of the sensor data associated with a first pose of the vehicle corresponds to a second portion of the sensor data associated with a second pose, the first alignment including an alignment of the first portion of the sensor data and the second portion of the sensor data; receiving a correction including at least one of instructions to add a proposed link, instructions to remove a link, or instructions to modify an uncertainty associated with the link; and determining a second alignment based at least in part on the sensor data and the correction.

[0084] H. The system of clause G, wherein the operations further include determining a map based at least in part on the second alignment and at least one of transmitting the map to the autonomous vehicle or causing the second alignment to be displayed.

[0085] I. The system of clause G or H, wherein the operations further include determining a degree of alignment associated with the first alignment or the unaligned sensor data, and generating a notification indicating an area associated with the degree of alignment, the notification indicating a suggested correction, and the correction is received based at least in part on the notification.

[0086] J. The system of any one of clauses G-I, wherein the sensor data includes first sensor data and second sensor data, the first sensor data associated with a first sensor of a first system, and the second sensor data associated with a second sensor data of a second system.

[0087] K. The system of any one of clauses G-J, wherein determining the second alignment includes determining a window of sensor data associated with the modification, and determining the second alignment includes redetermining a portion of the first alignment associated with a portion of the environment corresponding to the window of sensor data.

[0088] L. The system of any one of clauses G-K, wherein determining the window of sensor data includes determining a subset of the sensor data that is within a threshold distance or within a threshold time of a position or time associated with the modification.

[0089] M. The system of any one of clauses G-L, further comprising: based at least in part on receiving the modification, the modification, the first portion, and the second portion are added to a training dataset; and the operation further comprises training a machine learning model based at least in part on the training dataset, the machine learning model being trained to identify one or more modifications to add another link, remove a link, or change a parameter associated with the link.

[0090] N. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including receiving sensor data associated with an environment; determining a first alignment based at least in part on the sensor data and a link, the link including an indication that a first portion of the sensor data associated with a first pose of the vehicle corresponds to a second portion of the sensor data associated with a second pose, the first alignment including an alignment of the first portion of the sensor data and the second portion of the sensor data; receiving a modification including at least one of instructions to add a proposed link, instructions to remove a link, or modify a parameter associated with the link; and determining a second alignment based at least in part on the sensor data and the modification.

[0091] O. The one or more non-transitory computer-readable media of clause N, wherein the operations further include determining a map based at least in part on the second alignment and at least one of transmitting the map to the autonomous vehicle or causing the second alignment to be displayed.

[0092] P. The one or more non-transitory computer-readable media of clause N or O, wherein the operation further includes determining a degree of alignment associated with the first alignment or the unaligned sensor data, and generating a notification indicating an area associated with the degree of alignment, the notification indicating a suggested correction, the correction being received based at least in part on the notification.

[0093] Q. One or more non-transitory computer-readable media described in any one of clauses N-P, wherein the sensor data includes first sensor data and second sensor data, the first sensor data associated with a first sensor of a first system, and the second sensor data associated with a second sensor of a second system.

[0094] R. One or more non-transitory computer-readable media of any one of clauses N-Q, wherein determining the second alignment includes determining a window of sensor data associated with the modification, and determining the second alignment includes redetermining a portion of the first alignment associated with a portion of the environment corresponding to the window of sensor data.

[0095] S. One or more non-transitory computer-readable media of any one of clauses N-R, wherein determining a window of sensor data includes determining a subset of sensor data that is within a threshold distance or within a threshold time of a location or time associated with a modification.

[0096] T. The one or more non-transitory computer-readable media of any one of clauses N-S, further comprising: based at least in part on receiving the modification, the modification, the first portion, and the second portion are added to a training dataset; and the operation further includes training a machine learning model based at least in part on the training dataset, the machine learning model being trained to identify one or more modifications to add links or remove links.

[0097] U. The method of any one of clauses A-F, wherein the perturbing includes at least one of longitudinal, lateral, vertical, yaw, roll, or pitch movement of the estimated position and orientation associated with the first sensor relative to the second sensor.

[0098] V. The method of any one of clauses A-F or U, further including receiving additional sensor data from the autonomous vehicle and determining at least one of a position or orientation of the additional autonomous vehicle based at least in part on comparing the additional sensor data to a second map.

[0099] W. The method of any one of clauses A-F, U or V, further including controlling the autonomous vehicle based at least in part on at least one of the position or orientation.

[0100] X. The method of any one of clauses A-F, U-W, wherein determining the map includes projecting the first sensor data and the second sensor data into voxel space based at least in part on the set of links, the first pose data and the second pose data, and determining a mesh representation of the voxel space.

[0101] While the example clauses described above are described with respect to one particular implementation, it should be understood that in the context of this specification, the contents of the example clauses can also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Furthermore, any of the examples A-X can be implemented alone or in combination with one or more of the other examples A-X.

[0102] (Conclusion) Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0103] The components described herein represent instructions that may be stored on any type of computer-readable medium and implemented in software and / or hardware. All of the methods and processes described above may be embodied and fully automated via software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Alternatively, some or all of the methods may be embodied in specialized computer hardware.

[0104] At least some of the processes described herein are illustrated as logical flow graphs, with each operation representing a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, cause a computer or autonomous vehicle to perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0105] Unless expressly stated to the contrary, conditional terms such as "can," "may," "can," or "may" are understood within the context to, among other things, state that an example includes certain features, elements, and / or steps that other examples do not include. Thus, such conditional terms are not intended to imply that a certain feature, element, and / or step is required in any way in one or more examples in general, or that the one or more examples necessarily include logic for determining whether a certain feature, element, and / or step is included in or should be performed in any particular example, with or without user input or prompting.

[0106] Conjunctions such as the phrase "at least one of X, Y, or Z," unless specifically stated otherwise, should be understood to mean that the item, term, etc. can be either X, Y or Z, or any combination thereof, including collections of each of the elements. Unless expressly described as singular, "a" means singular as well as plural.

[0107] Any routine illustrations, elements, or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code that include one or more computer-executable instructions for implementing a particular logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, in which elements or functions may be omitted or performed out of order from that shown or described, including substantially simultaneously with additional operations, or in reverse order, depending on the functionality involved, as will be understood by those skilled in the art.

[0108] Numerous variations and modifications can be made to the above-described examples, and those elements should be understood to be within the scope of other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

1. one or more processors; a memory for storing processor-executable instructions; the processor-executable instructions, when executed by the one or more processors, provide the system with: receiving first sensor data associated with a first pose of a first vehicle and second sensor data associated with a second pose of a second vehicle; determining a first alignment based at least in part on the first sensor data, the second sensor data, and a first link; the first link includes an indication that the first sensor data is associated with the same or a similar portion of an environment as the second sensor data; the first alignment includes an alignment of the first sensor data with the second sensor data; And, receiving a modification including at least one of instructions to add a proposed second link, instructions to remove the first link, or instructions to change a parameter associated with the first link; determining a second alignment of the first sensor data and the second sensor data based at least in part on the first sensor data, the second sensor data, and the correction; determining map data based at least in part on the second alignment; and A system that causes a system to perform an operation including:

2. 2. The system of claim 1, wherein the operations further include at least one of transmitting the map data to a third vehicle or causing the second alignment to be displayed, and the first vehicle is the same vehicle as one or more of the second vehicle and the third vehicle.

3. The operation includes: determining a degree of alignment associated with the first alignment; generating a notification indicating a region associated with the degree of alignment, the notification indicating a revision suggestion, the revision suggestion received based at least in part on the notification; The system of claim 1 , further comprising:

4. The system of claim 1, wherein the first sensor data is associated with a first sensor of a first sensor system, and the second sensor data is associated with a second sensor of a second sensor system.

5. 2. The system of claim 1 , wherein determining the second alignment includes determining a window of sensor data associated with the modification, and wherein determining the second alignment includes redetermining a portion of the first alignment associated with a portion of the environment corresponding to the window of sensor data, the window of sensor data including at least one portion of the first sensor data and the second sensor data.

6. 6. The system of claim 5, wherein determining the window of sensor data includes determining a subset of the first sensor data and the second sensor data that are within a threshold distance from a location associated with the modification or within a threshold time of a time associated with the modification.

7. based at least in part on receiving the corrections, the corrections, the first sensor data, and the second sensor data are added to a training data set; The operations further include training a machine learning model based at least in part on the training dataset, the machine learning model being trained to identify one or more modifications to add the second link, remove the first link, or change a parameter associated with the first link. The system of claim 1.

8. The operation comprises: determining at least one of a first trajectory and a first map based at least in part on the first alignment; modifying at least one of the first trajectory and the first map based at least in part on the second alignment as at least one of a second trajectory and a second map; The system of claim 1 , further comprising:

9. The system of claim 8, wherein the first sensor data is associated with a first sensor of a first system and the second sensor data is associated with a second sensor of a second system.

10. The operation comprises: determining first map data based at least in part on the first alignment; the map data is second map data, The operation includes: determining a window of sensor data associated with the modification; determining the second map data including redetermining a portion of the first map data associated with a portion of the environment corresponding to the window of sensor data; The system of claim 8 , further comprising:

11. The operation comprises: determining at least one of a first trajectory and a first map based at least in part on the first alignment; modifying at least one of the first trajectory and the first map based at least in part on the second alignment as at least one of a second trajectory and a second map; The system of claim 8 , further comprising:

12. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to: receiving first sensor data associated with a first pose of a first vehicle and second sensor data associated with a second pose of a second vehicle; determining a first alignment based at least in part on the first sensor data, the second sensor data, and a first link; the first link includes an indication that the first sensor data is associated with the second sensor data; the first alignment includes an alignment of the first sensor data with a second sensor data; And, receiving a modification including at least one of instructions to add a proposed second link, instructions to remove the first link, or instructions to change a parameter associated with the first link; determining a second alignment based at least in part on the first sensor data, the second sensor data, and the correction; determining map data based at least in part on the second alignment; and One or more non-transitory computer-readable media for causing operations to be performed, including:

13. The operation includes: and at least one of transmitting the map data to a third vehicle or displaying the second alignment; 13. The one or more non-transitory computer-readable media of claim 12, wherein the first vehicle is the same vehicle as one or more of the second vehicle and the third vehicle.

14. The operation includes: determining a degree of alignment associated with the first alignment; generating a notification indicating a region associated with the degree of alignment, the notification indicating a revision suggestion, the revision suggestion received based at least in part on the notification; The one or more non-transitory computer-readable media of claim 13, further comprising:

15. One or more non-transitory computer-readable media of claim 14, wherein the first sensor data is associated with a first sensor of a first system and the second sensor data is associated with a second sensor of a second system.

16. 13. The one or more non-transitory computer-readable media of claim 12, wherein determining the second alignment includes determining a window of sensor data associated with the modification, and wherein determining the second alignment includes redetermining a portion of the first alignment associated with a portion of the environment corresponding to the window of sensor data, the window of sensor data including at least one portion of the first sensor data and the second sensor data.

17. 17. The one or more non-transitory computer-readable media of claim 16, wherein determining the window of sensor data includes determining a subset of the first sensor data and the second sensor data that are within a threshold distance from a location associated with the modification or within a threshold time of a time associated with the modification.

18. based at least in part on receiving the corrections, the corrections, the first sensor data, and the second sensor data are added to a training data set; 13. The one or more non-transitory computer-readable media of claim 12, wherein the operations further include training a machine learning model based at least in part on the training dataset, the machine learning model being trained to identify one or more modifications to add the second link or remove the first link.

19. The operation comprises: determining at least one of a first trajectory and a first map based at least in part on the first alignment; modifying at least one of the first trajectory and the first map based at least in part on the second alignment as at least one of a second trajectory and a second map; The one or more non-transitory computer-readable media of claim 12, further comprising:

Images