Map data updates

Abstract

Techniques for updating map data are disclosed. The techniques may include detecting a traffic signal in a first image and determining a proposed three-dimensional position of the traffic signal in a three-dimensional coordinate system associated with the map data based at least in part on the traffic signal detected in the first image. The proposed three-dimensional position may then be projected onto a second image to determine a two-dimensional position of the traffic signal in the second image, and the second image may be annotated with the proposed traffic signal position indicator associated with the traffic signal as an annotated image. The techniques further include displaying the annotated image to a user on a display, receiving user input associated with the annotated image, and updating the map data to include the location of the traffic signal within the map data as updated map data based at least in part on the user input.

Description

[Technical field]

[0001] The present invention relates to updating map data.

[0002] [CROSS REFERENCE TO RELATED APPLICATIONS] This PCT international application claims the benefit of priority to U.S. Patent Application No. 16 / 698,366, filed November 27, 2019, the disclosure of which is incorporated herein by reference. [Background technology]

[0003] Data can be captured within an environment and represented as a map of the environment. Often such a map can be used by a vehicle navigating within the environment, but the map can be used for a variety of purposes. In some cases, the environment can change such that the map no longer accurately reflects the environment, which can make the map data unusable. It can be difficult or expensive to update the map data to reflect changes in the environment. [Brief description of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears, and in which the same reference number in different drawings indicates similar or identical items.

[0005] [Figure 1] FIG. 1 illustrates an example autonomous vehicle system that detects objects in an environment and updates map data based thereon. [Figure 2A] FIG. 2A shows a sequence of images of an intersection captured at different times, and an example of object detection and tracking within the sequence by the perception component. [Figure 2B] FIG. 2B shows a sequence of images of an intersection captured at different times, and an example of object detection and tracking within the sequence of images by the perception component. [Figure 2C]FIG. 2C shows a sequence of images of an intersection captured at different times, and an example of object detection and tracking within the sequence by the perception component. [Figure 3A] FIG. 3A shows the example images of FIGS. 2A-2C with a proposed object position indicator for the object detected and tracked by the perception component. [Figure 3B] FIG. 3B shows the example images of FIGS. 2A-2C with proposed object position indicators for objects detected and tracked by the perception component. [Figure 3C] FIG. 3C shows the example images of FIGS. 2A-2C with proposed object position indicators for objects detected and tracked by the perception component. [Figure 4] FIG. 4 illustrates a flow diagram of an exemplary process for detecting, tracking, and flagging objects in an environment for processing by the map update system. [Diagram 5] FIG. 5 illustrates a flow diagram of an exemplary process for updating map data based on detecting and tracking objects in an environment.

[0006] [Detailed Description] The present disclosure is directed to systems and techniques for updating map data used, for example, by an autonomous vehicle navigating within an environment. For example, the systems and techniques according to the present disclosure may enable updating the map data of an environment based on image data acquired by the vehicle utilizing current map data.

[0007] Image data acquired by a vehicle utilizing current map data may be flagged, for example, by the vehicle's on-board computer if (1) an object detected in the image data does not correspond to an object in the map data, (2) an object in the map data is not detected in the image data, or (3) an object in the image data appears to correspond to an object in the map data but has a different location (e.g., a traffic signal that has been moved or relocated on a traffic signal assembly). The flagged image data may be provided to a map update system, which may be part of the central control system, which may operate to analyze the image data and generate proposed changes to the map data that may be provided to a system operator (e.g., a user for review).

[0008] In some examples, the perception component of the map update system may detect an object in the image data. The detected object may be tracked across multiple image frames (e.g., multiple frames of a video or multiple still image frames). Based on the tracking, a proposed location (also referred to herein as a candidate location or initial location) for the object in a coordinate system of the map data may be generated. The perception system may also classify the detected object (e.g., as a road sign, a traffic light, etc.). The map component of the map update system may be configured to determine whether the map data includes an object of the determined classification at the proposed location.

[0009] In cases where a detected object does not appear in the map data or corresponds to an object in the map data that has moved (e.g., a new object of a classification is detected and a previously mapped object of the same classification is missing), a projection component of the map update system may project the proposed location indicator into an image frame that includes a view of the detected object. In some examples, the projection component may perform projecting the proposed location indicator and corresponding identifier onto multiple images at a location in the two-dimensional image space of each image that corresponds to the three-dimensional coordinates of the proposed location of the detected object.

[0010] Similarly, in cases where the map data includes objects that do not appear in the image data, the projection component of the map update system may project a position indicator onto the image frame to indicate the previous location of the missing object.

[0011] The image frame including the location indicator may be displayed to the reviewing user. The reviewing user may provide feedback regarding the location indicator. For example, the reviewing user may approve or reject the proposed location. The map component may then add objects to the map data for approved proposed locations. In addition, the reviewing user may request additional proposed locations (e.g., based on additional image data) for objects whose proposed locations were rejected. In some embodiments, a central control system, of which the map update system may be a part, may command one or more autonomous vehicles to navigate to the location of the object of interest and capture additional image data. In the case of a previous location indicator, the reviewing user may provide feedback confirming that the object in the map data is no longer in the previous location.

[0012] The map update techniques described herein can improve the functionality of a computing device, such as an autonomous vehicle system, by providing updated map data that reflects changes in the environment for performing subsequent actions to control an autonomous vehicle with reduced difficulty and cost. For example, the updated map data can enable subsequent processes, such as localization, perception (e.g., detection, identification, segmentation, classification, tracking, etc.), route planning, trajectory generation, etc., to be performed more accurately while requiring less manual intervention and / or interruption. For example, in some examples, faster and / or more accurate map updates can be used in generating a trajectory for an autonomous vehicle more quickly, reducing disruptions and shorter windows in which manual overrides may be required to address changes in the environment, and improving the safety of occupants of the autonomous vehicle (e.g., by reducing the window in which the map data is inaccurate or an autonomous vehicle may perform inappropriate actions based on inaccurately applying old map data to a changed environment). These and other improvements to the functionality of a computer are described herein.

[0013] [Example Architecture] FIG. 1 illustrates an example autonomous vehicle system 100 that detects objects in an environment and updates map data based thereon. In some examples, the autonomous vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout the entire journey without expecting a driver (or passenger) to be in constant vehicle control. However, in other examples, the autonomous vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some examples, the guidance isolation techniques described herein may be usable by non-autonomous vehicles as well. It is contemplated that the techniques described herein may be applied to more than robotic control for autonomous vehicles and the like. For example, the techniques described herein may be applied to video games, manufacturing, augmented reality, and the like.

[0014] In accordance with the techniques described herein, the autonomous vehicle 102 may receive sensor data from sensors 104 of the autonomous vehicle 102. For example, the sensor data may include position signals (e.g., GPS signals), inertial signals (e.g., accelerometer signals, gyroscope signals, etc.), magnetometer signals, wheel encoder signals, speedometer signals, point clouds of accumulated LIDAR and / or RADAR points, an image (or multiple images), audio signals, and / or bariatric or other environmental signals, etc. For example, the autonomous vehicle 102 may receive images 106(1) through 106(M) (collectively, “images 106”) from the sensors 104, where M is any integer greater than 1. While the description herein primarily discusses images for simplicity, it is contemplated that the techniques may be applied to any sensor data capable of discretely representing an object (e.g., a point cloud including points representing an object).

[0015] In some examples, the autonomous vehicle may include a computing device 108 that may include a position estimation component 110, a perception component 112, a planning component 114, a map component 116, and / or a control system interface 118.

[0016] In at least one example, the position estimation component 110 can include functionality for receiving data (also referred to herein as position estimation data) from the sensor system 104 to determine a position of the vehicle 102. For example, the position estimation component 110 can include and / or request / receive a map of the environment (e.g., from the map component 116) and can continuously determine a position of the autonomous vehicle within the map (e.g., estimating a position of the autonomous vehicle 102 on a global map and / or a local map). In some examples, the position estimation component 110 can receive image data, LIDAR data, radar data, IMU data, GPS data, wheel encoder data, etc., utilizing SLAM (Simultaneous Localization and Mapping) or CLAMS (Concurrent Calibration, Localization and Mapping) to accurately determine a position of the autonomous vehicle. In some examples, the position estimation component 110 can provide position estimation data to various components of the vehicle 102 to determine an initial position of the autonomous vehicle for generating candidate trajectories, as described herein.

[0017] The perception component 112 may include one or more ML models and / or other computer-executable instructions for detecting, identifying, segmenting, classifying, and / or tracking objects from sensor data collected from the environment of the autonomous vehicle 102. In some examples, the perception component 112 may receive sensor data from the sensors 104 of the autonomous vehicle 102 and localization data from the localization component, determine perception data from the sensor data, and transmit the perception data to the planning component 114 for use by the planning component 114 to determine one or more trajectories and / or control the movement of the autonomous vehicle 102 to traverse a path or route, although any such operations may be performed in various other components. In some instances, the perception data may include a region of interest ("ROI") and / or tracking associated with an object detected in an image. The planning component 114 may determine instructions for controlling the operation of the autonomous vehicle 102 based at least in part on the ROI and / or tracking.

[0018] For example, perception component 112 may detect objects in an environment and classify the objects (e.g., “traffic light,” “four-wheel vehicle,” “semi-truck,” “pedestrian,” “animal,” “construction vehicle”). In the illustrated example, autonomous vehicle 102 may receive image 106, including image 106(1) that includes a representation of two traffic lights. Perception component 112 may generate ROIs 122(1) and 122(2), each corresponding to a respective traffic light represented in image 106(1). In the illustrated example, ROI 122 is represented as a bounding box, although other techniques for identifying ROIs are contemplated.

[0019] In some examples, the perception component 112 may also track an object, such as a traffic light, that corresponds to the ROI 122. In some examples, tracking may include an association between objects detected in two different images. Thus, tracking may indicate that the perception component 112 is identifying the detection of two objects as corresponding to the same object. In some examples, tracking may additionally or alternatively include a current, predicted, and / or past position, heading, speed, acceleration, distance, ROI, and / or center of the object. In some examples, any of these characteristics may be real-world values ​​(e.g., the speed of the object in the real world in meters per second, kilometers per hour, heading relative to the sensor that captured the image) and / or image-related characteristics (e.g., speed associated with the movement of a representation of the object across the image in pixels per second, heading identifying the angle of movement of the representation of the object in the image). Of course, the tracking information of the object may include any data structure and / or format.

[0020] In some examples, the location estimation component 110 may determine a location of the autonomous vehicle 102 (e.g., using any sensor data to localize the autonomous vehicle 102). The perception component 112 may determine data related to objects in the vicinity of the autonomous vehicle 102, route data specifying the vehicle's destination, global map data identifying road characteristics (e.g., features detectable in different sensor modalities useful for localizing the autonomous vehicle), local map data identifying features detected in proximity to the vehicle (e.g., locations and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable in various sensor modalities), and the like. The data generated by the perception component 112 may be collectively referred to as "perception data." In some examples, the map data may be retrieved and provided by the map component 116. More specifically, based on the location of the autonomous vehicle 102, the map component 116 may load or otherwise retrieve from memory or a network local and global map data for use by the perception component 112 in detecting and classifying objects from images. Once the perception component 112 has identified and / or segmented objects from the image and / or determined other perception data, the perception may provide the perception data, including object detection and / or instance segmentation, to the planning component 114.

[0021] In some examples, the planning component 114 may use the sensory data, including the ROI, tracking, and / or segmentation (described further below), to generate instructions for controlling the operation of the autonomous vehicle 102. For example, the planning component 114 may determine a route for the autonomous vehicle 102 from a first location to a second location, generate multiple potential trajectories for controlling the movement of the autonomous vehicle 102 substantially simultaneously based at least in part on the ROI, tracking, and / or segmentation according to a receding horizon technique (e.g., 1 microsecond, ½ second, 2 seconds, 8 seconds, etc.), to control the vehicle to traverse the route, and select one of the potential trajectories as a trajectory for the autonomous vehicle 102 that may be used to generate drive control signals that may be transmitted to drive components of the autonomous vehicle 102.

[0022] Additionally, perception component 112 may determine that (1) an object detected in the image data does not correspond to an object in the map data, (2) an object in the map data is not detected in the image data, or (3) an object in the image data appears to correspond to an object in the map data but has a different location (e.g., a traffic light that has been moved or repositioned on a traffic light assembly). If so, perception component 112 may flag the detected object in the multiple image frames and cause control system interface 118 to transmit images of the flagged object over one or more networks 120 to map update system 130.

[0023] As mentioned above and as described throughout this disclosure, the vehicle 102 can transmit image data to one or more computing devices 124 via one or more networks 120. In some examples, the vehicle 102 can transmit raw sensor data (e.g., images or other raw sensor data without metadata or identification of detected objects) to the computing device 124. In other examples, the computing device 124 can transmit processed sensor data and / or representations of the sensor data (e.g., processed images or other processed sensor data with metadata or other additional data identifying detected objects whose presence caused the image to be flagged) to the computing device 124. In some cases, the vehicle 102 can transmit the sensor data (raw or processed) to the computing device 124 as one or more log files. For ease of understanding, in the following description, the sensor data may be referred to as image data. However, the embodiments are not so limited and other sensor data, such as those described above, may be utilized in addition to or instead of image data.

[0024] In at least some examples, the vehicle 102 may detect such traffic objects and / or characteristically determine a discrepancy between such detected information and information contained in the map data. Detection of such a discrepancy may be used as a trigger to transmit such data to the computing device 124 for confirmation and / or validation. In at least some such examples, the vehicle 102 may execute one or more maneuvers based on the discrepancy (such as leading to a safety stop) and / or based on weighting the contribution of such detections up and / or down to continue to plan a trajectory through the environment.

[0025] The computing device 124 may receive image data and may generate and / or update map data based on the image data. In at least one example, the computing device 124 may include one or more processors 126 and a memory 128 communicatively coupled to the one or more processors 126. In the illustrated example, the memory 128 of the computing device 124 stores a map update system 130 that may include or utilize the functionality of a perception and localization component 132 (hereinafter, perception component 132), a map component 134, a projection component 136, and a user interface component 138. In at least one example, the computing device 124 may be, or be part of, an autonomous vehicle control system that manages the operation of multiple autonomous vehicles 102.

[0026] As mentioned above, the map updating system 130 may receive the image data via the network 120. The map updating system 130 may be configured to update the map based on the received image data.

[0027] In some examples, the perception component 132 of the map update system 130 may perform perception processing on the image data to detect objects in the image data. The detected objects may be tracked across multiple image frames (e.g., multiple frames of video or multiple still image frames). The information may further include localization information (e.g., the position of the image capture sensor) in the coordinate system of the map data, or the map update system 130 may otherwise determine the localization information. More specifically, the localization information may localize the vehicle (or image capture sensor) in the environment and may be generated using various techniques (e.g., SLAM) and potentially using many types of sensor data (lidar, gps, image, IMU). The localization information may also include the pose (position, orientation, etc.) of the camera.

[0028] Based on the tracking and position estimation information, a proposed position of the object in the coordinate system of the map data may be generated. For example, based on the vehicle position and camera pose, a UTM (Universal Transverse Mercator) position (or other position such as latitude and longitude) and height associated with each pixel in the image data may be determined. Over a series of frames, the perception component 132 may triangulate the position of the tracked object in the environment and determine the UTM and / or height information for each frame with sufficient certainty. In other words, in some embodiments, the perception component 132 may project a ray from the camera origin to each pixel in the image and triangulate the position over time. In some embodiments, this process may be performed without depth data. The UTM and / or height information associated with the center of the tracked object determined over a series of frames may be utilized as the proposed position.

[0029] Perception component 132 may also classify the detected objects (e.g., as a road sign, a traffic signal, etc.). In some examples, perception component 132 may perform similar functions and / or in similar manners as described above with respect to perception component 112.

[0030] The map component 134 of the map update system 130 may retrieve map data provided about the environment, area, or geographic location where the image data was captured. In one example, the environment, area, or geographic location where the image data was captured may be provided as metadata in the received image data or may otherwise be provided by the system capturing the image data.

[0031] Based on the location and / or classification of the object detected by the perception component 132 and the map data provided for the environment, area, or geographic location in which the image data was captured, the perception component 132 of the map update system 130 may determine whether the map data includes an object of the determined classification at the proposed location.

[0032] In cases where the detected object does not appear in the map data or corresponds to an object in the map data that has moved, the projection component 136 of the map update system may project the proposed location indicator into an image frame containing a view of the detected object. In some examples, the projection component 136 may perform the projection by overlaying the proposed location indicator and a corresponding identifier into the multiple images at a location in the space of the two-dimensional image of each image that corresponds to the three-dimensional coordinates of the proposed location of the detected object. For example, UTM and / or elevation information associated with the center of the tracked object is utilized as the proposed location and projected from the 3D coordinates of the UTM and elevation information to a 2D location in image space.

[0033] In this or any other example described herein, the projection component 136 may project the indicator into any image or video captured by any vehicle (as long as the vehicle is correctly localized in the environment). In other words, the image onto which the proposed position indicator is projected may be the same image used for detection and location, a completely different image, or some mixture thereof. Furthermore, the image onto which the proposed position indicator is projected may be captured at a different time than the image used for detection and location.

[0034] Similarly, in cases where the map data includes objects that do not appear in the image data, the projection component 136 of the map update system 130 may project a previous location indicator onto the image frame to indicate the previous location of the missing object.

[0035] The user interface component 138 may display an image frame including the location indicator to the reviewing user. The reviewing user may provide feedback regarding the location indicator to the user interface component 138. For example, the reviewing user may provide feedback to accept or reject the proposed location. In the case of a previous location indicator, the reviewing user may provide feedback to confirm or reject that the object in the map data is no longer at the previous location.

[0036] Based on the feedback, the map component 134 may add the object to the map data for the proposed location that was approved. In cases where the user confirms that the object is no longer present at the previous location, the map component 134 may be configured to remove the object from the map data. If the proposed location is rejected, the reviewing user may request an additional proposed location for the object whose proposed location was rejected (e.g., based on additional image data). In some embodiments, a control system, of which the map update system 130 may be a part, may command one or more autonomous vehicles to navigate to the location of the object of interest to capture additional image data.

[0037] Although the exemplary system of FIG. 1 operates based on image data captured by an autonomous vehicle, the embodiments are not so limited and the source of image data may vary. Other variations or features may be incorporated into some embodiments. For example, some embodiments may incorporate a filtering function configured to determine differences between captured data and existing map data regarding new, missing, or modified objects. The filtering function may determine if the differences are above a threshold and discard the projection and review process, or otherwise not execute if the differences are below the threshold. This may reduce the number of false positives caused by issues such as miscalibrated sensors or similar factors.

[0038] Additionally, although examples described herein relate to projecting a position indicator onto image data, embodiments are not so limited. For example, in addition or instead, the projection component 136 may project other information onto the image. Some other non-limiting types of information that may be projected onto the image may include range, dimensions and orientation of detected objects, status, etc. For example, directional information may include the yaw, pitch, and roll of a traffic light. In a particular example, the yaw of a traffic light may be projected onto the image to assist a reviewing user in determining an approach road or lane associated with the traffic light.

[0039] 2A-2C illustrate a series of images 200, 220, and 240 of an intersection captured at different times and exemplary detection and tracking of the series of images by perception component 132. More specifically, FIG 2A illustrates an image 200 (also referred to as an image frame or frame) of the intersection captured at a first time, FIG 2B illustrates an image 220 of the intersection captured at a second time, and FIG 2C illustrates an image 240 of the intersection captured at a third time.

[0040] For ease of illustration and explanation, the detection and tracking of objects in the figures are limited to traffic signals. The embodiments are not limited to traffic signals, and any other objects included in the map data may be detected and tracked in a similar process. For example, other objects that may be included in the map data may include other signs and road features. More specifically, such objects may include signs and signals such as crosswalk signals (walk and / or hand indicators), speed limit signs, stop signs, yield signs, street name boards, one-way road signs, turn policy signs (e.g., no red light, no U-turn, no entering and going wrong way, no left turn, etc.). Road features that may be included in the map data may include painted markings such as crosswalks, stop lines, lane markings, etc. Similarly, such detection may more generally include characteristics of objects in the environment used for traffic control. As a non-limiting example, such characteristics may include, but are not limited to, the size, range, direction (e.g., yaw angle), or state (e.g., red, yellow, green, etc.) of a traffic signal, etc.

[0041] 2A-2C, more or less image frames may be utilized, and the following description assumes that additional frames, not shown, are processed between frames 200 and 220, and between 220 and 240 to avoid undue confusion, however, embodiments may utilize any number of frames.

[0042] 2A, perception component 132 may perform detection and tracking processing on image 200. Perception component 132 may detect three objects corresponding to traffic signals within corresponding regions of interest (ROIs): tracked object 1 202, tracked object 2 204, and tracked object 3 206. Each detected object may be associated with a classification and / or object tracking as described herein.

[0043] In FIG. 2B, the perception component 132 may perform detection and tracking processing on the image 220. Similar to FIG. 2A, the perception component 132 may detect and track the left-most and right-most traffic lights as tracked object 1 202 and tracked object 3 206. However, due to any of a variety of reasons, the perception component 132 may detect the middle traffic light but not assign it to the tracking of tracked object 2 204. For example, the middle traffic light may not have been detected in one or more image frames between image frames 200 and 220, resulting in a loss of tracking, or the location detected in FIG. 2B may be sufficiently different from the location detected in FIG. 2A that the perception component 132 begins a new tracking. More specifically, the middle traffic light may be detected as a new object and assigned a new tracking as tracked object 4 208.

[0044] In Figure 2C, perception component 132 may perform detection and tracking processing on image 240. In particular, perception component 132 may detect and maintain tracking of each traffic light from Figure 2B as tracked object 1 202, tracked object 4 208, and tracked object 3 206, respectively.

[0045] Based on the detection and tracking and the position of the camera when each of the images 200, 220, and 240 was captured, the perception component may estimate the position of each tracked object 202-208 (e.g., based on the position of the camera when each of the images 200, 220, and 240 was captured and the positions of the tracked objects 202-208 relative to the camera). The position of the camera in the coordinate system of the map data when the camera captured each image may be included with the image as metadata or the map update system 130 may determine the position information. The perception component 132 may also classify the detected objects (e.g., as a road sign, a traffic light, etc.). Further description of techniques and systems for determining object position and classification is provided in U.S. Patent Application Serial No. 15 / 814,870, entitled "Pose Determination From Contact Points," the entirety of which is incorporated herein by reference.

[0046] The map component 134 of the map update system 130 may retrieve map data provided for the environment, area, or geographic location where the image data was captured. In one example, the environment, area, or geographic location where the image data was captured may be provided as metadata to the received image data or otherwise provided by the system that captures the image data.

[0047] Based on the location and / or classification of the tracked objects 202-208 detected by the perception component 132 and the map data provided by the map component 134, the perception component 132 may determine whether the map data includes a corresponding object of the determined classification at the proposed location.

[0048] In cases where the detected object does not appear in the map data, the projection component 136 of the map update system may project the proposed location indicator into an image frame containing a view of the tracked object. As described above, the projection component 136 may perform the projection by overlaying the proposed location indicator and a corresponding identifier into the multiple images at a location in the space of the two-dimensional image of each image that corresponds to the three-dimensional coordinates of the proposed location of the detected object. More generally, such detected traffic characteristics may be associated with a graphical element to be presented on the display. For example, as non-limiting examples, a traffic signal's yaw indication may be presented as an arrow indicator, a range may be presented as a bounding box, a traffic signal's state may be associated with a color (e.g., red, green, yellow), etc. Further description of techniques and systems for projecting indicators or other information to positions in image space corresponding to positions in three-dimensional space is provided in U.S. Patent Application Serial No. 15 / 814,870, entitled "Pose Determination From Contact Points," which is incorporated herein by reference in its entirety.

[0049] 3A-3C illustrate a series of image frames from FIGS. 2A-2C along with example proposed location indicators (with corresponding identifiers) for tracked objects detected and tracked by perception component 132. Such image frames may be displayed to a reviewing user by user interface component 138, who may provide feedback based thereon (e.g., accept, reject, and / or modify the indicated proposed location). Additionally, although three image frames are shown in the example illustrated in FIGS. 3A-3C, more or fewer image frames may be utilized, and the following description assumes that additional frames may be processed and displayed to the user between frames 300 and 320, and frames 320 and 340, which are not illustrated to avoid unnecessary clutter. However, embodiments may utilize any number of frames greater than one.

[0050] More specifically, in the view of image 300, proposed object position 1 302 corresponding to tracked object 1 202 is shown as having an identifier "6" and a proposed location indicated by an "x" on the center of the leftmost traffic light. Proposed object position 2 304 corresponding to tracked object 2 204 is shown as having an identifier "4" and a proposed location indicated by an "x" on the center of the middle traffic light. Proposed object position 3 306 corresponding to tracked object 3 206 is shown as having an identifier of "5" and a proposed location indicated by an "x" on the center of the rightmost traffic light. Proposed object position 4 308 corresponding to tracked object 4 208 is shown as having an identifier "10" and a proposed location indicated by a second "x" on the center of the middle traffic light.

[0051] 3B and 3C show images 320 and 340 corresponding to images 220 and 240. Each of images 320 and 340 also includes proposed object positions 302-308 as projected into the view of the respective image. As shown, proposed object positions 1 302, 3 306, and 4 308 maintain their position relative to the traffic lights in the image views of images 300-340. Because the indicators corresponding to proposed object positions 302, 306, and 308 maintain their position relative to the corresponding traffic lights, a reviewing user may determine that the proposed positions are correct. Meanwhile, the indicator corresponding to proposed object position 2 304 drifts away from the center traffic light from image 300 to image 320, and from image 320 to image 340. This indicates that the proposed position for proposed object position 2 304 is inaccurate, and a reviewing user may determine so. In fact, the proposed location of proposed object location 2 304 is behind the center traffic light in the coordinate space of the map data (e.g., a location between the traffic light and the overpass behind the traffic light), resulting in drifting of the proposed location indicator relative to the center traffic light. Thus, a reviewing user may provide feedback rejecting proposed object location 2 304.

[0052] Although not shown, in cases where the map data includes objects that do not appear in the image data, the projection component 136 of the map update system 130 may project a previous location indicator onto the image frame to indicate the previous location of the missing object. In reviewing the image frame, a user may utilize the previous location indicator to determine that the object is still located at the indicated location (e.g., if the indicator maintains its position for an undetected object in the image frame, detection may have failed and the object may still be present).

[0053] As described above, the user interface component 138 may display image frames 300, 320, and 340 containing the proposed object locations 302-308 to the reviewing user. The reviewing user may provide feedback to the user interface component 138 as described above with respect to the location indicators. For example, the reviewing user may provide feedback to accept or reject the proposed location. In the case of a previous location indicator, the reviewing user may provide feedback to confirm or reject that the object in the map data is no longer at the previous location.

[0054] Based on the feedback, the map component 134 may add objects to the map data for proposed object locations that were approved. If the user confirms that the object is no longer at its previous location, the map component 134 may be configured to remove the object from the map data. If a proposed object location is rejected, the reviewing user may request additional proposed locations for the object whose proposed location was rejected (e.g., based on additional image data).

[0055] [Example Process] 4 illustrates an example process 400 for detecting, tracking, and flagging objects in an environment that do not appear in the map data for processing by the map update system. In some examples, the process 400 may be accomplished by components of the system 100. In some examples, the example process 400 may be performed by an on-board computer system of the autonomous vehicle 102, or may be performed in whole or in part by aspects of the computing device 124. Although FIG. 4 is described in the particular context of traffic signals and traffic lights, some embodiments may perform similar operations with respect to any type of object in an environment represented in the map data.

[0056] At operation 402, sensors of the autonomous vehicle (or any vehicle or system that captures sensor data) may capture images during operation of the autonomous vehicle. At operation 404, a computing device onboard the autonomous vehicle may perform object detection and classification on the captured images. The detected and classified objects in the images may then be compared at 406 to objects in the map data.

[0057] In operation 408, the on-board computing device may determine whether any detected objects are classified as traffic signals and are not included or differently located within the map data. If so, the process may proceed to 410. If not, the process may proceed to operation 414 and end.

[0058] At operation 410, the on-board computing device may flag in metadata or other data a detected object that is classified as a traffic signal and is not included in the map data or is differently located. Then, at 412, the on-board computing device may transmit a plurality of images of the flagged object and metadata or other data regarding the detected and flagged object to a map update system.

[0059] It should be noted that while the exemplary process 400 is illustrated as sequential operations, it is understood that the operations described herein may be performed in different orders, simultaneously, and / or by one or more devices. Additionally, while the process 400 is described as occurring on an autonomous vehicle, the embodiments are not so limited. For example, in some embodiments, the detection and flagging of objects for map updates may be performed by a manually operated vehicle having an on-board sensor and computing device system that performs the operations described above. Additionally, in some embodiments, the sensors and computing devices that perform the operations described above may not be on-board the vehicle or associated with the vehicle in any manner. Additionally, in other embodiments, all images may be uploaded to the map update system without processing and flagging.

[0060] 5 illustrates an example process 500 for updating map data based on detection and tracking of objects (e.g., traffic lights) in an environment. In some examples, the example process 500 may be accomplished by components of system 100. In some examples, the process 500 may be performed by map update system 130 of an autonomous vehicle control system. Although FIG. 5 is described in the specific context of traffic lights and traffic signals, some embodiments may perform similar operations with respect to any type of object in an environment represented in map data.

[0061] In operation 502, the map update system may receive data associated with the autonomous vehicle including multiple images of objects classified as traffic signals not included in the map data or in different locations along with information regarding the detection. However, as noted above, embodiments are not limited to input received from an autonomous vehicle. In other embodiments, the input may be raw sensor data or another source of flagged or processed image data.

[0062] At 504, the map update system may perform object detection, tracking and classification. The detected objects in the image may then be compared to objects in the map data at 506. At 508, the map update system may determine whether any detected objects are classified as traffic signals and are not included or are differently located in the map data. If so, the process may proceed to 510. If not, the process may proceed to 518 and end.

[0063] At 510, the map update system may project an identifier and location indicator into the multiple images at a location in the space of the two-dimensional image that corresponds to the three-dimensional coordinates of a proposed location of a detected object classified as a traffic signal and not included or differently located within the map data. At 512, the map update system may output or display the multiple images including the projected identifier and location indicator in a user interface for review by a reviewing user.

[0064] At 514, the map update system may receive feedback regarding the proposed location via a user interface. For example, the received feedback may indicate acceptance or rejection of the proposed location. For example, if a location indicator for a detected object is "attached" or on the detected object across multiple frames and / or projected onto images taken from different distances or angles, a reviewing user may determine that the proposed location is correct and provide feedback approving the proposal.

[0065] At 516, in response to the feedback, the map update system may add or update one or more object records in the map data based on the proposed object location.

[0066] It should be noted that while the exemplary process 500 is shown as sequential operations, it is understood that the operations described herein may be performed in different orders, simultaneously, and / or by one or more devices. Additionally, while the process 500 is described as receiving image data from an autonomous vehicle, or as part of an autonomous vehicle control system, or as updating map data used by an autonomous vehicle, the embodiments are not so limited. For example, in some embodiments, the detection of objects and updating the map data based thereon may utilize images captured using a manually operated vehicle with an on-board sensor and computing device system that performs the operations described above. Additionally, in some embodiments, the sensors and computing devices used to capture the image data may not be on-board the vehicle or associated in any manner with the vehicle.

[0067] [Example of invention content] A. A method comprising: detecting a traffic signal in a first image; determining a proposed three-dimensional position of the traffic signal in a three-dimensional coordinate system associated with map data based at least in part on the traffic signal detected in the first image; projecting the proposed three-dimensional position onto a second image to determine a two-dimensional position of the traffic signal in the second image; annotating the second image with a proposed traffic signal position indicator associated with the traffic signal as an annotated image; causing a display to display the annotated image to a user; receiving user input associated with the annotated image; and updating the map data to include the location of the traffic signal in the map data based at least in part on the user input as updated map data.

[0068] B. The method of Example A, wherein the first image is associated with an image sensor of an autonomous vehicle, the method further including determining the proposed three-dimensional position of the traffic signal based at least in part on an orientation of the image sensor relative to the map data, and transmitting the updated map data to the autonomous vehicle.

[0069] C. The method of Example A, further comprising: detecting the traffic signal in a third image; determining a second proposed three-dimensional position of the traffic signal in the three-dimensional coordinate system; projecting the second proposed three-dimensional position onto the second image to determine a second two-dimensional position of the traffic signal in the second image; and annotating the second image as an annotated image to further include a second proposed traffic signal position indicator associated with the traffic signal, wherein the position of the second proposed traffic signal position indicator in the second image is based at least in part on the second two-dimensional position of the traffic signal.

[0070] D. The method of Example C, wherein receiving the user input includes receiving a first user input rejecting the proposed three-dimensional position and receiving a second user input accepting the second proposed three-dimensional position, and updating the map data includes adding a map object to the map data corresponding to the traffic signal based on the second proposed three-dimensional position.

[0071] E. The method of example C, wherein the first image is captured by a different device than the third image.

[0072] F. The method of Example A, wherein annotating the second image includes overlaying the proposed traffic light location indicator onto the second image in a space of a two-dimensional image of the second image corresponding to the proposed three-dimensional location.

[0073] G. One or more processors and memory storing processor-executable instructions; 11. A system comprising: a display device for displaying the graphical element to a user; receiving user input; and updating the map data based at least in part on the user input. 12. A system comprising: a display device for displaying the graphical element to a user; a display device for displaying the graphical element to a user; and a display device for displaying the graphical element to a user; and a display device for displaying the graphical element to a user based at least in part on the user input. 13. A system comprising: a display device for displaying the graphical element to a user;

[0074] H. The system of Example G, wherein the traffic object characteristics include one or more of a traffic signal, a traffic signal direction, a traffic signal state, a traffic sign, a lane indicator, or a crosswalk.

[0075] I. The system of Example G, wherein the user input includes a confirmation, and updating the map data includes determining that the map data does not include a traffic indicator associated with the traffic object, and associating the traffic object with the map based at least in part on the proposed characteristics.

[0076] J. The system of Example G, wherein the operations further include detecting the traffic object in third sensor data, determining a second proposed characteristic of the traffic object, determining a second representation of the second proposed characteristic to be displayed in the second sensor data, associating a second graphical element with the second sensor data based on the second representation, and causing the display to display the second graphical element to the user.

[0077] K. The system of Example J, wherein receiving the user input includes receiving a first user input rejecting the proposed characteristic and receiving a second user input accepting the second proposed characteristic, and updating the map data includes one or more of adding or updating the map data to include the traffic object based at least in part on the second proposed characteristic.

[0078] L. The system of example G, wherein the first sensor data includes a first image acquired by a first device and the second sensor data includes an image acquired by a second device.

[0079] M. The system of Example L, wherein the first sensor data includes a first image captured by a first image sensor, the second sensor data includes a second image captured by a second image sensor, the traffic object includes a traffic signal, the proposed characteristic includes a proposed position of the traffic signal relative to the map data, and associating a graphical element with the second sensor data includes projecting the proposed position onto the second image based at least in part on an orientation of an image sensor relative to the map data and a second orientation of the second image sensor relative to the map data.

[0080] N. Updating the map data includes determining updated map data; the first sensor data is captured by a device associated with the autonomous vehicle; The system of Example G, wherein the operations further include transmitting the updated map data to the autonomous vehicle, the autonomous vehicle being configured to traverse an environment based at least in part on the updated map data.

[0081] O. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations including: detecting a traffic object in first sensor data, the traffic object being associated with a traffic object characteristic; determining proposed characteristics of the traffic object in a coordinate system associated with map data; determining a representation of the proposed characteristics to be displayed in second sensor data; associating a graphical element with the second sensor data based on the representation; causing a display to display the graphical element to a user; receiving user input; and updating the map data as updated map data based at least in part on the user input.

[0082] P. One or more non-transitory computer-readable media as described in Example O, wherein the object is a traffic object, the user input includes a confirmation, and updating the map data includes determining that the map data does not include a traffic indicator associated with the traffic object, and associating the traffic object with the map data based at least in part on the proposed characteristic.

[0083] Q. One or more non-transitory computer-readable media of example O, wherein the operations further include detecting the object in third sensor data, determining a second proposed characteristic of the object, determining a second representation of the second proposed characteristic to be displayed in the second image, associating a second graphical element with the second image based on the second representation, and causing the display to display the second graphical element to the user.

[0084] R. One or more non-transitory computer-readable media of Example Q, wherein receiving the user input includes receiving a first user input rejecting the proposed characteristic and receiving a second user input accepting the second proposed characteristic, and updating the map data includes one or more of adding or updating the map data to include the object based at least in part on the second proposed characteristic.

[0085] S. The one or more non-transitory computer-readable media of Example O, wherein the first sensor data includes a first image captured by a first image sensor of a first device, the second sensor data includes a second image captured by a second image sensor of a second device, the object includes a traffic signal, the proposed property includes a proposed location of the traffic signal relative to the map data, and associating a graphical element with the second sensor data includes projecting the proposed location onto the second image based at least in part on an orientation of the second image sensor relative to the map data and a second orientation of the second image sensor relative to the map data.

[0086] T. The one or more non-transitory computer-readable media of Example 15, further comprising: the first sensor data is captured by a device associated with an autonomous vehicle, and the operation is to transmit the updated map data to the autonomous vehicle, wherein the autonomous vehicle is configured to traverse an environment based at least in part on the updated map data.

[0087] Although the exemplary subject matter described above has been described with respect to one particular embodiment, it should be understood in the context of this specification that the exemplary subject matter may also be implemented via a method, device, system, computer-readable medium, and / or other embodiment. In addition, any of Examples A-T may be implemented alone or in combination with any other one or more of Examples A-T.

[0088] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0089] The components described herein represent instructions that may be implemented in software and / or hardware, which may be stored on any type of computer-readable medium. All of the methods and processes described above may be embodied in and fully automated through software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Some or all of the methods may alternatively be embodied in dedicated computer hardware.

[0090] In particular, conditional language such as "may," "could," "may," or "might" is understood, within the context, to indicate that a particular example includes certain features, elements, and / or steps and not other examples, unless otherwise specified. Thus, such conditional language is not intended to generally imply that a particular feature, element, and / or step is required in any way for one or more examples, or that one or more examples necessarily include logic for determining, with or without user input or prompts, that a particular feature, element, and / or step should be included or performed in any particular example.

[0091] Conjunctive language such as the phrase "at least one of X, Y, or Z," unless specifically stated otherwise, should be understood to present that the item, term, etc. can be any combination thereof including any of X, Y, or Z, or a plurality of each element. "a" refers to both singular and plural unless expressly stated as singular.

[0092] Any routine descriptions, elements or blocks in the flow diagrams described herein and / or illustrated in the accompanying drawings should be understood as potentially representing modules, segments, or portions of code that include one or more computer-executable instructions for implementing a particular logical function or element in the routine. Alternative embodiments are included within the scope of the examples described herein, and elements or functions may be omitted or performed in a different order than that shown or described, including substantially synchronously, in reverse order, with additional operations, or omitting operations, depending on the functionality involved, as will be understood by those skilled in the art.

[0093] Many variations and modifications may be made to the above-described examples, and elements thereof should be understood to be within the scope of other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

1. detecting a traffic object in the first sensor data, the traffic object being associated with a traffic object characteristic; determining proposed characteristics of the traffic object in a coordinate system associated with map data; determining a representation of the proposed characteristic to be displayed in the second sensor data; associating a graphical element with the second sensor data based on the representation; causing a display to display said graphical element to a user; Receiving user input; updating the map data based at least in part on the user input; and The method includes:

2. The traffic object characteristics include: traffic lights, traffic light direction, Traffic signal status, traffic signs, lane indicators, or pedestrian crossing, The method of claim 1 , comprising one or more of:

3. The user input includes a confirmation; Updating the map data includes: determining that the map data does not include a traffic indicator associated with the traffic object; associating the traffic object with the map data based at least in part on the proposed characteristics; The method of claim 1 or 2, comprising:

4. detecting the traffic object in third sensor data; determining a second proposed characteristic of the traffic object; determining a second representation of the second proposed characteristic to be displayed in the second sensor data; and associating a second graphical element with the second sensor data based on the second representation; causing the display to display the second graphical element to the user; The method of claim 1 or 2, further comprising:

5. Receiving the user input includes: receiving a first user input rejecting the proposed characteristic; receiving a second user input approving the second proposed characteristic; and Including, Updating the map data includes: adding or updating the map data to include the traffic object based at least in part on the second proposed characteristic. The method according to claim 4.

6. The method of claim 1 , wherein the first sensor data comprises a first image captured by a first device and the second sensor data comprises an image captured by a second device.

7. the first sensor data includes a first image captured by a first image sensor; the second sensor data includes a second image captured by a second image sensor; the traffic object includes a traffic light; the proposed characteristics include a proposed location of the traffic signal relative to the map data; associating a graphical element with the second sensor data includes projecting the proposed location onto the second image based at least in part on an orientation of the first image sensor relative to the map data and a second orientation of the second image sensor relative to the map data.

7. The method of claim 6.

8. Updating the map data includes determining updated map data; the first sensor data is captured by a device associated with an autonomous vehicle; The method comprises: transmitting the updated map data to the autonomous vehicle, the autonomous vehicle being configured to traverse an environment based at least in part on the updated map data; and The method of claim 1 , further comprising:

9. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause one or more computing devices to perform the method of any one of claims 1 to 8.

10. one or more processors; one or more computer-readable media having instructions stored thereon that are executable by one or more processors, the instructions, when executed, causing the one or more processors to: detecting a traffic object in the first sensor data, the traffic object being associated with a traffic object characteristic; determining proposed characteristics of the traffic object in a coordinate system associated with map data; determining a representation of the proposed characteristic to be displayed in the second sensor data; associating a graphical element with the second sensor data based on the representation; causing a display to display said graphical element to a user; Receiving user input; updating the map data as updated map data based at least in part on the user input; and A system that causes a system to perform an operation including:

11. The user input includes a confirmation; Updating the map data includes: determining that the map data does not include a traffic indicator associated with the traffic object; associating the traffic object with the map data based at least in part on the proposed characteristics; The system of claim 10 , comprising:

12. The operation includes: detecting the traffic object in third sensor data; determining a second proposed characteristic of the traffic object; determining a second representation of the second proposed characteristic to be displayed in a second image; associating a second graphical element with the second image based on the second representation; causing the display to display the second graphical element to the user; The system of claim 10 further comprising:

13. Receiving the user input includes: receiving a first user input rejecting the proposed characteristic; receiving a second user input approving the second proposed characteristic; and Including, Updating the map data includes: adding or updating the map data to include the traffic object based at least in part on the second proposed characteristic. The system of claim 12.

14. the first sensor data includes a first image captured by a first image sensor of a first device; the second sensor data includes a second image captured by a second image sensor of a second device; the traffic object includes a traffic light; the proposed characteristics include a proposed location of the traffic signal relative to the map data; associating a graphical element with the second sensor data includes projecting the proposed location onto the second image based at least in part on an orientation of the first image sensor relative to the map data and a second orientation of the second image sensor relative to the map data. A system according to any one of claims 11 to 13.

15. the first sensor data is captured by a device associated with an autonomous vehicle; the operations further include transmitting the updated map data to the autonomous vehicle, the autonomous vehicle being configured to traverse an environment based at least in part on the updated map data. A system according to any one of claims 11 to 14.

Images