Object Localization for Autonomous Driving via Visual Tracking and Image Reprojection

Abstract

To verify the location of an object of a particular type.SOLUTION: For instance, multiple images (401A, 401B, 401C) of an environment of a vehicle (100) may be received (1010). Associated objects (420, 422) of the particular type may be identified in ones of the multiple images (1020). Multiple estimated locations (520, 522, 820A, 820B, 820C) of an object may be determined using multiple different localization techniques (1030). For each image of the ones of the multiple images, a reprojection error is to be determined for each of the multiple estimated locations. For each of the multiple estimated locations, an error score is determined based on the reprojection errors (1040). An estimated location may be selected from the multiple estimated locations based on the determined error score (1050). This selected location may be used to control a vehicle in an autonomous driving mode (1060).SELECTED DRAWING: Figure 9A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 883,745, filed August 7, 2019, which claims the benefit of the filing date of U.S. Patent Application No. 16 / 598,162, filed October 10, 2019, the disclosures of which are incorporated herein by reference. [Background technology]

[0002] Autonomous vehicles, such as vehicles that do not require a human driver, may be used to assist in the transportation of passengers or goods from one location to another. Such vehicles may operate in a fully autonomous driving mode, where the passenger provides some initial input, such as a destination, and the vehicle steers itself to that destination. Such vehicles may therefore be highly dependent on a system that can determine the location of the autonomous vehicle at all times, and detect and identify objects external to the vehicle, such as other vehicles, stop signals, pedestrians, etc. Summary of the Invention

[0003] One aspect of the disclosure provides a method for controlling a vehicle having an autonomous driving mode, the method including receiving, by one or more processors, a plurality of images of an environment of the vehicle, identifying, by the one or more processors, an associated object of a particular type in an image of the plurality of images, determining, by the one or more processors, a plurality of estimated locations for the object using a plurality of different localization techniques, determining, by the one or more processors, an error score for each of the plurality of estimated locations, selecting, by the one or more processors, an estimated location from the plurality of estimated locations based on the determined error score, and controlling, by the one or more processors, the vehicle in the autonomous driving mode based on the selected estimated location.

[0004] In one example, the particular type of object is a stop sign. In another example, the particular type of object is a yield sign. In another example, the method also includes causing the plurality of images to be captured based on a distance between the vehicle and a location of a stop sign in pre-stored map information used to control the vehicle in an autonomous driving mode. In another example, the method also includes causing the plurality of images to be captured based on a detection of the particular type of object by a perception system of the vehicle. In another example, the method also includes identifying a patch corresponding to the particular type of object in a first image of the images of the plurality of images, and the error score is determined further based on the patch. In another example, identifying the particular type of associated object includes identifying a first patch corresponding to the particular type of object in a first image of the images of the plurality of images, identifying a second patch corresponding to the particular type of object in a second image of the images of the plurality of images, projecting a center of the first patch onto the second image, and determining whether the projected center is within an area of ​​the second patch. In another example, the plurality of different localization techniques includes object size localization based on an expected size of the particular type of object. In another example, the plurality of different localization techniques includes depth map localization that is based on sensor data generated by a LIDAR sensor of the vehicle. In another example, the plurality of different localization techniques includes road graph based localization that includes projecting a ray corresponding to a direction of a center of a patch in an image of the plurality of images onto pre-stored map information used to control the vehicle in an autonomous driving mode. In this example, the road graph based localization includes identifying a particular type of object in the map information that is within a threshold distance of the projected ray. In another example, the method also includes determining a reprojection error of each of the plurality of estimated locations for each image of the plurality of images, and determining the error score is further based on the reprojection error.In another example, determining one of the error scores includes identifying a patch corresponding to a particular type of object in a first image of the images of the multiple images, projecting one of the multiple estimated locations onto the first image, and determining a distance between the projected location and the area of ​​the patch. In this example, determining the distance includes determining a distance between a center of the projected location and a center of the area of ​​the patch. Additionally or alternatively, projecting one of the multiple estimated locations onto the first image is based on the pose of the first image. Additionally or alternatively, the distance is one of the reprojection errors. Additionally or alternatively, determining one of the error scores includes averaging the reprojection errors for a given one of the multiple location estimates. In another example, selecting the selected estimated location includes selecting the estimated location having the lowest one of the error scores.

[0005] One aspect of the disclosure provides a system for controlling a vehicle having an autonomous driving mode, the system including one or more processors configured to receive a plurality of images of an environment of the vehicle, identify an associated object of a particular type in the images of the plurality of images, determine a plurality of estimated locations for the object using a plurality of different localization techniques, determine an error score for each of the plurality of estimated locations, select an estimated location from the plurality of estimated locations based on the determined error score, and control the vehicle in the autonomous driving mode based on the selected estimated location.

[0006] In one example, the system also includes a vehicle. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a functional diagram of an exemplary vehicle, according to aspects of the present disclosure. [Diagram 2] 1 is an exemplary map information according to an aspect of the present disclosure. [Diagram 3] FIG. 1 is an exemplary depiction of a vehicle according to aspects of the present disclosure. [Figure 4A] 1 is a plurality of images according to an aspect of the present disclosure. [Figure 4B] 1 is an exemplary image and patch according to an aspect of the present disclosure. [Figure 4C] 1 is an exemplary image and patch according to an aspect of the present disclosure. [Figure 4D] 1 is an exemplary image and patch according to an aspect of the present disclosure. [Diagram 5] 1 is an exemplary depiction of a vehicle operating within a geographic region, according to an aspect of the present disclosure. [Figure 6] 1 is an exemplary depiction of a projection of map information and light rays according to an aspect of the present disclosure. [Figure 7] 13 is an exemplary diagram of a projection of a patch center onto an image, according to aspects of the present disclosure. [Figure 8] 1 is an example depiction of a geographic region and an estimated location of an object, according to an aspect of the present disclosure. [Figure 9A] 1 is an example depiction of an image including image patches, projected estimated locations, and reprojection errors, according to aspects of the present disclosure. [Figure 9B] 1 is an example depiction of an image including image patches, projected estimated locations, and reprojection errors, according to aspects of the present disclosure. [Figure 9C] 1 is an example depiction of an image including image patches, projected estimated locations, and reprojection errors, according to aspects of the present disclosure. [Figure 10] FIG. 1 is an exemplary flow diagram according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] overview The present technology relates to a method of verifying or confirming the location of an object for an autonomous vehicle or a vehicle configured to operate in an autonomous driving mode. For example, different location determination technologies may provide different locations for the same object. As a result, the same object may appear to be in different locations at the same time. In a situation where the object is a stop sign, an autonomous vehicle that operates depending on these locations may react to each of the signs in different locations. For example, the vehicle may stop at various points along a roadway even though there are no stop signs at these locations. Thus, selecting an "optimal" location for the object may enable the vehicle to navigate the roadway more accurately or smoothly.

[0009] One or more cameras of the vehicle's perception system can capture multiple images that may include an object. In some cases, the multiple images can be a series of images captured over time by the same camera or different cameras. When using the same camera, each of these images can be captured when the camera is in the same pose (when the vehicle is not moving) or different poses (when the vehicle is moving). Additionally, the multiple images can be captured when the perception system detects a particular type of object and in response to the vehicle approaching the object, such as a stop or yield sign, or when the vehicle is a predetermined distance from the predicted location of a particular type of object as identified in map information.

[0010] The vehicle's computing device may be configured to identify an object in a first image of the multiple images. This may include using various object recognition techniques to identify a "patch" or a number of pixels in the first image that correspond to the object. The computing device may be configured to engage the object recognition technique continuously or only in certain circumstances, such as when the vehicle is a set distance from the location of a particular type of object, such as a stop sign or a yield sign, as determined from information stored in map information.

[0011] If the object identified in the first image is of a particular type, the object may be located using a number of different location techniques. A first exemplary location technique may include object size location. For example, based on the type of object, the vehicle's computing device may use the expected size of the object to estimate the distance between the vehicle and the object. Combining this estimated distance with the pose of the image capture location when the image was captured may be used to estimate the three-dimensional (3D) location of the object. In some cases, if the expected size of the object is incorrect, if the object in the image is partially occluded, or if the object simply does not fit the expected size (i.e., broken sign or manufacturing error), the estimated location of the object determined using this technique may be incorrect.

[0012] A second exemplary localization technique may include depth map localization. For example, the vehicle's computing device may generate a depth map using sensor data generated by a LIDAR sensor of the perception system. The camera pose may be used to project the depth map onto an image to estimate the 3D location of the object. In some cases, if the LIDAR sensor's view of the object is partially obstructed by another road user, debris, leaves, tree branches, signs, or other objects, the estimated location of the object determined using this technique may be incorrect.

[0013] A third exemplary localization technique may include road graph-based localization. For example, the vehicle's computing device may determine the orientation of the center or some other location on the patch relative to the location of the camera (and / or vehicle) and then use this orientation to project a ray onto the map information. This ray may represent the coordinates where the object may find a given location of the object in the image relative to the pose of the camera when the image was captured. One or more objects, such as a mapped stop sign, may be identified that are along or within a predetermined distance of the calculated ray. Another estimated location of the object may then be determined using the location of any identified objects in the map information. In some cases, if the object is damaged or moving (e.g., a bent sign pole for a stop sign or yield sign), temporary (such as a temporary stop sign), or new and not displayed in the map information, or if there are multiple objects of the same type along the ray, the estimated location of the object determined using this technique may be incorrect.

[0014] The vehicle's computing device may determine whether any object in an image is the same object as an object in other images of the multiple images and may associate these objects with each other. For example, the computing device may determine whether the same stop sign appears in each of the multiple images. The association may be generated by projecting a pixel location of one object in one image to a pixel location of another object in another image. The projected pixel location may be adjusted according to the difference between the camera poses when the two images were captured. If the projected pixel location of one object in one image is within a threshold pixel distance of the pixel location of another object in another image, an association is generated indicating that the patches in each image represent the same object (hereinafter, associated objects). These associations may be used to validate the estimated locations and select one of the estimated locations for the object's location when the first image was captured.

[0015] For each image that includes an object associated with another object in another image, the vehicle's computing device may determine a reprojection error for each of the estimated locations of the associated object. To do so, each estimated location for the associated object may be projected onto each image in which the associated object is identified. The reprojection error may be (or may be derived from) the distance between the projected estimated location and the patch corresponding to the one associated with that image. This process may be repeated for any other estimated locations and for a predetermined number (e.g., three or more images or less) or all of the images in which the associated object is identified.

[0016] The vehicle's computing device may then determine an error score for each localization technique, or rather, for each estimated location determined using different localization techniques. As an example, reprojection error scores determined from projections of the same estimated location projected onto different images in which the associated object is displayed may be averaged together. In this regard, if there are three localization techniques or three estimated locations for an object, there will be three error scores.

[0017] The vehicle's computing device may then select a localization technique or an estimated location of the object at the time the first image was captured based on the determined error score. For example, the estimated location with the lowest error score may be determined to be the location of the object at the time the first image was captured.

[0018] The computing device 110 may be configured to control the vehicle in an autonomous driving mode to respond to the object according to a selected estimated location. For example, if the object is a stop sign, the vehicle's computing device may operate one or more autonomous driving systems to navigate the autonomous vehicle to stop according to a selected estimated location and ignore all other estimated locations for the stop sign. Similarly, if the object is a yield sign, the vehicle's computing device may operate one or more autonomous driving systems to navigate the autonomous vehicle to yield according to a selected estimated location of the yield sign and ignore all other estimated locations for the yield sign.

[0019] The above features may provide a system that more accurately identifies the location of certain types of objects along the roadway of an autonomous vehicle. In particular, stop sign locations may be identified. As a result, the autonomous vehicle may be less likely to respond to an incorrect location of an object, for example, to improperly stop where there is no stop sign, or to yield where there is no yield sign. Such situations may be disruptive and potentially dangerous not only to the autonomous vehicle but also to other road users. Furthermore, because such behaviors can be avoided, occupants of the autonomous vehicle may experience a smoother ride and reach their destination more efficiently. In addition, the autonomous vehicle's resources may be more efficiently utilized since they do not need to be used to respond to incorrectly located objects.

[0020] Exemplary System 1 provides an example block diagram of a vehicle 100 including various components. Although some aspects of the disclosure are particularly useful in connection with certain types of vehicles, the vehicle may be any type of vehicle, including, but not limited to, automobiles, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle may have one or more computing devices, such as a computing device 110 that includes one or more processors 120, memory 130, and other components typically present in a general-purpose computing device.

[0021] Memory 130 stores information accessible by one or more processors 120, including instructions 132 and data 134 that may be executed or otherwise used by processor 120. Memory 130 may be any type of memory capable of storing information accessible by a processor, including computing device readable media or other media that store data that can be read using electronic devices such as hard drives, memory cards, ROM, RAM, DVDs or other optical disks, and other writable and read-only memories. Systems and methods may include different combinations of the above, whereby different portions of the instructions and data are stored on different types of media.

[0022] The instructions 132 may be any set of instructions executed directly (such as machine code) or indirectly (such as script) by a processor. For example, the instructions may be stored as computing device code on a computing device readable medium. In that regard, the terms "instructions" and "program" may be used interchangeably herein. The instructions may be stored in object code format for direct processing by a processor, or in any other computing device language, including a collection of scripts or independent source code modules that are interpreted on demand or precompiled. The functions, methods, and routines of the instructions are described in further detail below.

[0023] Data 134 may be retrieved, stored, or modified by processor 120 in accordance with instructions 132. For example, although claimed subject matter is not limited to any particular data structure, the data may be stored in a computing device register, in a relational database as a table having multiple different fields and records, an XML document, or a flat file. The data may also be formatted in any computing device readable format.

[0024] The one or more processors 120 may be any conventional processor, such as a commercially available CPU or GPU. Alternatively, the one or more processors may be dedicated devices, such as an ASIC or other hardware-based processor. Although FIG. 1 functionally illustrates the processor, memory, and other elements of the computing device 110 as being in the same block, it will be understood by those skilled in the art that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored in the same physical enclosure. For example, the memory may be a hard drive or other storage medium located in an enclosure different from that of the computing device 110. Thus, reference to a processor or computing device should be understood to include reference to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0025] In one aspect, computing device 110 may be part of an autonomous control system capable of communicating with various components of the vehicle to control the vehicle in an autonomous driving mode. For example, returning to FIG. 1 , computing device 110 may communicate with various systems of vehicle 100, such as deceleration system 160, acceleration system 162, steering system 164, routing system 166, planning system 168, positioning system 170, and perception system 172, to control the movement, speed, etc. of vehicle 100 in accordance with instructions 132 in memory 130 in the autonomous driving mode.

[0026] As an example, computing device 110 may interact with deceleration system 160 and acceleration system 162 to control the speed of the vehicle. Similarly, steering system 164 may be used by computing device 110 to control the direction of vehicle 100. For example, if vehicle 100 is configured for use on a road, such as a car or truck, the steering system may include components that control the angle of the wheels to turn the vehicle.

[0027] The planning system 168 may be used by the computing device 110 to determine and follow a route generated by the routing system 166 to a location. For example, the routing system 166 may use map information to determine a route from the vehicle's current location to a drop-off location. The planning system 168 may generate a short-term plan for controlling the vehicle periodically, trajectory, or over a period of time in the future to follow a route (the vehicle's current route) to a destination. In this regard, the planning system 168, the routing system 166, and / or the data 134 may store detailed map information, such as high-precision maps that identify roadway shapes and elevations, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects and information.

[0028] The map information may be pre-stored and may include one or more road graphs or graph networks of information such as roads, lanes, intersections, and connections between these features, which may be represented by road segments. Each feature may be stored as graph data and may be associated with information such as a geographic location, whether or not linked to other related features, e.g., a stop sign may be linked to a road and an intersection, etc. In some examples, the associated data may include a grid-based index of the road graph, allowing efficient searching of features of a particular road graph.

[0029] As shown in Fig. 2, an example of map information, here map information 200, includes details related to intersections 202 and 204. Roadways 210 and 212 merge at intersection 202, and roadways 212 and 214 merge at intersection 204. Roadway 210 has a lane 230 heading in a first direction and a lane 232 heading in a second direction opposite the first direction. Roadway 212 has a lane 234 heading in a third direction perpendicular to the first and second directions. Roadway 214 has a lane 236 heading in a third direction and a lane 238 heading in a fourth direction opposite the third direction and passing through intersection 202. The map information also includes location information for stop signs 220, 222. In particular, the location of stop sign 220 may be shown to be at intersection 202 on the lane A side of roadway 210 , and the location of stop sign 222 may be shown to be at intersection 204 on the lane 236 side of roadway 214 .

[0030] The positioning system 170 may be used by the computing device 110 to determine the relative or absolute location of the vehicle on a map or on the Earth. For example, the positioning system 170 may include a GPS receiver to determine the latitude, longitude, and / or altitude location of the device. Other location systems, such as laser-based location systems, inertial-aided GPS, or camera-based location, may also be used to determine the location of the vehicle. The location of the vehicle may include absolute geographic location information, such as latitude, longitude, and altitude, as well as relative location information, such as location with respect to other vehicles in the immediate vicinity, which can often be determined with less noise than absolute geographic location.

[0031] Positioning system 170 may also include other devices in communication with computing device 110, such as an accelerometer, gyroscope, or another direction / speed detection device for determining the vehicle's direction and speed, or changes thereto. By way of example only, an acceleration device may determine the pitch, yaw, or roll (or changes thereto) of the vehicle relative to the direction of gravity, or a plane perpendicular to gravity. The device may also track increases or decreases in speed, and the direction of such changes. Provision of device position and orientation data as described herein may be provided automatically to computing device 110, other computing devices, and combinations of the above.

[0032] Perception system 172 also includes one or more components for detecting objects external to the vehicle, such as other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc. For example, perception system 172 may include lasers, sonar, radar, cameras, and / or any other detection devices that record data that can be processed by a computing device of computing device 110. In the case where the vehicle is a passenger vehicle such as a minivan, the minivan may include a laser or other sensor mounted on the roof or other convenient location.

[0033] 3 is an example exterior view of vehicle 100 including an embodiment of perception system 172. For example, housing 310 and dome-shaped housing 312 on the roof may include a LIDAR sensor or system, as well as various cameras and radar units. In addition, housing 320 located at the front end of vehicle 100, and housings 330, 332 on the driver's and passenger's sides of the vehicle may each house a LIDAR sensor. For example, housing 330 is located in front of driver door 360. Vehicle 100 also includes housings 340, 342 for radar units and / or cameras also located on the roof of vehicle 100. Additional radar units and cameras (not shown) may be located at the front and rear ends of vehicle 100 and / or on the roof or other locations along housing 310 on the roof.

[0034] Computing device 110 may be capable of communicating with various components of the vehicle to control the movement of vehicle 100 in accordance with primary vehicle control code in memory of computing device 110. For example, returning to FIG. 1 , computing device 110 may include various computing devices in communication with various systems of vehicle 100, such as deceleration system 160, acceleration system 162, steering system 164, routing system 166, planning system 168, positioning system 170, and perception system 172, power system 174 (i.e., the vehicle's engine or motor), to control the movement, speed, etc. of vehicle 100 in accordance with instructions 132 in memory 130.

[0035] Various systems of the vehicle may function using autonomous vehicle control software to determine and control how to control the vehicle. As an example, the perception system software module of the perception system 172 may detect and identify objects and their features using sensor data generated by one or more sensors of the autonomous vehicle, such as cameras, LIDAR sensors, radar units, sonar units, etc. These features may include location, type, heading, orientation, speed, acceleration, change in acceleration, size, shape, etc. In some cases, the features may be input into a behavior prediction system software module that uses various behavior models based on the object type to output a predicted future behavior of the detected object. In other examples, the features may be put into one or more detection system software modules, such as a traffic light detection system software module configured to detect known traffic light conditions, a construction zone detection system software module configured to detect construction zones from sensor data generated by one or more sensors of the vehicle, as well as an emergency vehicle detection system software module configured to detect emergency vehicles from sensor data generated by the sensors of the vehicle. Each of these detection system software modules may use various models to output the likelihood that a construction zone or object is an emergency vehicle. Detected objects, predicted future actions, various probabilities from the detection system software module, map information identifying the vehicle's environment, location information from positioning system 170 identifying the vehicle's location and orientation, the vehicle's destination, and feedback from various other systems of the vehicle may be input to a planning system software module of planning system 168. The planning system may use this input to generate a trajectory for the vehicle to follow for some short period of time in the future based on a route generated by a routing module of routing system 166. A control system software module of computing device 110 may be configured to control the movement of the vehicle, for example, by controlling the braking, acceleration, and steering of the vehicle to follow the trajectory.

[0036] Computing device 110 can control the vehicle in an autonomous driving mode by controlling various components. As an example, computing device 110 can navigate the vehicle fully autonomously to a destination location using detailed map information and data from planning system 168. Computing device 110 can use positioning system 170 to determine the location of the vehicle and perception system 172 to detect and respond to objects when needed to safely arrive at the location. Again, to do so, the computing device 110 may generate trajectories and cause the vehicle to follow those trajectories, for example, by causing the vehicle to accelerate (e.g., by providing fuel or other energy to the engine or power system 174 via the acceleration system 162), decelerate (e.g., by reducing fuel provided to the engine or power system 174, changing gears, and / or applying the brakes via the deceleration system 160), change direction (e.g., by turning the front or rear wheels of the vehicle 100 via the steering system 164), and signal such changes (e.g., by illuminating the vehicle's turn signals). Thus, the acceleration system 162 and deceleration system 160 may be part of a driveline that includes various components between the vehicle's engine and the vehicle's wheels. Again, by controlling these systems, the computing device 110 may also control the vehicle's driveline in order to steer the vehicle autonomously.

[0037] Exemplary Methods In addition to the operations described above and illustrated in the figures, various operations are described herein. It should be understood that the following operations do not have to be performed in the exact order described below. Rather, various steps may be processed in different orders or simultaneously, and steps may also be added or omitted.

[0038] 10 is an example flow diagram 1000 of an example method of controlling a vehicle having autonomous driving that may be executed by one or more processors, such as one or more processors 120 of the computing device 110, one or more processors of the perception system 172, or a combination of both. For example, at block 1010, a plurality of images of an environment of the vehicle having an autonomous driving mode may be received. As an example, one or more cameras of the vehicle's perception system 172 may capture a plurality of images that may include an object. In some cases, the plurality of images may be a series of images captured over time by the same camera or different cameras. When using the same camera, each of these images may be captured when the camera is in the same pose (when the vehicle is not moving) or a different pose (when the vehicle is moving).

[0039] Multiple images may be captured for a variety of reasons. As the vehicle is driven through its environment, the vehicle's perception system may detect and identify objects, including objects of a particular type, such as stop signs or yield signs. If the vehicle is approaching an object of a particular type, the computing device 110 may determine an estimated location of the object by capturing multiple images. Alternatively, multiple images may be captured when the vehicle is at a predetermined distance, which may correspond to the maximum range of the vehicle's sensors, such as LIDAR sensors used for depth map localization, from the expected location of an object of a particular type as identified in the map information, and when the vehicle is approaching the object.

[0040] 5, vehicle 100 may be driven within geographic region 500 that includes the geographic region depicted in stored map portion 200. In this regard, the shape and location of each of intersections 502, 504 generally corresponds to the shape and location of each of intersections 202, 204, respectively, the shape and location of stop signs 520, 522 generally corresponds to the shape and location of stop signs 220, 222, and the shape and location of each of lanes 530, 532, 534, 536, 538 generally corresponds to the shape and location of each of lanes 230, 232, 234, 236, 238, respectively.

[0041] In this example, the perception system 172 may identify stop sign 520 as a stop sign and provide this information to the computing device 110. In response, the computing device may cause the perception system 172 to capture multiple images. Alternatively, based on the identification of the stop sign, the perception system may automatically capture multiple images. As another alternative, the computing device 110 may cause the perception system 172 to capture multiple images once the computing device determines that the vehicle is a predetermined distance from a stop sign 220 in the map information, either in a straight line or along the route the vehicle is currently following.

[0042] For example, multiple images may be captured when a stop sign is detected by the perception system 172 or when the vehicle 100 is a predetermined distance from the location of the stop sign in the map information 200. In some cases, the multiple images may be a series of images captured over time by the same camera or different cameras. When using the same camera, each of these images may be captured when the camera is in the same pose (when the vehicle is not moving) or different poses (when the vehicle is moving). FIG. 4A provides an exemplary multiple images 400 including a first image 401A may be captured at a first time point with a first pose of the camera of the perception system 172, a second image 401B may be captured at a second time point after the first time point with a second pose of the camera, and a third image 401C may be captured at a third time point after the second time point with a third pose. The first, second, and third images may be captured consecutively at 10 Hz or other frequency. In this regard, FIG. 5 may represent the time when the image 401A was captured. As the vehicle approaches the stop sign 520, a second image 401B may be captured, followed by a third image 401C.

[0043] In some implementations, the multiple images may include a first image 401A and a second image 401B may be captured and processed as described herein. Additional images beyond the first image 401A and the second image 401B, such as a third image 401C, may be included in the multiple images and thus captured and processed as described herein. The number of additional images captured and processed may depend on the delay requirements of the system or the time frame in which an estimated location of a stop sign is required. For example, if an estimated location is required within 0.5 seconds of detecting a given stop sign and processing takes 0.1 seconds, four images may be captured and processed at a frequency of 10 Hz or greater.

[0044] The pose of each image of the multiple images may be determined based on the location of the vehicle and the relative location of the camera that captured the image with respect to the vehicle. For example, using the positioning system 170 and map information, the vehicle's computing device 110 may determine that when the first image 401A was captured from the camera of the perception system 172, the vehicle 100 was located in lane 530 of the roadway 210 traveling in a direction toward the intersection 504. This, in combination with the relative location and orientation of the camera that captured the image with respect to the vehicle 100 at the time the image was captured, may be used to determine a first pose of the first image 401A. A similar process may be repeated for each image of the multiple images.

[0045] In this example, the first image 401A may capture intersections 402A, 404A, roadways 410A, 412A, 414A, and stop signs 420A, 422A. The roadway 410A may include lane A (corresponding to lane 230), which is a travel lane of the vehicle 100 heading in a first direction, and lane B (corresponding to lane 232), which heads in a second direction opposite the first direction, the roadway 412A may include one or more lanes heading in a third direction perpendicular to the first and second directions, and the roadway 414A may include lane D (corresponding to lane 236), which heads in the third direction, and lane E, which heads in a fourth direction opposite the third direction. A second image 401B, taken after vehicle 100 has traveled a distance down lane A, may capture intersection 402B, 404B, roadway 410B, 412B, 414B, and stop sign 420B. A third image 401C, taken after vehicle 100 has traveled a further distance down lane A, may capture intersection 402C, roadway 410C, 412C, and stop sign 420C. In this example, stop signs 420A, 420B, and 420C may be the same stop sign, but this association is not "known" to computing device 110 based solely on the detection of the stop sign in each of these three images.

[0046] Returning to FIG. 10, at block 1020, an associated object of a particular type in an image of the plurality of images is identified. For example, the computing device 110 may be configured to identify an object of a particular type in a first image of the plurality of images. This first image may be the first image in time, the second image in time, or some other image of the plurality. Identifying the object may include identifying a "patch" or a number of pixels in the first image that corresponds to the object using various object recognition techniques. Such techniques may include convolutional deep networks that detect specific object regions in the camera images. These networks may be trained by supervised learning using labeled examples of regions of training images that contain specific objects, such as stop signs or yield signs.

[0047] For example, FIG. 4B depicts an example patch (represented by the area surrounded by dashed lines) of a stop sign 420A in image 401A. Similar patches 430B, 430C may also be determined for stop signs 420B and 420C in images 401B1 and 401C, respectively, as shown in FIGS. 4C and 4D. The computing device 110 may be configured to engage in object recognition techniques continuously, or only in certain circumstances, such as when the vehicle is a set distance from the location of a particular type of object, such as a stop sign or a yield sign, as determined from information stored in map information. This process may be used to identify other objects of a particular type in multiple other images.

[0048] The computing device 110 may determine whether any object in an image is the same object as an object in other images of the multiple images and may associate these objects with each other. For example, the computing device may determine whether the same stop sign appears in each of the multiple images. The association may be generated by projecting the center of a patch of one object in one image onto an image of another object in another image. The projected pixel location may be adjusted according to the difference between the camera poses when the two images were captured. If the projection overlaps or is within a region that comprises a patch corresponding to another object in another image, an association is generated indicating that the patches in each image represent the same object (hereinafter associated objects). Otherwise, the patches do not correspond to the same object. The association may be used to validate the estimated location and select one of the estimated locations for the location of the object when the first image was captured. The association may be generated before, simultaneously with, or after generating the estimated location.

[0049] As shown in FIG. 7, the center 710 of patch 430B of stop sign 420B may be projected onto image 401A. In this example, center 710 of patch 430B is within the area of ​​patch 430A. Thus, computing device 110 may determine that the patch of stop sign 420A and patch 430B are the same object. Furthermore, the center of patch 430A may be projected onto second image 401B. If the projected position of the center of patch 430A is within the area of ​​patch 430B, computing device 110 may determine that the patch corresponding to stop sign 420B and the patch corresponding to stop sign 420A are the same object. These projections may be performed for each pair of images among the multiple images having identified patches corresponding to a particular type of object.

[0050] Returning to FIG. 10, in block 1030, multiple estimated locations are determined for the object using multiple different localization techniques. A first exemplary localization technique may include object size localization. For example, based on the type of object, the vehicle's computing device may use the expected size or expected dimensions of the object to estimate the distance between the vehicle and the object. This may include comparing the patch dimensions, e.g., pixel height and or width, to the expected dimensions of the object at various distances and identifying a distance associated with the expected dimensions that are closest to the patch dimensions. Alternatively, the patch dimensions may be input into an equation that provides an estimated distance based on the expected dimensions of the object. This may be beneficial because the dimensions of certain objects, such as stop and yield signs, may be fairly consistent or otherwise regulated or mandated by law. Combining this estimated distance with the pose of the image capture location when the image was captured may be used to estimate the three-dimensional (3D) location of the object. For example, comparing images 401A, 401B, and 401C, stop signs 420A, 420B, 420C appear larger as the distance between the vehicle (or the camera that captured the image) and the stop sign decreases. Turning to the example of Figure 4A, the vehicle's computing device 110 may use a first localization technique, such as object size localization as described above, to determine a first estimated location of stop sign 420A.

[0051] A second exemplary localization technique may include depth map localization. For example, the vehicle's computing device may use sensor data generated by the LIDAR sensor of the perception system to provide a depth map. The 3D location of the object may be estimated by projecting the image onto the depth map using the camera pose. Returning to the example of FIG. 4A, the computing device 110 may use a second localization technique, such as depth map localization, to determine a second estimated location of the stop sign 420A. At the same time that the image 401A is captured, or very close in time, the LIDAR sensor of the perception system 172 may generate sensor data that identifies the location (x, y, z) and intensity of the surface of the object in the vehicle's environment. This sensor data may be used to generate a depth map that identifies the surfaces in the vehicle's environment relative to one another in 3D, and the image 401A, or simply the patch 430A, may be projected onto the depth map using the pose of the camera that captured the image 401A when the image was captured. The location on the surface of the depth map onto which patch 430A is projected may be a second estimated location of stop sign 420A.

[0052] A third exemplary localization technique may include road graph-based localization. For example, the vehicle's computing device may determine the orientation of the center or some other location on the patch relative to the location of the camera (and / or vehicle) and then use this orientation to project a ray onto the map information. This ray may represent coordinates where an object may find a given location of the object in the image relative to the pose of the camera when the image was captured. One or more objects, such as a mapped stop sign, may be identified that are along or within a predetermined distance of the calculated ray. Another estimated location of the object may then be determined using the location of any identified objects in the map information.

[0053] Returning to the example of FIG. 4A , the computing device 110 may determine a third estimated location of the stop sign 420A using a third localization technique, such as road graph-based localization. The road graph-based localization may locate the stop sign 420A to obtain another estimated location of the object. The road graph-based localization may include calculating a three-dimensional ray from a location of the camera that captured the first image 401A when the image was captured through the center of the patch 430A in the first image 401A. The location of the camera may be derived from the first pose, the position of the vehicle, and the location of the camera relative to the vehicle when the image 401A was captured.

[0054] 6, a ray of light 610 oriented in a direction corresponding to the center of patch 430 may be projected from the determined location of vehicle 100 onto map information based on a first pose of the camera. The angle of the projected ray relative to the first pose of the camera may be determined by determining a location of stop sign 420A relative to the center of first image 401A and mapping the difference between the location of stop sign 420A and the center of first image 401A to an angular distance in geographic region 500.

[0055] A mapped stop sign may be identified as having x, y coordinates along or within a threshold distance from the coordinates of the ray. In some cases, being within a threshold distance of the z coordinate may also be required. The threshold distance may be determined based on the density of stop signs to disambiguate between stop signs in a geographic area. For example, a higher density of stop signs may require a lower threshold distance than a lower density of stop signs. As shown in FIG. 6, using a threshold distance of 5 feet in the x dimension and 5 feet in the y dimension, the ray 610 may pass within 5 feet in the x and y dimensions of the first stop sign 220 according to the location information of the first stop sign 220 in the map information of the map portion 200. Other stop signs, such as the second stop sign 222, are not within 5 feet of the ray 610. Thus, the first stop sign 220 may be identified.

[0056] The coordinates of the first mapped stop sign stored in the map information may be determined to be an estimated location for the road graph-based localization. For example, the location information of the first stop sign 220 may be used as a third estimated location of the stop sign 420A captured in the first image 401A.

[0057] In some cases, an additional number of mapped stop signs may also be identified using the road graph-based localization described above as having x,y coordinates within a threshold distance from the calculated ray and having a detectable orientation. The coordinates of each of the additional number of mapped stop signs may be added to a set of estimated locations for the first stop sign. Alternatively, the mapped stop sign of the first mapped stop sign closest to the calculated ray, and the coordinates of the additional number of mapped stop signs may be selected as the estimated location of the first stop sign and added to the set of estimated locations.

[0058] In other examples, a first mapped stop sign identified as being along or within a predetermined threshold distance of the light beam may also be identified as having an orientation that allows the camera to detect the first mapped stop sign as a stop sign. For example, returning to FIG. 6, based on the map information of the map portion 200, the orientation of the first mapped stop sign 220 may be determined to face the lane 230 in which the vehicle 100 is traveling such that the word STOP on the first stop sign 220 is detectable by the camera. If the first mapped stop sign does not have an orientation that allows the camera to detect the first mapped stop sign as a stop sign, the first mapped stop sign may be excluded from the location process. For example, according to the map information, the orientation of the second stop sign 222 does not face the lane 230 such that the word STOP is captured by the camera of the vehicle 100. Thus, if the second stop sign 222 is within a threshold distance from the coordinates of the ray, the second stop sign 222 is excluded from the road graph location process and / or ignored by the vehicle.

[0059] FIG. 8 provides an example depiction of multiple estimated locations 820A, 820B, 820C of a stop sign 520 that may be determined by a first localization technique, a second localization technique, and a third localization technique, such as object size localization, depth map localization, and the road graph-based localization technique described above. In this example, each localization technique may have some associated error. For example, for object size localization, if the expected size or dimensions of the object are incorrect, if the object in the image is partially occluded, or if the object simply does not conform to the expected size or dimensions (i.e., broken sign, manufacturing error, or other inconsistency), the estimated location of the object determined using this technique may be incorrect. As another example, for depth map localization, if the LIDAR sensor's view of the object is partially occluded by another road user, debris, fallen leaves, tree branches, signs, or other objects, etc., the estimated location of the object determined using this technique may be incorrect. As another example, road graph-based localization may be inaccurate if the object is damaged or moved (e.g., a bent sign pole for a stop sign or yield sign), temporary (such as a temporary stop sign), or new and does not appear in the map information, or if there are multiple objects of the same type along the ray, the estimated location of an object determined using this technique may be incorrect.

[0060] Returning to FIG. 10, at block 1040, an error score may be determined for each of the multiple estimated locations. This error score may be based on a reprojection error. For example, for each image that includes an object associated with another object in another image, the computing device 110 may determine a reprojection error for each of the estimated locations of the associated object. To do so, each estimated location of the associated object may be projected onto each image in which the associated object was identified. The reprojection error may be (or may be derived from) the distance between the projected estimated location and the patch corresponding to the one associated with that image. This process may be repeated for any other estimated locations and for a predetermined number (e.g., three or more images or less) or all of the images in which the associated object is present.

[0061] 9A, 9B, and 9C represent detailed views of images 401A, 401B, and 401C, respectively, showing the areas of patches 430A, 430B, and 430C, respectively, without the details of the images themselves for ease of understanding. As shown in FIG. 9A, a first estimated location 820A, a second estimated location 820B, and a third estimated location 820C may be projected onto the first image 401A. Points 920A, 920B, and 920C represent the centers of estimated locations 820A, 820B, and 820C, respectively, and points 930A, 930B, and 930C represent the centers of patches 430A, 430B, and 430C, respectively.

[0062] Distances DA1, DB1, and DC1 between the estimated location and the patch 430A, e.g., between the center of the patch 430A and the center of each of the estimated locations, may be identified as a first reprojection error for each of the estimated locations. Turning to FIG. 8B, the first estimated location 820A, the second estimated location 820B, and the third estimated location 820C may be projected onto the first image 401B. Distances DA2, DB2, and DC2 between the estimated location corresponding to the pixel of the image 401B identified for the stop sign 420B and the patch 430B, e.g., the distance between the center of the patch and the center of each of the estimated locations, may be identified as a second reprojection error for each of the estimated locations. Turning to FIG. 8C, the first estimated location 820A, the second estimated location 820B, and the third estimated location 820C may be projected onto the first image 401B. Distances DA3, DC3, and DC3 between the estimated locations and patch 430C, for example between the center of patch 430C and the center of each of the estimated locations, may be identified as the reprojection error for each of the estimated locations.

[0063] The computing device 110 may then determine an error score for each localization technique, or rather, for each estimated location of the multiple estimated locations determined using different localization techniques. As an example, reprojection error scores determined from projections of the same estimated location projected onto different images in which associated objects are displayed may be averaged together. For example, the distances DA1, DA2, and DA3 may be averaged together to determine an error score for the estimated location 820A. Further, the distances DB1, DB2, and DB3 may be averaged together to determine an error score for the estimated location 820B, and the distances DC1, DC2, and DC3 may be averaged together to determine an error score for the estimated location 820A.

[0064] The number of error scores may depend on the number of localization techniques among the multiple localization techniques. For example, if there are two localization techniques or two estimated locations for an object, there will be two error scores. Similarly, if there are three localization techniques or three estimated locations for an object, there will be three error scores. Thus, if there are additional localization techniques or estimated locations for an object, there will be additional error scores.

[0065] Returning to FIG. 10, in block 1050, an estimated location is selected from the multiple estimated locations based on the determined error score. For example, the computing device 110 may select a localization technique or estimated location from the multiple estimated locations for the object at the time the first image is captured based on the determined error score. For example, the estimated location with the lowest error score may be determined to be the location of the object at the time the first image is captured. In other words, the estimated location with the shortest average projection distance to the patch of the multiple images may be selected. For example, the estimated location 820B may be selected as the average of the distances DC1, DC2, and DC3 may be less than the average of the distances DA1, DA2, and DA3, and the average of the distances DC1, DC2, and DC3.

[0066] Returning to FIG. 10, at block 1060, the vehicle is controlled in an autonomous driving mode based on the selected estimated location. For example, the computing device 110 may be configured to control the vehicle in an autonomous driving mode to respond to the object according to the selected estimated location. This may include inputting the selected location into the planning system 168 or a behavioral model. For example, if the object is a stop sign, the computing device of the vehicle operates one or more autonomous driving systems to navigate the autonomous vehicle to stop according to the selected estimated location and ignore all other estimated locations for the stop sign. Similarly, if the object is a yield sign, the computing device of the vehicle operates one or more autonomous driving systems to navigate the autonomous vehicle to yield according to the selected estimated location of the yield sign and ignore all other estimated locations for the yield sign.

[0067] In some cases, if the object is a stop sign and the estimated location of the selected stop sign does not correspond to (e.g., is not the same as or is very close to) the coordinates of the mapped stop sign, the stop sign is identified as a temporary, or possibly new, stop sign. The same may be true for yield signs or other types of objects. Additionally, this information may be stored in the map information and / or transmitted to a remote computing device for further analysis.

[0068] In an alternative example, the vehicle's computing device 110 may be further configured to identify a trajectory of a moving sign using the multiple images. The moving sign may be a temporary sign that moves to an active location, such as a stop sign on a handheld or a stop sign on a bus. A prediction regarding the location of the temporary sign or when the temporary sign will be at the active location may be determined by the computing device 110. The vehicle 100 may then be operated by the computing device to navigate according to the prediction.

[0069] The above features may provide a system that more accurately identifies the location of certain types of objects along the roadway of an autonomous vehicle. In particular, stop sign locations may be identified. As a result, the autonomous vehicle may be less likely to respond to an incorrect location of an object, for example, to improperly stop where there is no stop sign, or to yield where there is no yield sign. Such situations may be disruptive and potentially dangerous not only to the autonomous vehicle but also to other road users. Furthermore, because such behaviors can be avoided, occupants of the autonomous vehicle may experience a smoother ride and reach their destination more efficiently. In addition, the autonomous vehicle's resources may be more efficiently utilized since they do not need to be used to respond to incorrectly located objects.

[0070] Unless otherwise stated, the aforementioned alternatives are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the features discussed above may be utilized without departing from the subject matter defined by the claims, and therefore the foregoing description of the embodiments should be considered as illustrative, and not limiting, of the subject matter defined by the claims. In addition, the examples described herein, as well as the presentation of phrases such as "including," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example, but rather, the example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference symbols in different drawings may identify the same or similar elements.

Claims

1. 1. A method for controlling a vehicle having an autonomous driving mode, comprising: receiving, by one or more processors, a plurality of images of an environment of the vehicle; identifying, by the one or more processors, an object in each of the plurality of images; determining, by the one or more processors, a plurality of estimated locations for the object in each of the plurality of images based on the identifying and using a plurality of different localization techniques; selecting, by the one or more processors, in each of the plurality of images, an estimated location from the plurality of estimated locations based on an error score for each of the plurality of estimated locations; and controlling, by the one or more processors, the vehicle in the autonomous driving mode based on the selected estimated location.

2. The method of claim 1 , further comprising capturing the plurality of images at a plurality of different poses by a camera while the vehicle is moving.

3. The method of claim 1 , further comprising capturing the plurality of images when the vehicle is a predetermined distance from the object defined in pre-stored map information.

4. The method of claim 1, wherein identifying the object in each of the plurality of images includes projecting a region of an image patch corresponding to the object from one of the plurality of images onto another of the plurality of images.

5. The method of claim 1 , further comprising identifying a patch in a first image of the plurality of images that corresponds to the object, and determining the error score based on the patch.

6. The method of claim 1 , wherein selecting the estimated location comprises selecting the estimated location having the lowest one of the error scores.

7. Determining one of the error scores comprises: identifying a patch in a first image of the plurality of images that corresponds to the object; projecting an estimated location of the plurality of estimated locations onto the first image; and determining a distance between the projected location and an area of ​​the patch.

8. The method of claim 7 , wherein determining the distance comprises determining a distance between a center of the projected location and a center of the area of ​​the patch.

9. Determining the plurality of estimated locations for the object includes: identifying a first patch corresponding to the object in a first image of the plurality of images; identifying a second patch corresponding to the object in a second image of the plurality of images; projecting the center of the first patch onto the second image; and determining whether the projected center is within an area of ​​the second patch.

10. 1. A system for controlling a vehicle having an autonomous driving mode, comprising: receiving a plurality of images of an environment of the vehicle; identifying an object in each of the plurality of images; determining a plurality of estimated locations for the object in each of the plurality of images based on the identifying and using a plurality of different localization techniques; selecting an estimated location from the plurality of estimated locations based on an error score for each of the plurality of estimated locations in each of the plurality of images; and controlling the vehicle in the autonomous driving mode based on the selected estimated location.

11. The system of claim 10 , wherein the one or more processors are configured to capture the plurality of images with a camera at a plurality of different poses while the vehicle is moving.

12. 11. The system of claim 10, wherein the one or more processors are configured to capture the plurality of images when the vehicle is at a predetermined distance from the object defined in pre-stored map information.

13. 11. The system of claim 10, wherein the one or more processors are configured to identify the object in each of the plurality of images by projecting a region of an image patch corresponding to the object from one of the plurality of images onto another of the plurality of images.

14. The one or more processors: identifying a patch in a first image of the plurality of images that corresponds to the object; and determining the error score based on the patch.

15. The system of claim 10 , wherein selecting the estimated location comprises selecting the estimated location having the lowest one of the error scores.

16. The one or more processors: identifying a patch corresponding to the object in a first image of the plurality of images; projecting an estimated location of the plurality of estimated locations onto the first image; and determining a distance between the projected location and an area of ​​the patch.

17. The system of claim 16 , wherein determining the distance comprises determining a distance between a center of the projected location and a center of the area of ​​the patch.

18. The one or more processors: identifying a first patch corresponding to the object in a first image of the plurality of images; identifying a second patch corresponding to the object in a second image of the plurality of images; projecting the center of the first patch onto the second image; and determining whether the projected center is within an area of ​​the second patch.

Images