Emergency Vehicle Detection

Abstract

To detect an emergency vehicle.SOLUTION: For example, a plurality of images may be taken from a view point of an autonomous type vehicle. One or more gates representing regions of interest, which are located at respective distances from the vehicle, may be generated for images. A plurality of light beams may be detected within the one or more gates. A first candidate emergency vehicle may be identified from the detected light beams at one or more gates of one image of the images, and a second candidate emergency vehicle may be identified from the plurality of light beams detected at one or more gates of another image of the images. It is determined that the first and second candidate emergency vehicles are the same emergency vehicle and are active. An operational system of the autonomous type vehicle is controlled based on the determination that a given emergency vehicle is active.SELECTED DRAWING: Figure 11

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 16 / 682,747, filed November 13, 2019, which claims the benefit of the filing date of U.S. Provisional Patent Application No. 62 / 879,636, filed July 29, 2019, the disclosures of which are incorporated herein by reference in their entireties. [Background technology]

[0002] Autonomous vehicles, such as vehicles that do not require a human driver, may be used to assist in the transportation of passengers or goods from one location to another. Such vehicles may be operated in a fully autonomous driving mode, where the passenger provides some initial input, such as a destination, and the vehicle steers itself to that destination. Such vehicles may therefore be highly dependent on systems that can determine the location of the autonomous vehicle at all times, and detect and identify objects external to the vehicle, such as other vehicles, stop signals, pedestrians, etc. Summary of the Invention

[0003] An aspect of the disclosure provides a system for detecting emergency vehicles. The system includes an operation system of a vehicle and one or more computing devices in communication with the operation system. The one or more computing devices are configured to: receive a plurality of images of an environment from a viewpoint of the vehicle, the plurality of images including a first image and an additional image; generate one or more gates in the plurality of images, each of the one or more gates representing an area of ​​interest at a respective distance from the vehicle; detect a plurality of lights in the one or more gates; identify a first candidate emergency vehicle corresponding to a group of the detected plurality of lights in a given one of the one or more gates of the first image; identify a second candidate emergency vehicle corresponding to a group of the detected plurality of lights in the one or more gates of the additional image; determine that the first candidate emergency vehicle and the second candidate emergency vehicle are the same given emergency vehicle; determine that the given emergency vehicle is active; and operate the operation system of the vehicle based on a determination that the given emergency vehicle is active.

[0004] In one example, the one or more computing devices are further configured to select one or more regions for generating one or more gates. In another example, the one or more computing devices are further configured to mask pixels in the first image associated with areas in the environment that are not related to emergency vehicles. In another example, the one or more computing devices are configured to identify a first candidate emergency vehicle based on a first candidate vehicle polygon generated to encompass a group of the detected lights in a given gate in the first image. In this example, the one or more computing devices are configured to determine that the first candidate emergency vehicle and the second candidate emergency vehicle are the same given emergency vehicle based on a second candidate vehicle polygon generated to encompass the group of the detected lights and the given gate in the additional image, and a measure of similarity between the first candidate vehicle polygon and the second candidate vehicle polygon. Additionally or alternatively, the one or more computing devices are further configured to identify characteristics of an active emergency vehicle based on the first candidate vehicle polygon. In another example, the one or more computing devices are configured to determine that the given emergency vehicle is active based on validation of the first candidate emergency vehicle or the second candidate emergency vehicle. In this example, the validation includes matching characteristics of the active candidate emergency vehicle with other detected objects. Additionally or alternatively, the validation includes filtering false positives of the detected objects in the one or more gates. In another example, the one or more computing devices are further configured to identify characteristics of the active emergency vehicle based on the one or more gates of the first image or the one or more gates of the additional image. In another example, the vehicle operation system is a navigation system for planning a trajectory of the vehicle. In another example, the vehicle operation system is a deceleration system for decelerating and stopping the vehicle. In another example, the vehicle operation system is a steering system for controlling the angle of the wheels to turn the vehicle.In another example, the one or more processors are configured to detect the plurality of lights by identifying light within a particular frequency range. In another example, the one or more processors are configured to detect the plurality of lights by filtering out light outside of the particular frequency range.

[0005] Another aspect of the disclosure provides a method for detecting emergency vehicles, the method including: receiving, by one or more computing devices, a plurality of images taken from a perspective of an autonomous vehicle, the plurality of images including a first image and an additional image; generating, by one or more computing devices, one or more gates in the plurality of images, each of the one or more gates representing an area of ​​interest at a respective distance from the vehicle; detecting, by one or more computing devices, a plurality of lights in the one or more gates; identifying, by one or more computing devices, a first candidate emergency vehicle corresponding to a group of the detected plurality of lights in the one or more gates of the first image; identifying, by one or more computing devices, a second candidate emergency vehicle corresponding to a group of the detected plurality of lights in a given one of the one or more gates of the additional image; determining, by one or more computing devices, that the first candidate emergency vehicle and the second candidate emergency vehicle are the same given emergency vehicle; determining, by one or more computing devices, that the given emergency vehicle is active; and operating, by one or more computing devices, an operational system of the vehicle based on a determination that the given emergency vehicle is active.

[0006] In one example, the method also includes selecting, by the one or more computing devices, one or more regions for generating one or more gates. In another example, the method further includes masking, by the one or more computing devices, pixels in the first image associated with objects not related to the emergency vehicle. In another example, identifying the first candidate emergency vehicle includes generating a first candidate vehicle polygon to encompass the detected group of lights and the given gate in the first image. In this example, determining that the first candidate emergency vehicle and the second candidate emergency vehicle are the same given emergency vehicle includes generating a second candidate vehicle polygon to encompass the detected group of lights and the given gate in the additional image, projecting the second candidate vehicle polygon onto the first image, determining a measure of similarity between the first candidate vehicle polygon and the second candidate vehicle polygon, and determining that the first candidate emergency vehicle and the second candidate emergency vehicle are the same given emergency vehicle if the amount of similarity is greater than a threshold amount. In another example, determining that a given emergency vehicle is active includes validating a first candidate emergency vehicle or a second candidate emergency vehicle. In another example, identifying a characteristic of an active emergency vehicle based on one or more gates of the first image or one or more gates of the additional image. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a functional diagram of an exemplary vehicle, according to aspects of the present disclosure. [Diagram 2] FIG. 1 is an exemplary depiction of a vehicle according to aspects of the present disclosure. [Figure 3A] 3 illustrates an exemplary image 300 according to an embodiment of the present disclosure. [Figure 3B] 3 illustrates an exemplary image 300 according to an embodiment of the present disclosure. [Figure 3C] 3 illustrates an exemplary image 300 according to an embodiment of the present disclosure. [Figure 4A] 4 illustrates an exemplary image 400 according to an embodiment of the present disclosure. [Figure 4B]4 illustrates an exemplary image 400 according to an embodiment of the present disclosure. [Figure 4C] 4 illustrates an exemplary image 400 according to an embodiment of the present disclosure. [Diagram 5] 5 illustrates an exemplary output image 500, according to an embodiment of the present disclosure. [Figure 6A] 6 illustrates a cropped image portion 600 according to an embodiment of the present disclosure. [Figure 6B] 6 illustrates a cropped image portion 600 according to an embodiment of the present disclosure. [Figure 7] 4 illustrates an exemplary image 400 according to another aspect of the present disclosure. [Figure 8] 1 shows a series of images according to an aspect of the present disclosure. [Figure 9] 4 illustrates an exemplary image 400 according to a further aspect of the present disclosure. [Figure 10] 13 illustrates another series of images according to an aspect of the present disclosure. [Figure 11] 11 is a flow diagram 1100 of an exemplary method according to an aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] overview The present technology relates to long-range detection of emergency vehicles for autonomous vehicles. Being able to detect emergency vehicles early while they are far away allows the autonomous vehicle to adjust its path or behavior earlier or in a more accurate and efficient manner. Emergency vehicles include, for example, police cars, ambulances, and fire engines, among others. The present approach may include selecting one or more regions in a first image based on road geometry to restrict an image search space to locations where oncoming vehicles are present or likely to be present. Since flashing red and blue lights are often associated with active emergency vehicles, candidate boxes may be determined within the selected region according to where flashing red and blue lights are detected in the region.

[0009] The autonomous vehicle may include an image capture system configured to capture one or more images in a forward direction relative to the autonomous vehicle, including a first image. The vehicle's computing device may be configured to select one or more regions in the first image according to a road graph geometry. The selected one or more regions may be areas where an oncoming lane exists or is likely to exist. The road graph geometry may be determined by detecting lane features in the first image. The lane features may be used to identify lanes in the same direction as the autonomous vehicle's lane of travel and lanes in an opposite direction to the autonomous vehicle's lane of travel.

[0010] The vehicle's computing device may then generate one or more gates in the selected one or more regions. Each gate may represent an area of ​​interest at a particular distance from the autonomous vehicle. The one or more gates may be generated starting from where the range of the autonomous vehicle's short-range sensor ends. The one or more gates may end at a distance where the opposing lanes are no longer perceptible or where a nominal emergency vehicle is smaller than a first threshold amount of pixels in the first image. After the one or more gates are generated, the vehicle's computing device may determine whether any red or blue lights are within the one or more gates.

[0011] The vehicle's computing device may then identify candidate emergency vehicles corresponding to the group of red and / or blue lights at a given gate of the one or more gates. The group of red or blue lights may be multiple red or blue lights in the given gate that are close to each other. Identifying the candidate emergency vehicles may include generating a first candidate vehicle polygon according to the group of red or blue lights and the given gate. Identifying the candidate emergency vehicles may also include determining whether the red or blue lights are in one or more additional images taken before and / or after the first image. It may include using the one or more additional images to capture the flashing lights. The additional candidate vehicle polygons may be generated according to the red and / or blue lights for each of the one or more additional images.

[0012] A first candidate vehicle polygon in the first image may be associated with at least one additional candidate vehicle polygon in at least one of the one or more additional images. The association between the candidate vehicle polygons allows the autonomous vehicle to determine that the associated candidate vehicle polygons correspond to the same candidate emergency vehicle and also allows the autonomous vehicle to track the behavior of the candidate emergency vehicle over time. In addition, the association may capture the flashing lights of the candidate emergency vehicle over time. The association may be generated by projecting at least one additional candidate vehicle polygon onto the first image.

[0013] Projecting the at least one additional candidate vehicle polygon onto the first image may include aligning the at least one additional candidate vehicle polygon in the first image based on (i) a size and position of the at least one additional candidate vehicle polygon and (ii) a relative motion of the autonomous vehicle between the first image and the corresponding at least one additional image. A given additional image of the at least one additional image may be associated with the first image if the measure of similarity exceeds a second threshold amount. The measure of similarity may be determined based on an amount of physical intersection between the first candidate vehicle polygon and the given additional candidate vehicle polygon of the at least one additional candidate vehicle polygon projected onto the first image.

[0014] After associating the first candidate vehicle polygon with at least one additional candidate vehicle polygon, the vehicle's computing device may determine that the candidate emergency vehicle is an active emergency vehicle. This determination may include comparing red or blue lights in the first candidate vehicle polygon with red or blue lights in the at least one additional candidate vehicle polygon. A difference in the number of lights, the intensity of the lights, or the position of the lights may be an indication that the candidate emergency vehicle has a flashing light.

[0015] In some implementations, determining that the candidate emergency vehicle is an active emergency vehicle includes verifying the active emergency vehicle by removing possible false positives or by matching characteristics of the active candidate emergency vehicle with other detected objects. To remove false positives, the vehicle's computing device may filter out objects detected in a given image based on features that are not characteristic of an active emergency vehicle or low reliability indicators of an active emergency vehicle. Another example of removing false positives includes comparing candidate emergency vehicles in a given image with objects detected by a perception system of the autonomous vehicle.

[0016] After determining that the candidate emergency vehicle is an active emergency vehicle, the vehicle's computing device may operate the autonomous vehicle according to characteristics of the active emergency vehicle. The characteristics may include the location, movement, speed, or predicted route of the active emergency vehicle. Operating the autonomous vehicle may include determining a route for the autonomous vehicle to yield to the active emergency vehicle, controlling the autonomous vehicle's systems to slow down to a future stop, or other actions to appropriately respond to the active emergency vehicle.

[0017] The above features may provide an autonomous vehicle that is able to identify emergency vehicles, which may vary widely in appearance, earlier, at greater distances, and using less data than other machine learning approaches, so that the autonomous vehicle may begin preparing to respond to the emergency vehicle earlier. Additionally, the described features also provide a way to identify false positives that allows the autonomous vehicle to accurately respond to objects in its environment.

[0018] Exemplary System 1, a vehicle 100 according to an embodiment of the present disclosure includes various components. Although some embodiments of the present disclosure are particularly useful in connection with certain types of vehicles, the vehicle may be any type of vehicle, including, but not limited to, automobiles, trucks, motorcycles, buses, recreational vehicles, etc. The vehicle may have one or more computing devices, such as computing device 110, which includes one or more processors 120, memory 130, and other components typically present in a general-purpose computing device.

[0019] The one or more processors 120 may be any conventional processor, such as a commercially available CPU, or alternatively, the one or more processors may be dedicated devices, such as an ASIC or other hardware-based processor.

[0020] Memory 130 stores information accessible by one or more processors 120, including instructions 132 and data 134 that may be executed or otherwise used by processor 120. Memory 130 may be any type of memory capable of storing information accessible by a processor, including computing device readable media or other media that store data that can be read using electronic devices such as hard drives, memory cards, ROM, RAM, DVDs or other optical disks, and other writable and read-only memories. Systems and methods may include different combinations of the above, whereby different portions of the instructions and data are stored on different types of media.

[0021] The instructions 132 may be any set of instructions executed directly (such as machine code) or indirectly (such as script) by a processor. For example, the instructions may be stored as computing device code on a computing device readable medium. In that regard, the terms "instructions" and "program" may be used interchangeably herein. The instructions may be stored in object code format for direct processing by a processor, or in any other computing device language, including a collection of scripts or independent source code modules that are interpreted on demand or precompiled. The functions, methods, and routines of the instructions are described in further detail below.

[0022] Data 134 may be retrieved, stored, or modified by processor 120 in accordance with instructions 132. For example, although claimed subject matter is not limited to any particular data structure, the data may be stored in a computing device register, in a relational database as a table having multiple different fields and records, an XML document, or a flat file. The data may also be formatted in any computing device readable format.

[0023] 1 functionally depicts the processor, memory, and other elements of computing device 110 as being within the same block, it will be understood by those skilled in the art that a processor, computing device, or memory may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical enclosure. For example, a memory may be a hard drive or other storage medium located in a different enclosure than that of computing device 110. Thus, reference to a processor or computing device should be understood to include reference to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0024] Computing device 110 may include all of the components typically used in connection with a computing device, such as the processor and memory described above, as well as user input devices 150 (e.g., a mouse, keyboard, touch screen, and / or microphone) and various electronic displays (e.g., a monitor having a screen, or any other electrical device operable to display information). In this example, the vehicle includes an interior electronic display 152, as well as one or more speakers 154, to provide information or an audiovisual experience. In this regard, interior electronic display 152 may be located within the cabin of vehicle 100 and may be used by computing device 110 to provide information to passengers within vehicle 100.

[0025] The computing device 110 may also include one or more wireless network connections 156 to facilitate communication with other computing devices, such as client and server computing devices, described in more detail below. The wireless network connections may include short-range communication protocols, such as Bluetooth, Bluetooth Low Energy (LE), cellular connections, as well as a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, a private network using one or more enterprise-proprietary communication protocols, Ethernet, Wi-Fi, and HTTP, and various combinations thereof.

[0026] In one example, the computing device 110 may be an autonomous computing system integrated into the vehicle 100. The autonomous computing system may be capable of communicating with various components of the vehicle to operate the vehicle 100 in a fully autonomous mode and / or a semi-autonomous mode. For example, returning to FIG. 1, the computing device 110 may communicate with various operational systems of the vehicle 100, such as a deceleration system 160, an acceleration system 162, a steering system 164, a signaling system 166, a navigation system 168, a positioning system 170, and a perception system 172, a power system 174 (e.g., a gasoline or diesel motor or an electric engine), in accordance with instructions 132 in memory 130 to control the movement, speed, etc. of the vehicle 100. Again, these systems are shown as being external to the computing device 110, but in practice, these systems may also be integrated into the computing device 110 as an autonomous computing system to control the vehicle 100.

[0027] As an example, computing device 110 may interact with deceleration system 160 and acceleration system 162 to control the speed of the vehicle. Similarly, steering system 164 may be used by computing device 110 to control the direction of vehicle 100. For example, if vehicle 100 is configured for use on a road, such as a car or truck, the steering system may include components to control the angle of the wheels to turn the vehicle. Signaling system 166 may be used by computing device 110 to signal the vehicle's intentions to other drivers or vehicles, for example, by illuminating turn signals or brake lights as needed.

[0028] The navigation system 168 may be used by the computing device 110 to determine and follow a route to a location. For example, the navigation system may function to generate routes between locations and plan a vehicle's trajectory to follow this route. Although depicted as a single system, the navigation system may actually include multiple systems to accomplish the aforementioned routing and planning functions. In this regard, the navigation system 168 and / or the data 134 may store detailed map information, such as high-precision maps that identify roadway shapes and elevations, lane lines, intersections, crosswalks, speed limits, traffic signals, buildings, signs, real-time traffic information, vegetation, or other such objects and information. In other words, this detailed map information may define the geometry of the vehicle's expected environment, including roadways and the speed limits (legal speed limits) of those roadways. In particular, the map information may include a road graph that defines the geometry of roadway features, such as lanes, medians, curbs, crosswalks, etc. By way of example, a road graph may include a plurality of interconnected points and / or line segments that define the geometry (e.g., size, shape, dimensions, and location) of the aforementioned roadway features. The coordinates of such features may be defined in a Euclidean coordinate system such that the geometry includes x, y, and z dimensions that are the lateral, longitudinal, and elevation information of each roadway feature relative to a point on the Earth. Of course, these dimensions may be defined in GPS coordinates or other coordinate systems. A road graph may also include information that identifies how vehicles are expected to travel on a given roadway, including direction (i.e., legal traffic direction for each lane), lane location, speed, etc. For example, this map information may include information regarding traffic regulations such as traffic lights, stop signs, and yield signs. This information, in conjunction with real-time information received from the perception system 172, may be used by the computing device 110 to determine which direction of traffic is oncoming and / or has priority at a given location.

[0029] The perception system 172 also includes one or more components for detecting objects external to the vehicle, such as other vehicles, obstacles in the roadway, traffic signals, signs, trees, etc. For example, the perception system 172 may include one or more image sensors, including visible light cameras, thermal imaging systems, laser and radio frequency detection systems (e.g., LIDAR, RADAR, etc.), sonar devices, microphones, and / or any other detection devices that record data that can be processed by the computing device 110. The visible light camera or another type of image capture system may be configured to capture one or more images in a forward-facing direction relative to the vehicle. The laser detection sensor or other type of sensor may provide a direct distance measurement that may have a range constraint. There may be short-range and long-range sensors in one or more imaging sensors with different range constraints. For example, the range constraint of the short-range sensor may be greater than or equal to 60 meters. The range constraint of the long-range sensor may be greater than the range constraint of the short-range sensor, such as 150 meters. At the point where the range constraint of one or more of the short-range sensors is exceeded, the perception system 172 may have a reduced capacity to gather data.

[0030] One or more imaging sensors of perception system 172 may detect objects and characteristics of the objects, such as location, orientation, size, shape, type, direction of movement and speed. Raw data from the sensors and / or the aforementioned characteristics may be quantified or placed into a describing function or vector and transmitted to computing device 110 for further processing. As an example, computing device 110 may use positioning system 170 to determine the location of a vehicle and use perception system 172 to detect and respond to objects when needed to safely reach that location.

[0031] 2 is an example exterior view of vehicle 100 including an embodiment of perception system 172. For example, housing 210 and dome housing 212 on the roof may include a LIDAR sensor or system, as well as various cameras and radar units. In addition, housing 220 located at the front end of vehicle 100, and housings 230, 232 on the driver's and passenger's sides of the vehicle may each store a LIDAR sensor. For example, housing 230 is located in front of driver's door 260. Vehicle 100 also includes housings 240, 242 for radar units and / or cameras further positioned on the roof of vehicle 100. Additional radar units and cameras (not shown) may be positioned at the front and rear ends of vehicle 100 and / or at other locations along the roof or housing 210 on the roof.

[0032] Exemplary Methods In addition to the operations described above and illustrated in the figures, various operations are described herein. It should be understood that the following operations do not have to be performed in the exact order described below. Rather, various steps may be processed in different orders or simultaneously, and steps may also be added or omitted.

[0033] Using an image capture system in the perception system 172, the vehicle's computing device 110 may capture an image in a forward direction relative to the direction of travel of the vehicle 100. The direction of travel of the vehicle 100, or the pose of the vehicle 100, may be stored in association with the captured image. An exemplary first image 300 is shown in FIG. 3A. Captured in the first image 300 are a number of roadway features including lanes 302, 304, 306, 312, 314, 316, lane lines 308, 310, 318, 320, and a median strip 322. A first lane 302 in which the vehicle 100 is traveling. A second lane 304 and a third lane 306 are shown in the first image 300 for the same direction of travel as the first lane 302. Dashed lane lines 308 and 310 are shown defining each of the lanes 302, 304, and 306 from one another. A fourth lane 312, a fifth lane 314, and a sixth lane 316 are shown for an opposite direction of travel from the first lane 302. Dashed lane boundaries 318 and 320 are shown defining each of the lanes 312, 314, and 316 from one another. The lanes 312, 314, and 316 are shown in the first image 300 as separated from the lanes 302, 304, and 306 by a median strip 322. Other roadway features than those discussed above may also be captured in the first image, such as solid lanes, dual lanes, road signs, traffic lights, etc.

[0034] An exemplary second image 400 captured by the image capture system of the perception system 172 is shown in FIG. 4A. The multiple roadway features captured in the second image 400 include roadway features such as lanes 402, 404, 406, 412, 414, lane lines 408, 410, 416, dual lane lines 420, cross roads 424, and traffic lights 430, 432, 434, 436, among others. The stop lights of traffic lights 430, 432, 434, and 436 are illuminated with a red light frequency, as indicated by the shading in the figure. Other objects captured in the second image 400 also include vehicles 440, 442, 444, 446, 448, 450, 452, 454, 456, 460, and 462. The tail lights of vehicles 440 and 442 are illuminated with red frequency lights, as indicated by the shading in the diagram. Lights on the top of vehicle 454 and on the body of vehicle 454 are illuminated with red frequency lights, as indicated by the shading in the diagram. The lane of travel of vehicle 100 is lane 402. Lanes 404 and 406 are in the same direction of travel as lane 402. Lanes 412 and 414 are in the opposite direction of travel as lane 402. Cross street 424 runs at least approximately perpendicular to the direction of travel as lane 402. Vehicle 440 is in lane 402 and vehicle 442 is in lane 404. Vehicles 444, 446, 450, 452, and 456 are in lane 412 and vehicles 448 and 454 are in lane 414. Vehicles 460 and 462 are in cross street 424.

[0035] The vehicle's computing device 110 may be configured to select one or more areas in the captured image according to the roadway geometry stored in the map information. The selected one or more areas may be areas with oncoming vehicles. The roadway geometry depicted in the captured image may be determined by projecting the geometries of various roadway features, such as lanes, curbs, medians, etc., of a road graph onto the captured image depending on the pose of the vehicle 100 when the image is captured. Alternatively, the roadway geometry depicted in the captured image may be determined by detecting roadway features in the captured image and comparing them to the geometries of roadway features defined in the road graph.

[0036] 4A , given the vehicle's pose when the second image was captured, the vehicle's computing device 110 may determine the geometry of the road depicted in the first image 400 by projecting the geometry of various features of the road graph onto the image and / or by detecting roadway features in the first image (e.g., using various image processing techniques) and comparing those features to the geometry of the roadway features of the road graph. In this regard, the computing device 110 may detect lanes 402, 404, 406, 412, 414, lane lines 408, 410, 416, dual lane lines 420, and other roadway features.

[0037] Based on such information stored in the road graph, the roadway features may be used to identify lanes in the same direction as the lane of travel of the vehicle 100 and lanes in an opposite direction to the lane of travel of the vehicle. The opposite direction lanes may be part of the same road or may be part of a different road. The selected region or regions may be areas of the image in which there are lanes in the opposite direction.

[0038] Returning to FIG. 3A , the vehicle's computing device 110 may determine that the lanes 312, 314, and 316 are in the opposite direction to the lane 302 in which the vehicle 100 is currently traveling, or rather, are oncoming lanes, based on the physical location and type of roadway feature of the median 322 and / or the direction of traffic in the lane 302 (in which the vehicle 100 is currently traveling) compared to the directions of the lanes 312, 314, 316 identified in the road graph. Thus, as shown in FIG. 3B , a region 330 of the exemplary first image 300 may be selected by the vehicle's computing device 110 of the first image 300. The vehicle's computing device 110 may then define the region 330 based on the lanes 312, 314, and 316, lane lines 318 and 320, and the median 322 that are visible in the first image 300, and / or roadway features including information identified in the road graph. As a result, region 330 is selected as the area of ​​first image 300 that includes where lanes 312, 314, and 316 exist.

[0039] Turning to the example of FIG. 4B of the second image 400, a region 470 may be selected by the vehicle's computing device 110. The vehicle's computing device 110 may determine that the lanes 412 and 414 are opposite or oncoming lanes of the vehicle's travel lane 402 based on the location and type of roadway feature of at least the dual lane boundary 420. The vehicle's computing device 110 may then define the region 470 based on the roadway features including the lanes 412, 414 and the lane boundary lines 416, 420. As a result, the region 470 is selected as the area of ​​the second image 400 that includes the area where the lanes 412 and 414 are visible in the second image.

[0040] The vehicle's computing device 110 may then generate one or more gates or areas of interest in the selected area or areas. Each gate may represent an area of ​​interest at a particular distance from the vehicle. Each gate may span at least one or more road widths, including lanes of traffic or oncoming traffic from the vehicle. The height of each gate may be based on an estimated emergency vehicle size, such as the average height of the tallest emergency vehicle. For example, the height of each gate may be 3.5 meters, or more, or less. The height of each gate may be based on the expected maximum height of emergency vehicles in the area. The one or more gates may be generated at a fixed distance, such as every 15 meters, or may be variable depending on the degree of interest at a particular distance. These distances may be determined using detected 3D geometry information or estimated based on the viewpoint of the captured image.

[0041] The one or more gates may be generated starting from where the range of the vehicle's short-range sensor ends, such as above or below 60 meters. The one or more gates may end at a distance where the lane in the opposite direction to the vehicle becomes imperceptible in the image, or at a distance where a nominal emergency vehicle is smaller than a first threshold amount of pixels in the captured image. Alternatively, the one or more gates may end where the range of the vehicle's long-range sensor ends, such as 150 meters. In another alternative, the one or more gates may be generated by the vehicle's computing device 110 or a remote computing device using known map information. The one or more gates may then be generated for a given road at a particular location of the map information. The distance between each gate may be a set road length on the given road.

[0042] For example, as shown in FIG. 3C, the closest first gate 340 in the exemplary first image 300 may be at a distance of 60 meters from the vehicle 100 and span the width or lateral distance of the three lanes 312, 314, 316 contained within the selected region 330. Each subsequent gate, such as gates 342, 344, 346, 348, 350, and 352, may be determined for a distance of 15 meters further away from the previous gate. Each of the gates in the first image 300 of FIG. 3C may be approximately 3 meters in height. From the vantage point in the first image 300, each subsequent gate may appear shorter than the previous gate due to the increasing distance from the vehicle 100. The gates in the first image 300 continue for 150 meters, where the range of the long-range sensor ends.

[0043] Turning to FIG. 4C for an example second image 400, a first gate 480 may be generated as the closest gate, 60 meters away from the vehicle 100, spanning the width or lateral distance of the two lanes 412, 414 encompassed within the selected region 470. Each subsequent gate, such as gates 482, 484, 486, 488, 490, and 492, may be determined for a distance of 15 meters further away from the previous gate. Each of the gates in the second image 400 of FIG. 4C may be approximately 3 meters in height. From the vantage point in the second image 400, each subsequent gate may appear shorter than the previous gate due to the increasing distance from the vehicle 100. The gates in the second image 400 continue until 150 meters, where the range of the long-range sensor ends.

[0044] In some implementations, the vehicle's computing device 110 may also mask pixels associated with objects that are not relevant to the detection of the emergency vehicle. For example, the non-relevant objects may include traffic lights, vehicles traveling in the same direction as the vehicle 100, or other objects with blue and / or red lights or other light frequencies that are typically associated with emergency vehicles, but are not emergency vehicles. This image masking may be performed by classifying the detected objects as non-relevant objects, projecting the detected objects onto the captured image using 3D geometric shape information associated with the detected objects, and identifying pixels covered by the projection of the detected objects as non-relevant, or "masking" the pixels. As shown in FIG. 5, the second image 400 may be filtered to mask pixels associated with objects, such as traffic lights, lanes for traveling in the same direction as the vehicle, and vehicles traveling in the same direction. Specifically, traffic lights 430, 432, 434, 436, lanes 402, 404, 406 for the same direction of travel as vehicle 100, and vehicles 440, 442 traveling in these lanes in the second image 400 may be masked as shown by white portions 502, 504, 506, 508, and 510 in the output image 500.

[0045] After the one or more gates are generated, the vehicle's computing device 110 may determine whether any red or blue light is within the one or more gates. Red light may include light having a frequency in the range of approximately 405-480 THz. Blue light may include light having a frequency in the range of approximately 620-680 THz. Determining whether any red or blue light is within the one or more gates may include filtering the captured image for red or blue light frequencies. Although the example methods described herein discuss red and / or blue light, the methods may also be performed for other frequencies of light that may be associated with emergency vehicles or other types of vehicles that may require a particular response from the vehicle 100.

[0046] 6A illustrates a cropped image portion 600 of second image 400 captured by an image capture device of perception system 172. Objects depicted in cropped image portion 600 include, among others, road features such as lanes 402, 404, 412, 414, lane lines 408, 416, 420, cross road 424, and traffic light 436. Objects depicted in cropped image portion 600 also include vehicles 440, 448 and portions of vehicles 450, 452, 454, 456.

[0047] As shown in FIG. 6B, the cropped image portion 600 may be filtered based on red light frequency to identify red lights, as shown in output image 602. In output image 602, a plurality of red lights 604 on or around vehicle 454 are shown in contrast to the remainder of cropped image portion 600. The plurality of red lights 604 include lights A, B, C, D, and E, where lights A-D appear across the top of vehicle 454 and light E appears on the body of vehicle 454. As shown in FIG. 4B, traffic light 436 has been masked so that the red light of traffic light 436 is not shown in output image 602. Although only cropped image portion 600 is shown, the filtering process may be performed on the entirety of image 400. The same filtering process may be applied to image 400 based on blue light frequency to identify blue lights in the image.

[0048] In some examples, the captured image may be filtered by the vehicle's computing device 110 using a center-around filter to detect where prominent color contrasts exist in one or more gates before or instead of filtering by frequency to identify where colored light may be present. Additionally, the captured image may be filtered based on light energy before or instead of filtering by frequency to identify where bright light may be present. Further, determining whether red or blue light is within one or more gates may additionally or alternatively include identifying or labeling red or blue light within one or more gates. The identifying or labeling may be performed using deep networks or other types of machine learning methods.

[0049] The vehicle's computing device 110 may then identify candidate emergency vehicles corresponding to the group of red and / or blue lights at a given gate of the one or more gates. The group of red and / or blue lights may be multiple red and / or blue lights in the given gate that are in close proximity to each other at or within a threshold distance, such as within one meter or less of each other. Identifying the candidate emergency vehicles may include generating a first candidate vehicle polygon according to the group of red and / or blue lights and the given gate. A vehicle captured or captured more than 50% in the first candidate vehicle polygon may be identified as a candidate emergency vehicle. The size of the first candidate vehicle polygon may be an estimated or average width and height of the emergency vehicle. Alternatively, the width may be the same or similar to the width of the group of red and / or blue lights, or the height may be the same or similar to the height of the given gate. In addition, the size of the first candidate vehicle polygon generated in the captured image may correspond to the distance or dimension of a given gate, such as being the same or similar distance from the vehicle, or having the same or similar height as the given gate. The location of the first candidate vehicle polygon may also correspond to the location of a group of red or blue lights. That is, the first candidate vehicle polygon may be positioned such that the first candidate vehicle polygon encompasses a group of red and / or blue lights. Additional candidate emergency vehicles may be identified in the captured image in the same or similar manner.

[0050] As shown in FIG. 7, a first candidate vehicle polygon 702 (depicted herein as a box or rectangle, although other polygons with more or less vertices may be used) may be generated based on the group of red lights identified in the output image 602. Red lights A, B, C, and D shown in FIG. 6B may be identified as a group by the vehicle computing device 110 based on each light being within one meter of at least one other light of the plurality of red lights 604. Red light E may not be identified as part of the group because it is more than one meter from each of lights A, B, C, and D. The first candidate vehicle polygon 702 has a width that encompasses the width of the group of red lights A-D and has a height that is the same as the height of gate 486. Vehicle 454 is completely captured in the first candidate vehicle polygon 702 and is therefore identified as a candidate emergency vehicle.

[0051] Identifying the candidate emergency vehicle may additionally include determining whether additional red and / or blue lights are in one or more additional images taken before and / or after the captured image. It may include using one or more additional images to capture the flashing lights. The one or more additional images may be taken within one second of the captured image. The additional red and / or blue lights may be determined in the one or more additional images in the same or similar manner as described above, including selecting one or more regions and generating one or more gates. The additional red and / or blue lights in the one or more additional images may be used by the vehicle's computing device 110 to (i) generate one or more aggregated output images to generate a more accurate candidate vehicle polygon, and / or (ii) validate the first candidate vehicle polygon.

[0052] In some implementations, one or more aggregated output images are generated using the captured image and one or more additional images to increase the amount of light energy for detection purposes or to increase the location accuracy of the candidate vehicle polygon. Increasing the amount of detectable light energy may result in a higher vehicle detection score for the emergency vehicle. For example, red and / or blue lights from the captured image and light from at least one of the one or more additional images may be combined in the aggregated output image. Multiple aggregated output images may be generated, such as a first aggregated output image generated from a first group of images taken within a first time period, and a second aggregated output image generated from a second group of images taken in a second time period after (or immediately following or partially overlapping with) the first time period. Each of these time periods may be relatively short, for example, on the order of 0.5 seconds or more or less. Groups of red and / or blue lights may be detected in each aggregated output image, and a candidate emergency vehicle box may be generated corresponding to this group in addition to or instead of the first candidate vehicle polygon. Candidate vehicle polygons may be generated for the candidate emergency vehicles in each aggregated output image in the same or similar manner as the first candidate vehicle polygons in the captured image above.

[0053] As shown in FIG. 8 , column 1 shows a perspective view of image portion 600 of exemplary second image 400, column 2 shows a perspective view of image portion 802 of a first additional image taken after image 400, and column 3 shows a perspective view of image portion 804 of a second additional image taken after the first additional image. The capture times of the exemplary second image 400 and the first additional image may be within a tenth of a second, and the capture times of the first additional image and the second additional image may be within a tenth of a second. The image portions 802, 804 of the first and second additional images may show approximately the same geographic area as the image portion 600 of exemplary second image 400. In particular, each of the image portions 600, 802, and 804 may show a vehicle 454. As captured in the image portions, lights on the top of the vehicle 454 and on the body of the vehicle 454 may be illuminated in different patterns.

[0054] Row 1 of FIG. 8 shows the raw input of each image portion, and row 2 shows the filtered red light output of each image obtained as described with respect to FIGS. 6A and 6B. Row 3 shows the cumulative light output when the filtered red light outputs are added together. The output image in column 1 of row 3 is the same as the output image in column 1 of row 2. The output image in column 2 of row 3 is an aggregated output image showing the sum of the light outputs from the output images in columns 1 and 2 of row 2. The output image in column 3 of row 3 is an aggregated output image showing the sum of the light outputs from the output images in columns 1, 2, and 3 of row 2. The red light output from the aggregated output image in column 3 of row 3 can be used to generate a first candidate vehicle polygon 702 as described above, instead of output image 602.

[0055] In some cases, each candidate vehicle polygon may be verified or rather compared to another candidate vehicle polygon to confirm that the candidate vehicle polygon is an active emergency vehicle. To verify the first candidate vehicle polygon, additional candidate vehicle polygons may be generated according to additional red and / or blue lights for each of the one or more additional images. The first candidate vehicle polygon of the captured image may be verified if at least one additional candidate vehicle polygon in at least one of the one or more additional images corresponds to the same candidate emergency vehicle as the first candidate vehicle polygon. If the first candidate vehicle polygon and the additional candidate vehicle polygon correspond to the same candidate emergency vehicle, an association between the candidate vehicle polygons may be created by the vehicle's computing device 110. The vehicle's computing device 110 may then use the associated candidate vehicle polygon to track the behavior of the candidate emergency vehicle over time. The tracked behavior may include, for example, flashing lights, changes in direction of travel, movement, etc.

[0056] The verification may be performed by projecting at least one additional candidate vehicle polygon onto the captured image. Projecting the at least one additional candidate vehicle polygon onto the captured image may include aligning the at least one additional candidate vehicle polygon in the captured image based on (i) a size and a position of the at least one additional candidate vehicle polygon, and (ii) a relative motion of the vehicle between the captured image and the corresponding at least one additional image. If the measure of similarity exceeds a second threshold amount, the given additional image of the at least one additional image may be associated with the captured image. The measure of similarity may be determined based on an amount of physical intersection between the first candidate vehicle polygon and the given additional candidate vehicle polygon of the at least one additional candidate vehicle polygon projected onto the captured image. For example, the measure of similarity may be determined using a Jaccard coefficient or an intersection over union for the first candidate vehicle polygon and the given additional candidate vehicle polygon.

[0057] As shown in FIG. 9, the exemplary second image 400 has an additional candidate vehicle polygon 902 projected onto the second image in addition to the candidate vehicle polygon 702 generated with respect to FIG. 7. The additional candidate vehicle polygon 902 may have been generated for a previously captured image by the image capture system of the perception system 172. The previously captured image may have been captured within a tenth of a second of the second image 400. Alternatively, other candidate vehicle polygons from images captured either one second before or after the second image 400 may be used. The vehicle's computing device 110 may use a Jaccard coefficient to determine a measure of similarity between the candidate vehicle polygons 702 and 902. Given that a large portion of the candidate vehicle polygon 902 overlaps with the candidate vehicle polygon 702, the measure of similarity between the candidate vehicle polygons 702 and 902 is high. The vehicle's computing device 110 may determine that the measure of similarity is greater than a second threshold amount and validate the candidate vehicle polygon 702. The vehicle's computing device 110 may also create an association between the candidate vehicle polygon 702 and the candidate vehicle polygon 902 for tracking purposes.

[0058] After validating the first candidate vehicle polygon with at least one additional candidate vehicle polygon, the vehicle computing device 110 may determine that the candidate emergency vehicle is an active emergency vehicle. In some examples, if the candidate emergency vehicle has flashing lights, the candidate emergency vehicle may be determined to be active. This determination may include comparing red and / or blue lights in the first candidate vehicle polygon with red and / or blue lights in the at least one additional candidate vehicle polygon. A difference in the number of lights, the intensity of the lights, or the position of the lights may be an indication that the candidate emergency vehicle has flashing lights.

[0059] For example, the first candidate vehicle polygon and the at least one additional candidate vehicle polygon may be aligned based on features of the first candidate vehicle polygon and the at least one additional candidate vehicle polygon, such as the location of anchor lights or other portions of the candidate emergency vehicle. A measure of similarity between the light energies of the first candidate vehicle polygon and the at least one additional candidate vehicle polygon may be determined, such as by taking a sum of absolute differences between the candidate vehicle polygons or by comparing a maximum value of the total light energies of each candidate vehicle polygon. If the measure of similarity is less than a third threshold amount, the candidate emergency vehicle may be determined to have flashing lights and thus be active.

[0060] FIG. 10 illustrates a series of images depicting candidate emergency vehicles according to an embodiment of the disclosure. Row 1, column 1 illustrates a portion of an exemplary second image 400 showing a candidate emergency vehicle 454 in a first candidate vehicle polygon 702. Row 1, column 2 illustrates a portion of a previously captured image showing a candidate emergency vehicle 454 in an additional candidate vehicle polygon 902. The lights on the top and body of the candidate emergency vehicle 454 in the additional candidate vehicle polygon 902 may be illuminated in a different pattern than in column 1, as indicated by the shading. A filtered output 1002 of the exemplary portion of the second image and a filtered output 1012 of the previously captured image are illustrated in row 2, columns 1 and 2, respectively. The filtered output may be obtained from a filtered output image. For example, the filtered output 1002 may be obtained from the output image 602. The vehicle's computing device 110 may compare the filtered outputs 1002 and 1012 and determine that a measure of similarity between the total light energy of the filtered output 1002 and the total light energy of the filtered output 1012 is below a third threshold amount. Thus, the vehicle's computing device 110 may determine that the candidate emergency vehicle 454 is an active candidate emergency vehicle.

[0061] In some implementations, determining that a candidate emergency vehicle is an active emergency vehicle may also validate the candidate emergency vehicle. Validation may include filtering out possible false positives, such as matching characteristics of the active candidate emergency vehicle with characteristics of other detected objects and / or using data from sensors other than the camera to confirm detection of an object at the location of the candidate emergency vehicle. To filter out false positives, the vehicle's computing device 110 may filter out objects detected in a given image based on characteristics of those detected objects detected by other systems of the vehicle (e.g., different sensors and / or processing systems of the perception system) that are not characteristic of an active emergency vehicle, or unreliable indicators of an active emergency vehicle. Characteristics may include when another detected object is (i) stationary, (ii) other than a vehicle, (iii) an unidentifiable object moving less than 1 meter / second, (iv) within a minimum distance from the vehicle, (v) clipped at the edge of the image, or (vi) has a heading greater than 60 degrees from a heading directly opposite the vehicle.

[0062] The vehicle's computing device 110 may then discard the candidate emergency vehicle as a false positive if there is a detected object used to validate the candidate emergency vehicle that is not characteristic of an active emergency vehicle. For example, if the detected object used to validate the candidate emergency vehicle has a non-oncoming heading relative to the vehicle, the candidate emergency vehicle may be discarded as a false positive. For example, the heading may be determined to be non-oncoming if it is more than 60 degrees from a directly oncoming heading. The heading is used to filter out non-oncoming vehicles, particularly non-oncoming vehicles on curved roads where the tail lights of the non-oncoming vehicle may be captured by the detection system in one or more gates. Filtering out detected objects may also include projecting the location of the detected object that is filtered out in the previously captured image onto the more recently captured image and filtering out the object in the more recently captured image that is in the projected location.

[0063] As another exemplary validation, detection of objects at the location of a candidate emergency vehicle may be confirmed using data from sensors other than cameras, etc. For example, data from a radar sensor, even if noisy, may be used to confirm the distance and location to the object, which may provide confirmation of objects beyond the perception range of other sensors, such as a LIDAR sensor.

[0064] In other examples, validating the candidate emergency vehicle may include determining a validation score for the detected object. For each additional information or signal that indicates that the candidate emergency vehicle is an active emergency vehicle, such as any of the examples above, the validation score for the candidate emergency vehicle may be increased. Similarly, for each additional information or signal that indicates that the candidate emergency vehicle is not an active emergency vehicle (e.g., any of the false positive features of the examples above), the validation score for the candidate emergency vehicle may be decreased. As an example of this, in some cases, an orange flag fluttering in the wind may appear as a flashing light, tail lights of other vehicles that appear to be flashing because they are sporadically obstructed, etc.

[0065] This validation score of the candidate emergency vehicle may be compared to one or more thresholds. For example, the candidate emergency vehicle may be "revealed" only if the validation score reaches a certain minimum value, or may be used by various systems of the vehicle 100, such as the navigation system 168, to determine how to control the vehicle. Additionally or alternatively, the score may be used by the navigation system 168 to determine whether to simply ignore the candidate emergency vehicle. For example, depending on the location relative to some map feature, the distance from the vehicle 100, the speed, and the predicted future trajectory, even a relatively high validation score may be ignored. At the same time, given these same considerations, a relatively low validation score may also be treated as something that the vehicle 100 should respond to. For example, even if the candidate emergency vehicle has a relatively high validation score, it may not be as important to respond to the candidate emergency vehicle when considering its location in the median strip, as in the example of the first image 300 (e.g., the candidate emergency vehicle is unlikely to enter lane 302 or otherwise directly interact with vehicle 100), but it may be more important to respond to a candidate emergency vehicle that is close to an intersection, even if the candidate emergency vehicle has a relatively low validation score, when considering the possibility that vehicle 100 may need to interact with candidate emergency vehicle 454, as in the example of the second image 400.

[0066] In some cases, features of objects detected by the perception system may be estimated by the vehicle's computing device 110 and assigned to or used to characterize a candidate emergency vehicle. Specifically, the location of the object may be detected using a noisy sensor of the perception system, such as a radar sensor, which is more reliable in the vertical dimension than in the horizontal dimension. The location may be estimated by calculating an error, such as a three-dimensional error ellipse. The error ellipse of a given detected object may be determined based on the smooth coordinates and corresponding covariance estimates of the given detected object. The estimated location may be projected onto a given image to generate a bounding box. The vehicle's computing device 110 may then determine whether the center of the candidate emergency vehicle is within the bounding box of the given detected object. If the center is within the bounding box, it may increase the score of the given detected object. In some implementations, the score may be a number that increases by one if the center is within the bounding box. Additionally, if the candidate emergency vehicle is not within the bounding box of the detected object, the number may be decreased or reset. In other examples, a more general test of whether the center falls within the bounding box may be used, which takes into account the noise characteristics of the alternative sensor used to generate the measurements. The detected object with the highest score for the candidate emergency vehicle may be used to characterize the candidate emergency vehicle. For example, the distance, speed, heading, or other determined or detected characteristic of the detected object with the highest validation score may be assigned to the candidate emergency vehicle.

[0067] After determining that a candidate emergency vehicle is an active emergency vehicle, the vehicle's computing device 110 may operate the vehicle according to the characteristics of the active emergency vehicle. The characteristics may include the location, movement, speed, or predicted route of the active emergency vehicle. The vehicle's computing device 110 may identify the characteristics of the active emergency vehicle based on the characteristics of one or more candidate vehicle polygons or one or more gates corresponding to the vehicle determined to be an active emergency vehicle. For example, the distance and location of the active emergency vehicle 454 may be the same distance and location relative to the vehicle 100 as the distance and location of the candidate vehicle polygon 702 or another current candidate vehicle polygon. Operating the vehicle may include providing these characteristics to the navigation system to enable the navigation system to generate a trajectory that yields to or avoids the active emergency vehicle, controlling the steering system to move away from or around the active emergency vehicle, controlling the deceleration system to decelerate to a future stop, and / or other actions to appropriately respond to the active emergency vehicle.

[0068] While the examples herein relate to detecting emergency vehicles in oncoming lanes, there may be cases where the emergency vehicle is traveling in the same travel lane but in the opposite direction. For example, at the intersection depicted in second image 400, if there is a vehicle between candidate emergency vehicle 454 and the intersection, the driver of the emergency vehicle may decide to enter lane 402 to bypass the other vehicle. In this regard, similar features can be used to detect candidate emergency vehicles in non-oncoming lanes (e.g., those traveling in the same general direction as the lane in which vehicle 100 is currently traveling). However, in such situations, false positives may increase significantly since it may be difficult to distinguish the tail lights of other vehicles.

[0069] 11 illustrates an exemplary flow diagram according to an aspect of the present disclosure. More specifically, FIG. 11 illustrates an exemplary method for detecting an emergency vehicle performed by a vehicle computing device 110. Alternatively, one or more steps in the exemplary method may be performed by one or more computing devices remote from the vehicle 100.

[0070] At block 1102, a plurality of images of an environment from a viewpoint of the vehicle are received. The plurality of images may include a first image, such as image 400 shown in FIG. 4A, and additional images. An image capture system of the autonomous vehicle may be used to capture the first image and the additional images. At block 1104, one or more gates are generated in the plurality of images. Each of the one or more gates represents an area of ​​interest at a respective distance from the vehicle. For example, for image 400, gates 480, 482, 484, 486, 488, 490, and 492 may be generated as shown in FIG. 4C. Each of the one or more gates represents an area of ​​interest at a given distance from the viewpoint of each image of the plurality of images. At block 1106, a plurality of lights are detected in the one or more gates. For example, red light and / or blue light may be detected in the one or more gates. For example, red lights A-E may be detected in image 400 as shown in FIGS. 6A-6B.

[0071] At block 1108, a first candidate emergency vehicle corresponding to the detected group of lights is identified at a given one of the one or more gates of the first image. In image 400, vehicle 454 may be identified as a candidate emergency vehicle. At block 1110, a second candidate emergency vehicle corresponding to the detected group of lights is identified at a gate of the one or more gates of the additional image. The second candidate emergency vehicle may be identified using the same method as the first candidate emergency vehicle. At block 1112, the first candidate emergency vehicle and the second candidate vehicle may be determined to be the same given emergency vehicle. For example, vehicle 454 in image 400 may be determined to be a candidate emergency vehicle identified in image 400 as well as a previously captured image. At block 1114, the given emergency vehicle may be determined to be active. At block 1116, an operation system of the autonomous vehicle may be operated based on a determination that the given emergency vehicle is active. For example, operational systems may be operated to yield to, or otherwise respond appropriately to, the movement of an active emergency vehicle in accordance with traffic laws and other best practices for driving.

[0072] The above features may provide an autonomous vehicle that is able to identify emergency vehicles, which may vary widely in appearance, earlier, at greater distances, and using less data than other machine learning approaches, so that the autonomous vehicle may begin preparing to respond to the emergency vehicle earlier. Additionally, the described features also provide a way to identify false positives that allows the autonomous vehicle to accurately respond to objects in its environment.

[0073] Unless otherwise stated, the alternative examples described above are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. These and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, and therefore the foregoing description of the embodiments should be considered as illustrative, and not limiting, of the subject matter defined by the claims. In addition, the examples described herein, as well as the presentation of phrases such as "including," "including," and the like, should not be construed as limiting the subject matter of the claims to any particular example, but rather, the example is intended to illustrate only one of many possible embodiments. Furthermore, the same reference symbols in different drawings may identify the same or similar elements.

Claims

1. 1. A system for detecting emergency vehicles, comprising: one or more computing devices in communication with an operational system of the vehicle, the one or more computing devices comprising: receiving an image of an environment from a perspective of the vehicle; generating a plurality of gates in the image, each of the plurality of gates representing an area of ​​interest at a respective distance from the vehicle; detecting a plurality of lights in at least one of the plurality of gates; identifying an emergency vehicle within the received image based on the detected lights and the at least one gate of the plurality of gates; verifying that the identified emergency vehicle is active; and and operating the operational system of the vehicle based on the identified emergency vehicle.

2. 2. The system of claim 1, wherein the one or more computing devices are configured to identify the emergency vehicle in the received image further based on one or more masked pixels of the received image.

3. 3. The system of claim 2, wherein the one or more computing devices are further configured to identify one or more pixels to mask in the received image based on 3D geometric information of the environment.

4. the 3D geometric information is associated with detected objects in the environment; and The one or more computing devices include projecting the 3D geometric information onto the received image; and identifying pixels in the received image that are covered by the projection; The system of claim 3 , further configured to: mask the covered pixels.

5. The system of claim 1 , wherein the one or more computing devices are further configured to remove one or more objects in the image before identifying the emergency vehicle.

6. 6. The system of claim 5, wherein the one or more computing devices are configured to remove the one or more objects based on characteristics of the one or more objects that are not characteristics of an active emergency vehicle.

7. The system of claim 5 , wherein the one or more computing devices are configured to remove the one or more objects based on a direction of travel of the one or more objects.

8. The system of claim 1 , wherein the one or more computing devices are further configured to select one or more regions within the image for generating the plurality of gates.

9. The system of claim 1 , wherein the one or more computing devices are configured to verify that the emergency vehicle is active based on an additional image of the environment.

10. 2. The system of claim 1, wherein the one or more computing devices are configured to verify that the emergency vehicle is active based on characteristics of the identified emergency vehicle and characteristics of other detected objects in the environment.

11. The system of claim 10 , wherein the one or more computing devices are further configured to identify characteristics of the emergency vehicle based on the plurality of gates in the image.

12. A system for detecting emergency vehicles, comprising: one or more computing devices in communication with an operational system of the vehicle, the one or more computing devices comprising: receiving an image of an environment from a perspective of the vehicle; generating a plurality of gates in the image, each of the plurality of gates representing an area of ​​interest at a respective distance from the vehicle and having a height based on a maximum expected height of an emergency vehicle in an area; detecting a plurality of lights in at least one of the plurality of gates; identifying an emergency vehicle within the received image based on the detected lights and the at least one gate of the plurality of gates; and operating the operational system of the vehicle based on the identified emergency vehicle.

13. The system of claim 1 , wherein each gate of the plurality of gates spans a width corresponding to one or more lanes contained within the region of interest.

14. A system for detecting emergency vehicles, comprising: one or more computing devices in communication with an operational system of the vehicle, the one or more computing devices comprising: receiving an image of an environment from a perspective of the vehicle; generating a plurality of gates in the image, each of the plurality of gates representing an area of ​​interest at a respective distance from the vehicle and separated from each subsequent gate of the plurality of gates by a predetermined distance; detecting a plurality of lights in at least one of the plurality of gates; identifying an emergency vehicle within the received image based on the detected lights and the at least one gate of the plurality of gates; and operating the operational system of the vehicle based on the identified emergency vehicle.

15. The system of claim 1 further comprising the vehicle.

16. 1. A method for detecting an emergency vehicle, comprising: receiving, by one or more computing devices, an image of an environment taken from a perspective of an autonomous vehicle; generating, by the one or more computing devices, a plurality of gates in the image, each of the plurality of gates representing an area of ​​interest at a respective distance from the autonomous vehicle; detecting, by the one or more computing devices, a plurality of lights within at least one gate of the plurality of gates; identifying, by the one or more computing devices, an emergency vehicle within the received image based on the plurality of lights detected within the at least one gate of the plurality of gates; verifying, by the one or more computing devices, that the identified emergency vehicle is active; and and operating, by the one or more computing devices, an operational system of the autonomous vehicle based on the identified emergency vehicle.

17. The method of claim 16 , wherein the identifying the emergency vehicle in the received image comprises identifying one or more masked pixels of the received image.

18. The method of claim 17 , wherein the identifying of the one or more masked pixels of the received image is based on 3D geometric information of the environment.

19. the 3D geometric information is associated with detected objects in the environment; and The identifying of the one or more masked pixels includes: projecting the 3D geometric information onto the received image; and identifying pixels in the received image that are covered by the projection; and masking the covered pixels.

Images