Use the presence of road surface and surrounding area lighting to detect occluded objects

Abstract

The technology involves detecting an object based on the illumination emitted from a nearby occluded object. The illumination of one or more areas in the surrounding area by the occluded object, such as the headlights of the occluded vehicle, is detected by the perception system of the autonomous vehicle. The autonomous vehicle is able to classify the detected objects to determine whether the illumination is caused by the vehicle or other road users, or by objects in the surrounding environment. The illumination data and other information can be evaluated by the autonomous vehicle, for example, to identify the type of object, the location of the object along the road, to disambiguate the direction of travel of other objects, etc. As a result, the autonomous vehicle can infer the behavior of other objects and modify its own driving operations to account for the presence and possible behavior of other objects.

Description

Background Art

[0001] A self-driving vehicle operating in an autonomous driving mode may transport passengers, cargo, or other items from one location to another. In many cases, it may be difficult to determine the presence of another vehicle or other object along the road, particularly when there is no line of sight to the other object. This may be particularly problematic at night and in other low visibility conditions. The inability of a self-driving vehicle to effectively determine the presence of other road users may limit the vehicle's ability to take certain driving actions or otherwise change its driving behavior. Summary of the invention

[0002] The technology relates to a method for an autonomous vehicle to detect the presence of another vehicle or other road user in the vicinity, even when the other objects are obscured or environmental conditions are poor. Illumination of one or more areas in the surrounding area by other objects is detected by a perception system of the autonomous vehicle, such as by its headlights. The illumination can include reflections off the road, signs, or other objects. The autonomous vehicle can determine whether such illumination is caused by the vehicle or other road user, or from objects in the surrounding environment (e.g., street lights, parking lights, etc.). The lighting information and information about different types of vehicles and other objects can be evaluated by the autonomous vehicle, for example to identify the type of vehicle, the location of the vehicle, or to disambiguate the direction of travel of other vehicles. Thus, the autonomous vehicle can modify its operating (driving) behavior to take into account the presence of other objects.

[0003] According to one aspect, a method for operating a vehicle in an autonomous driving mode is provided. The method includes: obtaining, by one or more sensors of a perception system of the vehicle, lighting sensor data from an external environment surrounding the vehicle; detecting, by one or more processors of the vehicle, the presence of an obscured object in the external environment based on the obtained lighting sensor data; identifying, by the one or more processors, at least one of (i) a type of the object or (ii) one or more characteristics of the object; and, based on at least one of (i) the type of the object or (ii) the one or more characteristics of the object, the one or more processors controlling the operation of the vehicle in the autonomous driving mode.

[0004] In one example, the method further includes one or more processors inferring a behavior of the object based on at least one of (i) a type of the object or (ii) one or more characteristics of the object.

[0005] In another example, detecting or identifying includes distinguishing between multiple lighting sources to determine the presence of one or more road users in the external environment. Here, distinguishing between multiple lighting sources may include identifying at least one fixed lighting source and at least one mobile lighting source. The at least one fixed lighting source may be a street light, a parking light, a sign, or a parked vehicle. The at least one mobile lighting source may be a passenger vehicle, a truck, a motorcycle, a bicycle, or an emergency vehicle.

[0006] In another example, detecting or identifying includes correlating multiple sets of lighting sensor data to determine the source of the light. Here, the multiple sets of lighting sensor data can be obtained from different sensors of the vehicle's perception system. The one or more characteristics of the object can include at least one of location, orientation relative to the road, size, shape, heading, or speed relative to the vehicle.

[0007] In another example, the method further includes: disambiguating the direction of travel of the object before controlling the operation of the vehicle. In another example, detecting the presence of the obscured object in the external environment includes evaluating the obtained lighting sensor data to determine whether the detected amount of light has one or more characteristics. Here, the one or more characteristics may include at least one of a minimum amount of brightness, a wavelength in a particular band, or light emitted by a particular type of lighting device.

[0008] In yet another example, the detection includes comparing the obtained lighting sensor data with a baseline map of ambient lighting of the external environment surrounding the vehicle. In this case, the baseline map may include identifying the location of one or more stationary lighting objects in the external environment.

[0009] According to another aspect, a vehicle configured to operate in an autonomous driving mode is provided. The vehicle includes a driving system including a steering subsystem, an acceleration subsystem, and a deceleration subsystem to control the driving of the vehicle in the autonomous driving mode. It also includes a perception system including one or more sensors configured to detect objects in an environment outside the vehicle. The perception system is configured to obtain lighting sensor data from the external environment. The vehicle also includes a positioning system configured to determine the current position of the vehicle and a control system including one or more processors. The control system is operably coupled to the driving system, the perception system, and the positioning system. The control system is configured to: detect the presence of an obscured object in the external environment based on the obtained lighting sensor data; identify at least one of (i) the type of the object or (ii) one or more characteristics of the object; and control the operation of the vehicle in the autonomous driving mode based on at least one of (i) the type of the object or (ii) one or more characteristics of the object.

[0010] In an example, the control system is further configured to infer the behavior of the object based on at least one of (i) the type of the object or (ii) one or more characteristics of the object. Detection or identification may include distinguishing between multiple lighting sources to determine the presence of one or more road users in the external environment. Detection or identification may alternatively or additionally include correlating multiple sets of lighting sensor data to determine the source of the light.

[0011] In another example, the control system is further configured to disambiguate a direction of travel of the object.And in yet another example, the detecting includes comparing the obtained lighting sensor data to a baseline map of ambient lighting of an external environment surrounding the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1A-1B An example autonomous vehicle is shown in accordance with aspects of the present technology.

[0013] Figure 1C-Figure 1D An example freight vehicle configured for use with aspects of the present technology is shown.

[0014] Figure 2 Components of an autonomous vehicle according to aspects of the present technology are shown.

[0015] Figure 3A-3B is a block diagram of a system of an example freight vehicle in accordance with aspects of the present technology.

[0016] Figure 4A-4G A view of a raised roof sensor assembly is shown in accordance with aspects of the present technique.

[0017] Figure 5A-Figure 5B Exemplary scenarios in accordance with aspects of the present technology are shown.

[0018] Figure 6A-6F A lighting scene in accordance with aspects of the present technique is shown.

[0019] Figure 7A-7B Additional scenarios in accordance with aspects of the present technique are shown.

[0020] Figure 8A-8B Example systems in accordance with aspects of the present technology are shown.

[0021] Fig. 9 Example methods according to aspects of the present technology are shown. DETAILED DESCRIPTION

[0022] Operating a vehicle in an autonomous driving mode involves evaluating information about the vehicle's external environment. For example, this can include determining whether there are static objects such as buildings, trees, signs, parked cars, etc. on one side of the road, and determining whether there are vehicles or other road users moving along nearby sections of the road. In certain scenarios, such as dense urban areas or other situations where the autonomous vehicle does not have an unobstructed view of oncoming traffic (unobstructed line of sight) at night, detection of other road users can be very challenging. In many cases, the autonomous vehicle may need to push out onto the road to obtain a better view of the traffic from its onboard sensors. However, entering the path of traffic can itself be dangerous, especially when there is limited time to react to nearby oncoming vehicles.

[0023] At night, in bad weather, or in other low visibility conditions, vehicles typically turn on their headlights (or fog lights, parking lights, or daytime running lights) to illuminate the road and nearby objects. In addition, high beams may be used instead of low beams in some cases. According to one aspect of the present technology, the vision system of an autonomous vehicle is able to detect such lights illuminating the road. This signal can be used as an input to a planning module of an onboard computer system, for example to predict or anticipate one or more vehicles from the direction of an illuminated road along the road, even if such vehicles are not in the line of sight of the autonomous vehicle's perception system. As discussed further below, this can include estimating the vehicle's speed, disambiguating the direction of travel, distinguishing between headlights and streetlights or other fixed lighting, and the like.

[0024] Example Vehicle Systems

[0025] Figure 1A A perspective view of an example passenger vehicle 100 , such as a minivan or sport utility vehicle (SUV), is shown. Figure 1BA perspective view of another example passenger car 150 (such as a sedan) is shown. The passenger car may include various sensors for obtaining information about the external environment of the vehicle. For example, the roof shell unit (roof compartment assembly) 102 may include a laser radar (lidar) sensor and various cameras (e.g., optical or infrared), a radar unit, an acoustic sensor (e.g., a microphone or sonar type sensor), an inertial (e.g., an accelerometer, a gyroscope, etc.) or other sensors (e.g., a positioning sensor such as a GPS sensor). The housing 104 located at the front end of the vehicle 100 and the housings 106a, 106b located on the driver's side and the passenger side of the vehicle may each incorporate a laser radar, a radar, a camera and / or other sensors. For example, the housing 106a may be located in front of the driver's side door along the side panel of the vehicle. As shown, the passenger car 100 also includes housings 108a, 108b for radar units, laser radars and / or cameras, which are also positioned toward the rear roof portion of the vehicle. Additional lidar, radar units and / or cameras (not shown) may be located elsewhere along the vehicle 100. For example, arrow 110 indicates that a sensor unit (not shown) may be located along the rear of vehicle 100, such as on or adjacent to a bumper. Depending on the vehicle type and sensor housing configuration(s), acoustic sensors may be arranged in any or all of these housings around the vehicle.

[0026] Arrow 114 indicates that the roof pod 102 shown includes a base portion coupled to the roof of the vehicle. And, arrow 116 indicates that the roof pod 102 also includes an upper portion that is elevated above the base portion. Each of the base portion and the upper portion may house a different sensor unit configured to obtain information about objects and conditions in the vehicle's surroundings. The roof pod 102 and other sensor housings may also be located along Figure 1B The vehicle 150 is arranged. As an example, each sensor unit may include one or more sensors of the above-mentioned types, such as lidar, radar, camera (e.g., optical or infrared), acoustic (e.g., passive microphone or active sounding sonar type sensor), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensor such as GPS sensor).

[0027] Figure 1C-Figure 1DAn example freight vehicle 150 is shown, such as a tractor-trailer truck. The truck may include, for example, a single, double or triple trailer, or may be another medium or heavy truck, such as commercial weight class 4 to 8. As shown, the truck includes a tractor unit 152 and a single cargo unit or trailer 154. Depending on the type of cargo to be transported, the trailer 154 may be fully enclosed, open (such as a flatbed), or partially open. In this example, the tractor unit 152 includes an engine and steering system (not shown) and a cab 156 for the driver and any passengers.

[0028] The trailer 154 includes a hitch point 158, which is referred to as a kingpin. The kingpin 158 is typically formed as a solid steel axle that is configured to be pivotally attached to the tractor unit 152. Specifically, the kingpin 158 is attached to a trailer coupling 160, which is referred to as a fifth-wheel that is mounted behind the cab. For a double tractor-trailer or a triple tractor-trailer, the second trailer and / or the third trailer may have a simple hitch connection to the lead trailer. Alternatively, each trailer may have its own kingpin. In this case, at least the first trailer and the second trailer may include a fifth-wheel type structure that is arranged to be coupled to the next trailer.

[0029] As shown, the tractor can have one or more sensor units 162, 164 arranged along it. For example, one or more sensor units 162 can be arranged on the roof or top of the cab 156, and one or more side sensor units 164 can be arranged on the left and / or right side of the cab 156. The sensor units can also be positioned along other areas of the cab 156, such as along the front bumper or hood area, at the rear of the cab, adjacent to the trailer wheels, under the chassis, etc. The trailer 154 can also have one or more sensor units 166 arranged along it, such as along the side panels, front, rear, roof and / or landing gear of the trailer 154.

[0030] and Figure 1A-1B Like the sensor units of a passenger vehicle, each sensor unit of a freight vehicle may include one or more sensors such as lidar, radar, camera (e.g., optical or infrared), acoustic (e.g., microphone or sonar type sensor), inertial (e.g., accelerometer, gyroscope, etc.) or other sensors (e.g., positioning sensor such as a GPS sensor).

[0031] Although certain aspects of the present disclosure may be particularly useful in conjunction with certain types of vehicles, the vehicle may be of different types of vehicles including, but not limited to, automobiles, motorcycles, freight vehicles, buses, recreational vehicles, emergency vehicles, construction equipment, and the like.

[0032] For vehicles operating in a partially or fully autonomous driving mode, different degrees of autonomy may occur. The National Highway Traffic Safety Administration and the Society of Automotive Engineers have identified different levels to indicate how much or how little the vehicle controls driving. For example, level 0 has no automation, and the driver makes all driving-related decisions. The lowest semi-autonomous mode, level 1, includes some driving assistance, such as cruise control. At this level, the vehicle can operate in a strict driver information system without any automatic control of the vehicle. Here, the vehicle's onboard sensors, the relative position knowledge between them, and the way they exchange data can be used to implement various aspects of the technology discussed in this article. Level 2 has partial automation of certain driving operations, while level 3 involves conditional automation, which enables the person in the driver's seat to control as needed. In contrast, level 4 is a high automation level, in which the vehicle can drive without assistance under selected conditions. In addition, level 5 is a fully autonomous mode, in which the vehicle can drive without assistance in all situations. The architecture, components, systems and methods described herein can function in any of a semi-autonomous mode or a fully autonomous mode (e.g., levels 1-5), which are referred to as autonomous driving modes in this article. Therefore, references to autonomous driving modes include both partially automated and fully automated.

[0033] Figure 2 A block diagram 200 of an exemplary vehicle (such as passenger vehicle 100 or 150) operating in an autonomous driving mode with various components and systems is shown. As shown, block diagram 200 includes one or more computing devices 202, such as a computing device containing one or more processors 204, memory 206, and other components typically found in a general-purpose computing device. Memory 206 stores information accessible by one or more processors 204, including instructions 208 and data 210 that can be executed or otherwise used by processor(s) 204. When operating in an autonomous driving mode, the computing system can control the overall operation of the vehicle.

[0034] The memory 206 stores information accessible by the processor 204, including instructions 208 and data 210 that can be executed or otherwise used by the processor 204. For example, the memory may include lighting related information for performing, for example, obscured vehicle detection. The memory 206 may be of any type capable of storing information accessible by the processor, including computing device readable media. The memory is a non-transitory medium, such as a hard drive, memory card, optical disk, solid state, etc. The system may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.

[0035] Instructions 208 can be any set of instructions executed by a processor (or multiple) directly (such as machine code) or indirectly (such as a script). For example, instructions can be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions", "modules" and "programs" can be used interchangeably herein. Instructions can be stored in an object code format for direct processing by a processor, or in any other computing device language, including a collection of scripts or independent source code modules that are interpreted on demand or pre-compiled. Data 210 can be retrieved, stored or modified by one or more processors 204 according to instructions 208. In one example, some or all of memory 206 can be an event data recorder or other secure data storage system configured to store vehicle diagnostics and / or detected sensor data, which can be on-board or remote depending on the implementation.

[0036] Processor 204 may be any conventional processor, such as a commercially available CPU. Alternatively, each processor may be a dedicated device, such as an ASIC or other hardware-based processor. Figure 2 The processor, memory, and other elements of computing device 202 are shown functionally as being within the same block, but such a device may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, memory 206 may be a hard drive or other storage medium located in a housing different from that of processor 204. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

[0037] In one example, computing device 202 may form an autonomous driving computing system incorporated into vehicle 100. The autonomous driving computing system may be able to communicate with various components of the vehicle. For example, computing device 202 may communicate with various systems of the vehicle, including a driving system including a deceleration system 212 (for controlling the braking of the vehicle), an acceleration system 214 (for controlling the acceleration of the vehicle), a steering system 216 (for controlling the orientation of the wheels and the direction of the vehicle), a signaling system 218 (for controlling the turn signal), a navigation system 220 (for navigating the vehicle to a certain location or around an object), and a positioning system 222 (for determining the location of the vehicle, for example, including the posture of the vehicle, for example, the position and orientation along the road or the pitch, yaw and roll of the vehicle chassis relative to the coordinate system). The autonomous driving computing system may employ a planner module 223 based on navigation system 220, positioning system 222 and / or other components of the system, for example, for determining a route from a starting point to a destination, or for modifying various driving aspects in view of current or expected traction conditions.

[0038] Computing device 202 is also operatively coupled to perception system 224 (for detecting objects in the vehicle environment), power system 226 (e.g., a battery and / or a gas or diesel powered engine), and transmission system 230 to control the movement, speed, etc. of the vehicle according to instructions 208 of memory 206 in an autonomous driving mode that does not require or need continuous or periodic input from a vehicle occupant. Some or all of wheels / tires 228 are coupled to transmission system 230, and computing device 202 may be able to receive information about tire pressure, balance, and other factors that may affect driving in the autonomous mode.

[0039] The computing device 202 can control the direction and speed of the vehicle by controlling various components, for example, via the planner module 223. As an example, the computing device 202 can use data from the map information and navigation system 220 to fully autonomously navigate the vehicle to a destination location. The computing device 202 can use the positioning system 222 to determine the location of the vehicle, and use the perception system 224 to detect and respond to objects when needed to safely reach the location. To do so, the computing device 202 can cause the vehicle to accelerate (e.g., by increasing the fuel or other energy provided to the engine by the acceleration system 214), decelerate (e.g., by reducing the fuel supplied to the engine, changing gears and / or by applying brakes by the deceleration system 212), change direction (e.g., by turning the front wheels or other wheels of the vehicle 100 by the steering system 216), and signal such changes (e.g., by illuminating a turn signal of the signaling system 218). Therefore, the acceleration system 214 and the deceleration system 212 can be part of a powertrain or other type of transmission system 230, wherein the transmission system 230 includes various components between the engine of the vehicle and the wheels of the vehicle. Again, by controlling these systems, the computing device 202 may also control the vehicle's drivetrain 230 in order to autonomously maneuver the vehicle.

[0040] The navigation system 220 may be used by the computing device 202 to determine and follow a route to a location. In this regard, the navigation system 220 and / or the memory 206 may store map information, for example, highly detailed maps that the computing device 202 may use to navigate or control a vehicle. As an example, these maps may identify the shape and elevation of a road, lane markings, intersections, crosswalks, speed limits, traffic lights, buildings, signs, real-time traffic information, vegetation, or other such objects and information. Lane markings may include features such as solid or dashed double lane lines or single lane lines, solid or dashed lane lines, reflectors, and the like. A given lane may be associated with a left lane line and / or a right lane line or other lane markings that define lane boundaries. Thus, most lanes may be bounded by the left edge of one lane line and the right edge of another lane line.

[0041] The perception system 224 includes sensors 232 for detecting objects outside the vehicle. The detected objects may be other vehicles, obstacles in the road, traffic signals, signs, trees, etc. The sensors 232 may also detect certain aspects of weather conditions, such as snow, rain, or water spray, or puddles, ice, or other materials on the road.

[0042] By way of example only, sensors of the perception system may include light detection and ranging (lidar) sensors, radar units, cameras (e.g., optical imaging devices, with or without neutral density (ND) filters), positioning sensors (e.g., gyroscopes, accelerometers, and / or other inertial components), infrared sensors, and / or any other detection device that records data that can be processed by the computing device 202. The perception system 224 may also include, for example, one or more microphones or other acoustic arrays arranged along the roof compartment 102 and / or other sensor mounting housings. The microphones may be capable of detecting sounds across a wide frequency band (e.g., 50 Hz to 25 kHz) in order to detect various types of noises, such as horn honking, tire squealing, brake actuation, etc.

[0043] Such sensors of the perception system 224 can detect objects outside the vehicle and their characteristics, such as location, orientation (posture) relative to the road, size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), direction of travel, speed of movement relative to the vehicle, etc., as well as environmental conditions surrounding the vehicle. The perception system 224 may also include other sensors within the vehicle to detect objects and conditions within the vehicle (e.g., in the passenger compartment). For example, such sensors can detect, for example, one or more people, pets, packages, etc., as well as conditions inside and / or outside the vehicle, such as temperature, humidity, etc. Further sensors 232 of the perception system 224 can measure the rotation rate of the wheel 228, the braking amount or type of braking of the deceleration system 212, and other factors associated with the equipment of the vehicle itself.

[0044] As data is generated by the perception system 224, the raw data obtained by the sensors can be processed by the perception system 224 and / or periodically or continuously sent to the computing device 202 for further processing. The computing device 202 can use the positioning system 222 to determine the location of the vehicle, and use the perception system 224 to detect and respond to objects when necessary to safely reach the location, for example, via adjustments made by the planner module 223, including adjustments in operation to handle occlusions and other issues. In addition, the computing device 202 can perform verification or calibration between individual sensors, all sensors in a particular sensor assembly, or sensors in different sensor assemblies or other physical housings.

[0045] like Figure 1A-1BAs shown, some sensors of the perception system 224 may be incorporated into one or more sensor assemblies or housings. In one example, these may be integrated into a front perimeter sensor assembly, a rear perimeter sensor assembly, or a side perimeter sensor assembly around the vehicle. In another example, other sensors may be part of the roof housing (roof compartment) 102. The computing device 202 may communicate with sensor assemblies located on the vehicle or otherwise distributed along the vehicle. Each assembly may have one or more types of sensors, such as those described above.

[0046] Return to Figure 2 , computing device 202 may include all components typically used in conjunction with a computing device, such as the processor and memory described above, as well as a user interface subsystem 234. User interface subsystem 234 may include one or more user inputs 236 (e.g., a mouse, keyboard, touch screen, and / or microphone) and one or more display devices 238 (e.g., a monitor with a screen or any other electrical device operable to display information). In this regard, an internal electronic display may be located within a vehicle cabin (not shown) and may be used by computing device 202 to provide information to passengers within the vehicle. Other output devices such as speakers 240 may also be located within the passenger vehicle.

[0047] The vehicle may also include a communication system 242. For example, the communication system 242 may also include one or more wireless configurations to facilitate communication with other computing devices, such as passenger computing devices within the vehicle, computing devices outside the vehicle (such as in other nearby vehicles on the road), and / or remote server systems. The network connection may include short-range communication protocols such as Bluetooth. TM 、Bluetooth TM Low Energy (LE), cellular connectivity, and various configurations and protocols including the Internet, the World Wide Web, Intranets, Virtual Private Networks, Wide Area Networks, Local Area Networks, private networks using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the foregoing.

[0048] Figure 3A A vehicle (e.g., Figure 1C-Figure 1D As shown in block diagram 300, the vehicle includes a control system of one or more computing devices, such as a processor 304, a memory 306, and a plurality of computing devices, such as a plurality of computing devices, such as a plurality of computing devices, and a plurality of computing devices. Figure 2The computing device 302 may include other components similar or equivalent to the components 202, 204, and 206 discussed above. For example, the memory may include lighting related information for performing, for example, occluded vehicle detection.

[0049] The control system may constitute an electronic control unit (ECU) of a tractor unit of a freight vehicle. As with instructions 208, instructions 308 may be any set of instructions that are executed by a processor directly (such as machine code) or indirectly (such as a script). Similarly, data 310 may be retrieved, stored, or modified by one or more processors 304 in accordance with instructions 308.

[0050] In one example, computing device 302 may form an autonomous driving computing system incorporated into vehicle 150. Figure 2 As discussed above, the autonomous driving computing system of block diagram 300 may be able to communicate with various components of the vehicle in order to perform route planning and driving operations. For example, computing device 302 may communicate with various systems of the vehicle, such as a driving system including a deceleration system 312, an acceleration system 314, a steering system 316, a signaling system 318, a navigation system 320, and a positioning system 322, each of which may be described above with respect to FIG. Figure 2 Works as discussed.

[0051] Computing device 302 is also operably coupled to perception system 324, power system 326, and drive system 330. Some or all of wheels / tires 228 are coupled to drive system 230, and computing device 202 may be able to receive information about tire pressure, balance, rotation rate, and other factors that may affect driving in autonomous mode. As with computing device 202, computing device 302 may control the direction and speed of the vehicle by controlling various components. As an example, computing device 302 may use data from map information and navigation system 320 to fully autonomously navigate the vehicle to a destination location. Similar to the above description for Figure 2 As described, computing device 302 may employ planner module 323 in conjunction with positioning system 322, perception system 324, and other subsystems to detect and respond to objects when needed to safely reach a location.

[0052] Similar to perception system 224, perception system 324 also includes one or more sensors or other components, such as those described above for detecting objects external to the vehicle, objects or conditions internal to the vehicle, and / or the operation of certain vehicle equipment (such as wheels and deceleration system 312). For example, Figure 3AAs shown, the perception system 324 includes one or more sensor assemblies 332. Each sensor assembly 232 includes one or more sensors. In one example, the sensor assembly 332 can be arranged as a sensor tower integrated into a side mirror on a truck, agricultural equipment, construction equipment, etc. Figure 1C-Figure 1D As described, sensor mounts 332 may also be located at different locations on the tractor unit 152 or trailer 154. Computing device 302 may communicate with sensor mounts located on both the tractor unit 152 and trailer 154. Each mount may have one or more types of sensors such as those described above.

[0053] Figure 3A Also shown is a coupling system 334 for connection between the tractor unit and the trailer. The coupling system 334 may include one or more power and / or pneumatic connections (not shown), as well as a trailer wheel 336 at the tractor unit for connection to a kingpin at the trailer. A communication system 338, which is equivalent to the communication system 242, is also shown as part of the vehicle system 300.

[0054] Similar to Figure 2 In this example, the freight truck or other vehicle may also include a user interface subsystem 339. The user interface subsystem 339 may be located within the cab of the vehicle and may be used by the computing device 202 to provide information to passengers within the vehicle, such as a truck driver who may be driving the truck in a manual driving mode.

[0055] Figure 3B A trailer such as Figure 1C-Figure 1D An example block diagram 340 of a system for a vehicle (e.g., a trailer 154) is shown. As shown, the system includes an ECU 342 of one or more computing devices, such as a computing device that includes one or more processors 344, a memory 346, and other components typically found in a general-purpose computing device. The memory 346 stores information accessible by the one or more processors 344, including instructions 348 and data 350 that can be executed or otherwise used by the processor 344. Figure 2 and Figure 3A The description of processors, memory, instructions and data applies to Figure 3B of these components.

[0056] The ECU 342 is configured to receive information and control signals from the trailer unit. The onboard processor 344 of the ECU 342 can communicate with various systems of the trailer, including the deceleration system 352, the signaling system 354, and the positioning system 356. The ECU 342 can also be operably coupled to a perception system 358, wherein one or more sensors are arranged in a sensor assembly 364 for detecting objects in the trailer environment. The ECU 342 can also be operably coupled to a power system 360 (e.g., a battery power source) to provide power to local components. Some or all of the wheels / tires 362 of the trailer can be coupled to the deceleration system 352, and the processor 344 can be capable of receiving information about tire pressure, balance, wheel speed, and other factors that may affect driving in autonomous mode, and relaying this information to the processing system of the tractor unit. The deceleration system 352, the signaling system 354, the positioning system 356, the perception system 358, the power system 360, and the wheels / tires 362 can be arranged in a manner such as described above with respect to Figure 2 and Figure 3A Operate in the manner described.

[0057] The trailer also includes a set of landing gear 366 and a coupling system 368. The landing gear can provide a support structure for the trailer when separated from the tractor unit. The coupling system 368 can be a part of the coupling system 334, which provides a connection between the trailer and the tractor unit. Therefore, the coupling system 368 can include a connecting portion 370 (e.g., for communication, power and / or pneumatic connection with the tractor unit). The coupling system also includes a king pin 372 configured to be connected to the trailer wheel of the tractor unit.

[0058] Example Implementation

[0059] Figure 4A-4G It shows that Figure 1A-1B An example of a roof compartment assembly 300 is shown. As described above, a roof compartment assembly can include lidar sensors as well as various cameras (e.g., optical or infrared), radar units, acoustic sensors (e.g., microphones or sonar type sensors), inertial (e.g., accelerometers, gyroscopes, etc.), or other sensors (e.g., positioning sensors such as GPS sensors). In the example shown, the roof compartment assembly 400 includes a base 402 arranged closer to the roof and an upper portion 404 located above the base and away from the roof. Each of these portions can include various sensor packages of the types described above, local processing systems (e.g., for processing raw data from the sensors), and other equipment, such as wipers or other cleaning systems for cleaning the various sensors. By way of example only, the front side 406 of the base 402 can include an image sensor (e.g., an optical camera) and / or a radar sensor arranged to detect objects in front of the vehicle when the vehicle is driving in autonomous mode.

[0060] The raised upper portion 404 may include different types of sensors arranged in different layers or configurations, such as part of a dome-type or layer cake-type arrangement. As an example, a series of image sensors (e.g., optical cameras) may be arranged in a circular or other configuration in the first portion 408 of the upper portion, such as to provide overlapping fields of view around the vehicle. And, the second portion 410 of the upper portion may include one or more lidar units or other sensors that may be configured to rotate 360° or otherwise provide a full field of view around the vehicle. In this example, the first portion 408 is mounted on the upper surface of the base 402, and the second portion 410 is arranged on top of the first portion 308.

[0061] like Figure 4A-4G As shown, the roof compartment assembly 400 also includes a pair of legs or other support members 412. For example, Figure 1A and 1B As shown, support members 412 are fixed to the bottom surface of base 402 and are configured to be connected along the left and right sides of the roof of the vehicle. Support members 412 are arranged so that the bottom surface of base 402 does not rest on the roof or directly contact the roof. Roof compartment assembly 400 may also include duct members 414 so that power, data and / or fluid ducts can run between the vehicle and roof compartment assembly 400. Front support members 412 may be fixed adjacent to or along the left / right A-pillars of the vehicle frame, while rear support members 412 may be fixed adjacent to or along the left / right C (or D) pillars of the vehicle frame.

[0062] Raising the roof hatch assembly above the roof allows some features to be incorporated into the bottom of the assembly. For example, one or more drain ports (not shown) can be included along the bottom, e.g., adjacent the support members. One or more air intake ports (not shown) can also be provided, e.g., for use as part of an active or passive cooling system for the roof hatch assembly.

[0063] Figure 5A A scenario 500 is shown in which an autonomous vehicle 502 is stopped at a first road segment 504 with a stop sign, and the first road segment 504 intersects with a second road segment 506. In this scenario, another vehicle 508 is traveling east along the second road segment 506. As shown, a pair of vehicles 510a, 510b are parked along the side of the second road segment 506. Vehicle 502 is shown by the shaded area as having a field of view 512 for one or more sensors of its perception system. However, due to the presence of vehicles 510a and 510b, vehicle 508 is obscured and is not within the field of view 512.

[0064] Figure 5BAnother scenario 520 is shown in which an autonomous vehicle 522 is stopped due to a traffic light at a first road segment 524 that intersects a second road segment 526. In this scenario, another vehicle 528 is traveling eastward along the second road segment 526. As shown, a truck 530 is also moving eastward along the second road segment 506, while another vehicle 532 is traveling westward along the second road segment 526. The vehicle 522 is shown by the shaded area as having a field of view 534 for one or more sensors of its perception system. However, due to the presence of the truck 530 moving eastward with the vehicle 528, the vehicle 528 is obscured and is not within the field of view 534.

[0065] As described above, at night, in bad weather, or other low visibility conditions, vehicles typically turn on their headlights, fog lights, or other lighting devices to illuminate the road surface and nearby objects. Such lighting can reflect from the road, other objects (e.g., the side of a truck, a building, a glass door or window, a street sign, etc.), and can be detected by the sensors of the autonomous vehicle. Such detected signals can be used as a planning module (e.g., Figure 2 223 or Figure 3A The onboard system may be able to estimate the speed of the vehicle, disambiguate the direction of travel, and distinguish between vehicle lights and street lights or other fixed lighting. This enables the planner module or another part of the onboard system to make driving decisions, such as waiting for the obscured vehicle to pass, changing lanes, braking, etc.

[0066] Fig. 6A Shown based on Figure 5A600 of an arrangement of vehicles 508. As shown here, vehicle 508 turns on its headlights, thereby creating a beam pattern 602 on road segment 506. Here, while vehicle 508 is obscured from field of view 512 by parked cars 510a and 510b, as indicated by the shadowed area in front of vehicle 502, one or more sensors of autonomous vehicle 502 have another field of view 604. The dotted area 606 indicates an area in which field of view 604 can detect at least a portion of beam pattern 602. This can be done, for example, via an optical camera sensor. Here, because vehicles 510a and 510b can be determined to be not moving and / or their headlights are not turned on, the presence of a detected portion of beam pattern 602 can be used by the autonomous vehicle as a signal that there is another object on the road that is obscured from its view. The size or shape of a detected portion of the beam pattern 602, alone or in combination with other information from the signal (e.g., the intensity of the light, the wavelength(s) of the light, the Doppler effect, etc.), can be used to indicate the presence of an obscured vehicle, its approximate speed, the type of object, the direction of travel, and / or other information that can be used by the autonomous vehicle to make driving decisions in real time.

[0067] Figure 6B Shown based on Figure 5B610. Similar to scene 610, vehicle 528 turns on its lights, thereby creating a beam pattern 612 on road segment 526. Here, not only is vehicle 528 obscured from the view of autonomous vehicle 522, but truck 530 also turns on its lights, creating a beam pattern 614 on road segment 526. In this scenario, autonomous vehicle 522 has a forward field of view 616. In this case, the dotted area 618 indicates the area where field of view 616 is able to detect at least a portion of beam pattern 612. In this example, field of view 616 may also cover a portion of beam pattern 614 from truck 530. Although beam pattern 614 may be expected to come from truck 530, especially at night or in poor visibility conditions (e.g., cloudy, fog, snow, sandstorm, etc.), the detected portion of beam pattern 612 may not be expected due to the obstruction of the view of vehicle 528. Similar to scene 600, the size or shape of the detected portion of beam pattern 612, alone or in combination with other information from the signal (e.g., intensity of light, wavelength(s) of light, Doppler effect, etc.), can be used to indicate the presence of the obscured vehicle, its approximate speed, object type, direction of travel, and / or other information that can be used by the autonomous vehicle to make driving decisions in real time. The method can also include using information about beam pattern 614 and detected truck 530 to identify the position and / or speed of vehicle 528 relative to the truck. As an example, the angle of the beam pattern from the truck can be different from the angle of the beam pattern from other vehicles, which can indicate a difference in elevation above the road surface, and therefore, different heights of the vehicles.

[0068] Figure 6C-D Another scenario 620 is shown. In this scenario, an autonomous vehicle 622 is waiting at an intersection. Figure 6C As shown, bicycle 624 is on the cross street adjacent to building 626, but Fig.6D624 is unable to see a bicycle due to a building. However, in this scenario, bicycle 624 has a light, such as a halogen or LED light with a relatively narrow beam pattern 628. The beam pattern 628 can be detected by one or more sensors (e.g., an optical imaging device) of the autonomous vehicle having a field of view 630. In this case, the dotted area 632 indicates the area where the field of view 630 is able to detect at least a portion of the beam pattern 628. Similar to the above scenario, the size or shape of the detected portion 632 of the beam pattern 628, alone or in combination with other information from the signal (e.g., the intensity of the light, the wavelength (or multiple) of the light, etc.), can be used to indicate the presence of the obscured object, its approximate speed, the type of object, the direction of travel, and / or other information that can be used by the autonomous vehicle to make driving decisions in real time. In this case, the system can determine that the object has only a single light. It can use other information, including the brightness of the light or the sweep path of the beam pattern 628, as shown by the arcuate dashed line 634, to identify that the object is a bicycle (or scooter or motorcycle), for example, because the sweep path is caused by the pivoting of the object's front wheel. Additional information such as whether the light is strobing or otherwise flashing can also help determine what type of object it is (e.g., a bicyclist vs. a motorcyclist, because a bicycle's lights are more likely to flash than a motorcycle's lights flashing on and off).

[0069] In addition to detecting light emitted from vehicles and other moving objects along (or adjacent to) the road, the perception system of an autonomous vehicle may also detect light from stationary objects. Figure 6C As shown, street lights 636 can provide lighting to sidewalks or other areas near intersections, traffic lights 638 can provide different colors of light to indicate stop, avoid, and go, and barbershop revolving paint bollard 639 can flash or provide other light patterns adjacent to the road. In this scenario, depending on the arrangement and type (or multiple) of the sensor of the perception system, the autonomous vehicle can detect such ambient lighting.

[0070] Because such ambient lighting may affect the identification of occluded objects of interest, identifying the cause(s) of the ambient lighting can be used to ignore or discard such lighting during analysis of the perception system data. This can include identifying whether the object is a stationary object, and the orientation of the light source relative to the road (e.g., pointing downward substantially vertically, such as street light 636, pointing downward at an oblique angle, such as traffic light 638, or illuminating the road from a source away from the road, such as barbershop bollard 639). It can also include determining the type of bulb of the light source (e.g., compact fluorescent, metal halide, or high pressure sodium for street light 636, incandescent or LED for traffic light 638, LED or neon for barbershop bollard 639, etc.). The wavelength(s) of such light can also be taken into account in the evaluation.

[0071] Any or all of this information can be used to create a baseline map of ambient lighting, such as during one or more times of the day (e.g., evening, night, or day), under different weather conditions, etc. The baseline can be stored in a database maintained by the autonomous vehicle or uploaded to the autonomous vehicle. For example, it can be used as a layer or overlay in a highly detailed map used by the autonomous vehicle. The map information can identify the location of stationary lighting objects. In one example, the baseline map of ambient lighting can be planned by a remote system used, which can propagate such information to a fleet of autonomous vehicles. By evaluating such information, the system can compare static scenes with changing lighting, which can be used as a sign of the presence of potential objects that are moving. This can include distinguishing between street lights or flashing lights (e.g., from building signs) and other lights that may be moving. Since moving objects will cause light or illuminated areas that change in size and shape, such differences can be used to identify the presence of obscured road users.

[0072] Identifying the source of illumination can be challenging for a number of reasons. Fig. 6EAs shown in scene 650 of FIG. 6 , an autonomous vehicle 652 may encounter beam patterns from different road users, such as (i) a beam pattern 654 from a cyclist (who is blocked from view by a building 656), (ii) a crossing beam pattern 658 from a car, and (iii) an opposing beam pattern 660 from a motorcyclist approaching the intersection from the opposite direction (who is blocked from view by foliage 662). In addition, water, ice, or another item 664 on the road surface, or fog, mist, smoke, dust, etc., can reflect, diffuse, and / or scatter light. If light is reflected from other surfaces, the system can use ray tracing to estimate such light where the light source may be located. For example, if the perception system is receiving data from a wall that the light is bouncing off of, and its shape and size are changing, the vehicle's onboard system can use this in conjunction with a stored map to make an estimate of where the light may be originating from.

[0073] Fig. 6F Another situation shown in scene 670 of FIG. 6 is the reflection of light from an obscured object due to a vehicle or other object in the environment. For example, autonomous vehicle 672 is shown traveling west on a road, while truck 674 is turning left from a southbound direction to travel east along the road. Vehicle 676 is behind truck 674 and is obscured by a building for autonomous vehicle 672 at the corner of the intersection. Here, beam pattern 678 from vehicle 676 may be partially blocked by truck 674, and partially reflected, as shown by dashed line 680. This reflection observed by the perception system of autonomous vehicle 672 may indicate the presence of vehicle 676. Alternatively or additionally, light emitted by the obscured object may be reflected from other objects in the environment (such as windows of passenger cars or windows of adjacent buildings), from signs (e.g., stop or yield signs), etc. In one example, autonomous vehicle 672 may use such information to change how it approaches an intersection, how long it may wait at an intersection, and how it proceeds through the intersection. For example, the autonomous vehicle may change its rate of deceleration to more slowly approach an intersection. It may wait at an intersection longer than initially planned (e.g., 10-25% longer) to determine whether more information about the occluded object is received. And, when advancing through an intersection, vehicle 672 may change the way it pulls forward to obtain more information from its perception system.

[0074] Figure 7A-7B Additional scenarios are shown where the environment itself affects the line of sight of other road users. Fig. 7A700 shows a scene where a passenger car 702 is traveling up a hill in one direction and a truck 704 is oriented in the opposite direction at the top of the hill. In this example, the truck 704 is shown with a sensor field of view 706 in the area in front of the truck. Here, if the passenger cars 702 turn on their low beams, they may only illuminate a portion of the road (shown by beam pattern 708) that may not be in the field of view 706 or may otherwise be difficult to detect. However, if the passenger cars 702 turn on their high beams, they may illuminate other portions of the road (shown by beam pattern 710) that are easily detectable within the field of view 706, as shown by the dotted area 712, depending on environmental conditions (e.g., fog, dust, or particles in the air).

[0075] Figure 7B View 720 of illustrative embodiment shows a scene in which a truck 722 is traveling along a curved road in one direction and a passenger car 724 is traveling in the opposite direction. In this example, the truck 722 is shown with a sensor field of view 726 in the area in front of the truck. Here, the lights of the passenger car 724 illuminate a portion of the curved road shown by the beam pattern 728, which is within the field of view 726, as shown by the dash-dotted area 730. Other reflections may also be detected and used in the analysis, depending on the road and nearby environment. For example, on a forest road, trees may be illuminated by the headlights of an approaching vehicle. Also, the lane lines themselves or reflective components in the road can illuminate and reflect light from another vehicle.

[0076] Lighting information about the nearby environment around the autonomous vehicle may itself be sufficient to identify the presence of obscured vehicles or other road users (e.g., cyclists or joggers). However, data obtained by other sensors of the autonomous vehicle and / or other information about the environment can be used to enhance the assessment process. As an example, acoustic information detected by on-board sensors can be tires on the road, the sound of horns honking, braking, etc. Lidar and / or radar sensors can provide a supplementary field of view for optical sensors. Maps of the local area around the autonomous vehicle may include information about static light emitters, such as street lights, parking lights, flashers on sidewalks, etc. It may also include altitude, slope, curvature and other data about the road itself. In addition, environmental data about the amount of ambient lighting, weather (rain, snow, fog, temperature, humidity, etc.) and other aspects of the surrounding environment of the autonomous vehicle can also be considered.

[0077] On-board processing systems (e.g. Figure 2 The computing device 200 or Figure 3AThe computing device 302 of the system can use the lighting information alone or in combination with any other signal to detect the presence of an obscured object. In one example, the system can compare the amount of light detected in a particular area and compare it to the expected amount of light (e.g., a baseline amount of light). If the amount of light detected exceeds the expected amount of light, it can be determined that an obscured object exists. In another example, if the amount of light detected has one or more characteristics (e.g., a minimum amount of brightness, a wavelength in a particular band, light emitted by a particular kind of lamp or a particular type of lighting device), it can also be determined that an obscured object exists. Alternatively, the system can generate a probability that an obscured vehicle exists. For example, based on the analyzed information, the system can indicate an 85% probability that another vehicle exists. In conjunction with determining the presence of an object or the probability of its presence, the system can estimate the speed and / or advance (direction of travel) of the object. This can include disambiguating the direction of travel (e.g., eastbound vs. westbound). Alternatively or additionally, the system may also be able to determine which side (or lane) of the road the object is on.

[0078] Object recognition / classification may be performed by the onboard computing device(s) in conjunction with recognition of the presence of other objects to determine the type of object and / or one or more characteristics of the object. As an example, each vehicle may maintain a database or other record of different types of road user objects (e.g., cars, trucks, motorcycles, mopeds, bicycles, emergency vehicles, construction vehicles, farm equipment, pedestrians or joggers, etc.), static environmental objects (e.g., street lights, stop lights, billboards and other illuminated signs, reflective signs (such as stop and yield signs), etc.), and / or lighting information associated with these objects (such as the type of light source (e.g., incandescent, halogen, LED, neon), wavelength and intensity information of the light source, relative height of the light source to the ground, whether the object typically has two or more light sources (e.g., cars, trucks) or only one light source (e.g., bicycles, motorcycles, scooters, joggers), whether such light is continuous or typically pulsed on and off), etc. Including information about different types of emergency vehicles (e.g., ambulances, fire trucks, police cars, etc.) may also be particularly beneficial because detecting these types of vehicles may be particularly important, enabling the autonomous vehicle to take corrective action as needed (e.g., drive over, slow down, change lanes, etc.). In one scenario, the system compares received sensor signals (e.g., detection of lighting along a road) and compares it to lighting and other information in a database to identify the type of object.

[0079] Object recognition / classification can include distinguishing between and / or correlating between different light sources to identify the type(s) of the occluded object(s). For example, the different light sources can include multiple road users, illumination from streetlights or parking lights, and reflections from one or more other objects such as vehicle windows, building windows, or body panels (e.g., body panels of a large truck). Here, the system can flag or exclude light from certain sources in the evaluation (e.g., light from overhead streetlights or parking lights). The system can also correlate light from one or more reflective surfaces, such as using triangulation or trilateration techniques to determine the source(s) of the reflected light.

[0080] Once the object of interest has been classified or otherwise identified, the system can use this information to infer the behavior of the identified object, to enhance or otherwise modify the current driving behavior of the autonomous vehicle, or both. Fig. 6A In scenario 600, the autonomous vehicle 502 may determine with a high probability (e.g., 75% certainty or greater) that a portion of the beam pattern 602 detected by its perception system is from a passenger vehicle (not a motorcycle) traveling east at 30 mph. Using map information for the road segment and the detected presence of a stop sign on the intersecting street, the autonomous vehicle may infer that the passenger vehicle will maintain its current speed and pass through the intersection within 5-10 seconds. Based on this, the autonomous vehicle may change its driving plan of slowly moving into the intersection and instead wait an additional amount of time before proceeding through the intersection.

[0081] In another example, given scenario 670, a westbound autonomous vehicle 672 detects that a truck 674 is turning east onto the road. Based on the reflection shown by the dashed line 680 and possibly by directly observing the beam pattern 678, the autonomous vehicle 672 can determine that there is a car behind the truck that is obscured by a building on the corner of the intersection. Here, the autonomous vehicle can evaluate the state of the traffic light (not shown) and the estimated speed of the obscured car to predict whether the car will stop at the light or follow the truck through the intersection. The autonomous vehicle can determine whether to perform braking or passive deceleration operations and / or take other corrective actions based on this. For intersections, the system can estimate whether there is a red light / stop sign runner or any vehicle in the blind spot of the autonomous vehicle. In one example, this can be tested by injecting a fake vehicle / object that may be moving at a high rate of speed into the data set. The presence of lights can be used to make such a test more accurate, rather than assuming that there is a motorcyclist going down the road at a high rate of speed (e.g., 2-3 times the posted speed limit). Using the information from the lights, the system can estimate whether there may be vehicles coming towards it at X mph, or even estimate the number of vehicles that may be present.

[0082] While the detection, classification, or other identification and inference of behavior may all be performed by an onboard processing system of an autonomous vehicle, in another example, off-board processing may be performed. For example, a backend system may perform fleet management operations for multiple autonomous vehicles and may be able to communicate directly with some or all of the autonomous vehicles in the fleet in real time. The backend system may have more processing resources available to it than an individual autonomous vehicle. Thus, in some cases, the backend system may be able to quickly classify and infer the behavior of a lighting object in real time and relay that information to the autonomous vehicle so that it can take corrective action or otherwise modify its planned driving operations.

[0083] In some examples, a machine learning model that may include a neural network may be trained on lighting and other information, for example, to determine object types. This may be performed for different types of driving environments, including highways, urban, suburban, and rural surface streets, tunnels, bridges, and the like, with and without other nearby objects (e.g., cars or other vehicles on the road, adjacent buildings, vegetation, road dividers, and the like). Training may be based on collected real-world data, such as with specific types of lighting in a single environment or in different environments (e.g., labeled by lighting type, location type, environmental conditions such as fog, time of day, temperature, and the like). From this, one or more light-based models may be developed and used in real-time signal evaluation of an autonomous vehicle, after-the-fact (e.g., post-processing) evaluation by a backend system, or both. As an example, the model structure may be a deep network, where the exact structure and parameters can be searched for by automated machine learning (e.g., using a neural architecture search (NAS) type model).

[0084] Based on this, an onboard system (e.g., a planner module and / or a navigation system of the vehicle's autonomous driving system) can utilize the model(s) to evaluate the detected light, determine the likelihood that the obscured object is on the road, and decide whether to change the planned driving maneuvers of the autonomous vehicle. The information can also be used in a reinforcement learning process. By way of example only, a reinforcement learning process may show that when a pair of headlights is detected at an intersection at night, the vehicle should wait at least a minimum number of seconds (e.g., 2-5 seconds) before turning or proceeding through the intersection.

[0085] As mentioned above, the technology is applicable to various types of autonomous vehicles, including passenger cars, buses, motorcycles, emergency vehicles, RVs, construction vehicles, and large trucks or other cargo vehicles. In addition to using lighting information for the operation of a single autonomous vehicle, the information can also be shared with other autonomous vehicles (such as vehicles that are part of a convoy).

[0086] An example of this is in Fig. 8A and Figure 8B In particular, Fig. 8A and Figure 8B 800 and 814 are schematic and functional diagrams of an example system 800 including a plurality of computing devices 802, 804, 806, 808 and a storage system 810 connected via a network 816. The system 800 also includes vehicles 812 and 814 configured to operate in an autonomous driving mode, which can be configured to communicate with each other. Figure 1A -B and Figure 1C -D vehicles 100 and 150 are the same or similar. Vehicle 812 and / or vehicle 814 may be part of a fleet. Although only a few vehicles and computing devices are depicted for simplicity, a typical system may include significantly more vehicles and computing devices.

[0087] like Figure 8B As shown, each of computing devices 802, 804, 806, and 808 may include one or more processors, memories, data, and instructions. Such processors, memories, data, and instructions may be similar to those described above. Figure 2 or Figure 3A The processors, memory, data, and instructions described are similarly configured.

[0088] The various computing devices and vehicles may communicate directly or indirectly via one or more networks, such as network 816. Network 816 and intermediate nodes may include various configurations and protocols, including short-range communication protocols such as Bluetooth. TM 、Bluetooth LE TM , the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the foregoing. Such communication may be facilitated by any device capable of sending data to and from other computing devices, such as modems and wireless interfaces.

[0089] In one example, computing device 802 may include one or more server computing devices with multiple computing devices, for example, a load balancing server group, which exchanges information with different nodes of the network for receiving data from other computing devices, processing data, and sending data to other computing devices. For example, computing device 802 may include one or more server computing devices that can communicate with computing devices of vehicle 812 and / or 814 and computing devices 804, 806, and 808 via network 816. For example, vehicle 812 and / or 814 may be part of a fleet of autonomous vehicles that can be dispatched to various locations by server computing devices. In this regard, computing device 802 may be used as a dispatch server computing system that can be used to dispatch vehicles to different locations in order to pick up and drop off passengers, or to pick up and deliver goods. In addition, server computing device 802 may use network 816 to send and present information to a user of one of the other computing devices or a passenger of the vehicle. In this regard, computing devices 804, 806, and 808 may be considered client computing devices.

[0090] like Fig. 8A As shown, each client computing device 804, 806, and 808 can be a personal computing device intended for use by a corresponding user 818, and has all components commonly used in conjunction with a personal computing device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and internal hard drive) to store data and instructions, a display (e.g., a monitor with a screen, a touch screen, a projector, a television, or other devices such as a smart watch display that can be operated to display information), and a user input device (e.g., a mouse, keyboard, touch screen, or microphone). The client computing device may also include a camera for recording video streams, speakers, a network interface device, and all components for connecting these elements to each other.

[0091] Although each of the client computing devices may include a full-size personal computing device, they may alternatively include mobile computing devices capable of wirelessly exchanging data with a server over a network such as the Internet. By way of example only, client computing devices 806 and 808 may be mobile phones or devices such as wireless-enabled PDAs, tablet PCs, wearable computing devices (e.g., smart watches), or netbooks capable of obtaining information via the Internet or other networks.

[0092] In some examples, the client computing device 804 may be a remote assistance workstation used by an administrator or operator to communicate with the driver of the dispatched vehicle. Figure 8A-8BOnly a single remote assistance workstation 804 is shown in the figure, but any number of such workstations may be included in a given system. In addition, although the operation workstation is depicted as a desktop computer, the operation workstation may include various types of personal computing devices, such as laptops, netbooks, tablet computers, etc. As an example, a technician or other user may use the remote assistance workstation to help determine the source of observed lighting, classify the source, and / or identify an appropriate response for the autonomous vehicle. The results of this evaluation may be provided to one or more vehicles in the vicinity of the observed lighting. Additionally, this may also be a process for obtaining labeled data to: 1) evaluate the accuracy of the method; and 2) train any networks involved.

[0093] The storage system 810 may be any type of computerized memory capable of storing information accessible by the server computing device 802, such as a hard drive, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a flash drive, and / or a tape drive. In addition, the storage system 810 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located in the same or different geographic locations. The storage system 810 may be connected to computing devices (such as a server computing device) via a network 816. Fig. 8A -B) and / or can be directly connected to or incorporated into any computing device.

[0094] The storage system 810 can store various types of information. For example, the storage system 810 can also store automatic vehicle control software that will be used by a vehicle such as vehicle 812 or 814 to operate such a vehicle in an autonomous driving mode. The storage system 810 can also store driver-specific or nominal driving models, as well as lighting and / or vehicle models and training data for different types of lighting. Model information can be shared with specific vehicles or fleets as needed. It can be updated in real time, periodically, or offline as additional driving information is obtained. The storage system 810 can also include map information, route information, weather information, etc. This information can be shared with vehicles 812 and 814, as an example, to help the on-board computer system perform behavioral analysis during real-time driving of the specific vehicle, such as when light from a potentially obscured object is detected in the external environment of the specific vehicle.

[0095] Fig. 9An exemplary operating method 900 for operating a vehicle in an autonomous driving mode according to the above discussion is shown. At box 902, the method includes obtaining lighting sensor data from an external environment around the vehicle by one or more sensors of a perception system of the vehicle. At box 904, the processor detects the presence of an obscured object in the external environment based on the obtained lighting sensor data. At box 906, the processor identifies at least one of the following: (i) the type of the object or (ii) one or more characteristics of the object. And, at box 908, based on at least one of the following: (i) the type of the object or (ii) one or more characteristics of the object, the processor controls the operation of the vehicle in the autonomous driving mode, such as by changing a braking, deceleration, or acceleration profile, by changing a lane or a position within a lane, by changing a planned progress through an intersection, etc.

[0096] Although the technology herein has been described with reference to specific embodiments, it will be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it will be understood that many modifications may be made to the illustrative embodiments and other arrangements may be designed without departing from the spirit and scope of the technology as defined by the appended claims.

Claims

1. A method of operating a vehicle in an autonomous driving mode, the method comprising: obtaining, by one or more sensors of a perception system of the vehicle, lighting sensor data from an external environment surrounding the vehicle; detecting, by one or more processors of the vehicle, the presence of an obscured object in the external environment based on the obtained illumination sensor data; identifying, by one or more processors, at least one of (i) a type of the object or (ii) one or more characteristics of the object; inferring a behavior of the object based on at least one of (i) a type of the object or (ii) one or more characteristics of the object; as well as Based on the inferred behavior of the object, one or more processors control the operation of the vehicle in an autonomous driving mode, wherein the obtained illumination sensor data indicates an area where a field of view of a perception system of the vehicle encompasses at least a portion of a beam pattern created by an occluded object, The obtained lighting sensor data also indicates an area in which the field of view of the vehicle's perception system encompasses a portion of a light beam pattern created by another object to identify a position and / or velocity of the obscured object relative to the other object.

2. The method according to claim 1, wherein: The detecting or identifying includes distinguishing between a plurality of illumination sources to determine the presence of one or more road users in the external environment.

3. The method according to claim 2, wherein: Distinguishing between the plurality of illumination sources includes identifying at least one fixed illumination source and at least one mobile illumination source.

4. The method according to claim 3, wherein: The at least one fixed lighting source is a street light, a parking light, a sign, or a parked vehicle.

5. The method according to claim 3, wherein: The at least one mobile illumination source is a passenger car, truck, motorcycle, bicycle, or emergency vehicle.

6. The method according to claim 1, wherein: The detecting or identifying includes correlating multiple sets of lighting sensor data to determine the light source.

7. The method according to claim 6, wherein: The multiple sets of lighting sensor data are obtained from different sensors of the vehicle's perception system.

8. The method according to claim 1, wherein: The one or more characteristics of the object include at least one of a location, an orientation relative to a road, a size, a shape, a heading, or a speed of movement relative to the vehicle.

9. The method according to claim 1, further comprising: Disambiguate the direction of travel of an object before controlling the operation of the vehicle.

10. The method according to claim 1, wherein: Detecting the presence of the occluded object in the external environment includes evaluating the acquired illumination sensor data to determine whether the detected amount of light has one or more characteristics.

11. The method according to claim 10, wherein: The one or more characteristics include at least one of a minimum amount of brightness, a wavelength in a particular band, or light emitted by a particular type of lighting device.

12. The method according to claim 1, wherein: The detecting includes comparing the acquired lighting sensor data to a baseline map of ambient lighting of the external environment surrounding the vehicle.

13. The method according to claim 12, wherein: The baseline mapping includes identifying the location of one or more stationary lighting objects in the external environment.

14. A vehicle configured to operate in an autonomous driving mode, the vehicle comprising: a driving system, including a steering subsystem, an acceleration subsystem, and a deceleration subsystem to control the driving of the vehicle in an autonomous driving mode; a perception system including one or more sensors, the one or more sensors being configured to detect objects in an external environment of the vehicle, the perception system being configured to obtain lighting sensor data from the external environment; a positioning system configured to determine a current location of the vehicle; as well as A control system, including one or more processors, the control system is operably coupled to the driving system, the perception system, and the positioning system, the control system is configured to: Based on the obtained illumination sensor data, detecting that the occluded object exists in the external environment; identifying at least one of (i) a type of the object or (ii) one or more characteristics of the object; inferring a behavior of the object based on at least one of (i) a type of the object or (ii) one or more characteristics of the object; and Controlling the vehicle's operation in autonomous driving mode based on the inferred behavior of objects, wherein the obtained illumination sensor data indicates an area where a field of view of a perception system of the vehicle encompasses at least a portion of a beam pattern created by an occluded object, The obtained lighting sensor data also indicates an area in which the field of view of the vehicle's perception system encompasses a portion of a light beam pattern created by another object to identify a position and / or velocity of the obscured object relative to the other object.

15. The vehicle of claim 14, wherein: The detecting or identifying includes distinguishing between a plurality of illumination sources to determine the presence of one or more road users in the external environment.

16. The vehicle of claim 14, wherein: The detecting or identifying includes correlating multiple sets of lighting sensor data to determine the light source.

17. The vehicle of claim 14, wherein: The control system is also configured to disambiguate the direction of travel of the object.

18. The vehicle of claim 14, wherein: The detecting includes comparing the acquired lighting sensor data to a baseline map of ambient lighting of the external environment surrounding the vehicle.

Images