Adjusting carrier sensor field of view volume

Abstract

An example method includes receiving, from one or more sensors associated with an autonomous vehicle, sensor data associated with a target object in an environment of the vehicle during a first environmental condition, wherein at least one of the sensors is configurable to be associated with one of a plurality of operational field of view volumes. The method further includes determining, based on the sensor data, at least one parameter associated with the target object. The method further includes determining a degradation of the parameter between the sensor data and past sensor data, wherein the past sensor data is associated with the target object in the environment during a second environmental condition different from the first environmental condition, and adjusting, based on the degradation, the operational field of view volume of the at least one sensor to a different one of the operational field of view volumes.

Description

[0001] Cross-reference to related applications

[0002] This disclosure claims priority to U.S. Nonprovisional Application No. 17 / 126,231, filed December 18, 2020, which claims priority to U.S. Provisional Application No. 62 / 952,879, filed December 23, 2019, the entire contents of which are incorporated herein by reference. This disclosure also relates to U.S. Patent Application No. 17 / 002,092, filed August 25, 2020, which claims priority to U.S. Provisional Application No. 62 / 952,879. The entire contents of U.S. Patent Application No. 17 / 002,092 are incorporated herein by reference. Background Technology

[0003] The vehicle can be configured to operate in an autonomous mode, in which it navigates its environment with little or no driver input. Such an autonomous vehicle can include one or more systems (e.g., sensors and associated computing devices) configured to detect information about the environment in which the vehicle operates. The vehicle and its associated computer-implemented controller use the detected information to navigate the environment. For example, if the systems(or systems) detect that the vehicle is approaching an obstacle determined by the computer-implemented controller, the controller adjusts the vehicle's directional control to allow the vehicle to maneuver around the obstacle.

[0004] For example, autonomous vehicles may include lasers, sonar, radar, cameras, thermal imagers, and other sensors that scan and / or record data about the vehicle's surrounding environment. Sensor data from one or more of these devices can be used to detect objects and their corresponding characteristics (position, shape, orientation, velocity, etc.). This detection and identification is useful for the operation of autonomous vehicles. Summary of the Invention

[0005] In one example, the disclosure provides a method. The method includes receiving, from one or more sensors associated with an autonomous vehicle, sensor data associated with a target object in an environment of the autonomous vehicle during a first environmental condition, wherein at least one sensor of the one or more sensors is configurable to be associated with one of a plurality of operational field of view volumes, and wherein each operational field of view volume represents a space within which the at least one sensor is expected to detect objects outside of the autonomous vehicle with a minimum level of confidence. The method further includes determining, based on the sensor data, at least one parameter associated with the target object. The method further includes determining a degradation of the at least one parameter between the sensor data and past sensor data, wherein the past sensor data is associated with the target object in the environment during a second environmental condition different from the first environmental condition. The method further includes adjusting the operational field of view volume of the at least one sensor to a different one of the plurality of operational field of view volumes based on the determined degradation of the at least one parameter.

[0006] In another example, the disclosure provides a system for controlling operations of an autonomous vehicle. The system includes one or more sensors, wherein at least one sensor of the one or more sensors is configurable to be associated with one of a plurality of operational field of view volumes, and wherein each operational field of view volume represents a space within which the at least one sensor is expected to detect objects outside of the autonomous vehicle with a minimum level of confidence. The system further includes one or more processors coupled to the one or more sensors. The system further includes a memory coupled to the one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations include receiving, from the one or more sensors during a first environmental condition, sensor data associated with a target object in an environment of the autonomous vehicle. The operations further include determining, based on the sensor data, at least one parameter associated with the target object. The operations further include determining a degradation of the at least one parameter between the sensor data and past sensor data, wherein the past sensor data is associated with the target object in the environment during a second environmental condition different from the first environmental condition. The operations further include adjusting the operational field of view volume of the at least one sensor to a different one of the plurality of operational field of view volumes based on the determined degradation of the at least one parameter.

[0007] In another example, the disclosure provides a non-transitory computer-readable storage medium having program instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations. The operations include receiving, from one or more sensors associated with an autonomous vehicle, sensor data associated with a target object in an environment of the autonomous vehicle during a first environmental condition, wherein at least one sensor of the one or more sensors is configurable to be associated with one of a plurality of operational field of view volumes, and wherein each operational field of view volume represents a space within which the at least one sensor is expected to detect objects outside of the autonomous vehicle with a minimum level of confidence. The operations also include determining, based on the sensor data, at least one parameter associated with the target object. The operations also include determining a degradation of the at least one parameter between the sensor data and past sensor data associated with the target object in the environment during a second environmental condition different from the first environmental condition. The operations also include adjusting the operational field of view volume of the at least one sensor to a different one of the plurality of operational field of view volumes based on the determined degradation of the at least one parameter.

[0008] These, and other, aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with appropriate reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 is a functional block diagram depicting aspects of an example autonomous vehicle.

[0010] Figure 2 depicts an external view of an example autonomous vehicle.

[0011] Figure 3 is a conceptual illustration of wireless communication between various computing systems related to an autonomous vehicle.

[0012] Figure 4 illustrates a set of example sensor fields of view.

[0013] Figure 5 illustrates an example method.

[0014] Figure 6A and Figure 6B depicts an example image for adjusting an operational field of view volume of at least one sensor.

[0015] Figure 7A and Figure 7B depicts an example image for adjusting an operational field of view volume of at least one sensor. DETAILED DESCRIPTION

[0016] Example methods, devices, and systems are described herein. It should be understood that the words “example” and “exemplary” as used herein denote “serving as an example, instance, or illustration.” Any implementation or feature described herein as “example” or “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations or features. Other implementations can be utilized, and other changes can be made, without departing from the scope of the subject matter presented herein.

[0017] Accordingly, the example embodiments described herein are not meant to be limiting. As generally described herein and illustrated in the drawings, aspects of the disclosure can be arranged, substituted, combined, separated, and designed in numerous different configurations, all of which are contemplated herein.

[0018] Furthermore, the features illustrated in each figure can be used with one another in various combinations, as is understood by those skilled in the art. Thus, the figures generally should be considered in the context of the entire disclosure, and not the context of a single figure. The following detailed description is made with reference to the accompanying drawings.

[0019] I. SUMMARY

[0020] Many vehicles include various sensing systems to aid in the navigation and control of the vehicle. Some vehicles can operate in a fully autonomous mode in which no human interaction is used, in a semi-autonomous mode in which little human interaction is used, or in a human operated mode in which a human operates the vehicle and sensors can aid the human. Sensors can be used to provide information about the area surrounding the vehicle. This information can be used to identify features of the road and other objects in the vicinity of the vehicle, such as other vehicles, pedestrians, etc.

[0021] A vehicle’s sensor system can include, for example, a light detection and ranging (lidar) system and a radar system. A lidar uses laser pulses to measure the distance to and velocity of objects that reflect the laser light. A radar uses radio waves to measure the distance to and velocity of objects that reflect the radio waves. Data from a lidar and radar system can be used, possibly along with data from other sensors of the vehicle’s sensor system such as cameras, to determine the location of objects in the vehicle’s surrounding environment. Particular lidar sensors, radar sensors, and / or cameras can each have a field of view. A sensor’s field of view can include one or more angular regions in which the sensor can detect objects and an associated range corresponding to the maximum distance from the sensor at which the sensor can reliably detect objects in the field of view. In some cases, the associated range can vary for various azimuth / elevation angles within the field of view. The parameter values defining the field of view, such as the range value, azimuth, and elevation angles, together form a volume that can be referred to as a field of view volume or operational field of view volume.

[0022] The operational field of view volume of a particular sensor can be considered to be an accurate representation of a space within which the particular sensor can be expected to detect objects outside the autonomous vehicle with a minimum level of confidence (e.g., a confidence level indicative of a high level of confidence). In other words, one or more processors of the vehicle system (e.g., a chip controlling operation of the sensor, or a processor of the vehicle control system) can be configured to reliably rely on sensor data acquired within the space defined by the operational field of view volume of the sensor. For example, the processor associated with a particular sensor can be configured to associate a higher level of confidence (e.g., above a predetermined confidence threshold level) with objects or other information detected at ranges, azimuths, and / or elevations within the operational field of view volume of that sensor, and can be configured to associate a lower level of confidence (e.g., at or below the predetermined confidence threshold level) with objects or other information detected at ranges, azimuths, and / or elevations outside the operational field of view volume.

[0023] A vehicle can be exposed to changing conditions while in operation, such as changes in weather (e.g., fog, rain, snow), changes in time of day, changes in speed limits, changes in terrain or other geographic conditions, changes in residential areas (e.g., city, suburb, rural), changes in the number of other vehicles or objects in close proximity to the vehicle, other changes outside the vehicle, and / or internal changes to the vehicle system (e.g., sensor errors, sensor surface cleanliness, vehicle subsystem malfunctions, etc.). One or more of these or other conditions can be present in the operating environment of the vehicle at any given point in time. In the context of the present disclosure, the “operating environment” of the vehicle can be or include one or more conditions inside and / or outside the vehicle that can change over time, including but not limited to the conditions described above and other conditions described elsewhere in the present disclosure. Thus, the operating environment of the vehicle changes when one or more such conditions change.

[0024] In some embodiments, the operating field of view volume of at least one sensor of a vehicle can be adjusted based on an operational design domain (ODD) of the vehicle. An ODD is defined by or includes conditions under which a given vehicle or other driving automation system or feature thereof is specifically designed to operate, including but not limited to environmental, geographic, and time-of-day restrictions, and / or the necessary presence or absence of certain traffic or road features. A vehicle can have multiple ODDs, each of which can include at least one of an environmental condition, a geographic condition, a time condition, a traffic condition, or a road condition. The vehicle system can associate the vehicle with a first ODD at one point in time, resulting in the vehicle system operating in a particular manner corresponding to the first ODD. At a later time, the vehicle system can detect a change in the operating environment of the vehicle, in which case the vehicle system can responsively associate the vehicle with a different second ODD, resulting in the vehicle system operating in a different manner corresponding to the second ODD.

[0025] A particular sensor can be configured to be associated with one of a plurality of operating field of view volumes. The plurality of operating field of view volumes can be unique to the particular sensor, or can be associated with multiple sensors of the same type (e.g., lidar, camera, or radar).

[0026] In some examples, the plurality of operating field of view volumes can be a finite / predetermined number of operating field of view volumes, each of which can be mapped in memory of the vehicle system (e.g., in a table) to a corresponding operating environment and / or ODD. Additionally or alternatively, any of the plurality of operating field of view volumes for a particular sensor or multiple sensors can be determined in real-time in response to a triggering condition (e.g., a change in the operating environment). Additionally or alternatively, any predetermined operating field of view volume associated with a particular sensor or multiple sensors can be compared to newly acquired sensor data to determine whether the predetermined operating field of view volume still accurately represents the extent to which the sensor(s) should be relied upon for a particular operating environment.

[0027] Some example methods and systems for adjusting an operating field of view volume of a sensor based on an operating environment of an autonomous vehicle (e.g., based on the vehicle system detecting the operating environment or detecting a change from one operating environment to another), and some example methods and systems for associating a vehicle with a particular ODD based on an operating environment of an autonomous vehicle, are described in U.S. Patent Application No. 17 / 002,092, the entirety of which is hereby incorporated by reference.

[0028] The present disclosure provides systems and methods for adjusting an operational field of view volume of one or more vehicle sensors using parameters associated with identifiable target objects in a vehicle environment, and the parameters can be determined from sensor data associated with the target objects. In particular, changes from a first environmental condition (e.g., sunny weather) to a second environmental condition (e.g., foggy or snowy weather) can cause at least one of these parameters to degrade. Accordingly, the disclosed systems and methods use the degradation of the parameter (or parameters) as a basis for adjusting the operational field of view volume of at least one of the vehicle sensors to a different one of a plurality of operational field of view volumes, thereby enabling the vehicle to operate using an operational field of view volume that precisely represents the space within which the sensor can confidently detect objects when the second environmental condition is present. Because the vehicle system compares newly acquired sensor data associated with a target object to past sensor data associated with the target object, the disclosed methods can occur in real-time or near real-time.

[0029] As one example, a lidar sensor of a vehicle can have an operational field of view volume that is used during sunny weather conditions and during the day, and this operational field of view volume can include a range of 200 meters or more. During these environmental conditions, the vehicle can determine lidar intensities from laser beams reflected by distant target objects such as utility poles. However, later during a thick fog condition during the day, the lidar intensities from the same target objects can degrade. Based on this degradation, the vehicle system can adjust the operational field of view volume of at least one lidar sensor of the vehicle to an operational field of view volume that includes, for example, a range of 100 meters.

[0030] The present disclosure also provides for adjusting the operation of an operational field of view volume of one type of sensor (e.g., a camera) based on a degradation parameter (or parameters) determined from sensor data received from another type of sensor (e.g., a lidar sensor), and vice versa.

[0031] Implementations of the disclosed systems and methods advantageously enable a vehicle system to adjust received sensor data in real-time in order to dynamically adapt to changing conditions during travel, and enable the vehicle to accurately and confidently continuously detect objects in its environment. The disclosed systems and methods also provide an efficient and reliable way to use known target objects in a vehicle environment to determine accurate adjustments to sensor operational field of view volumes.

[0032] II. Example Systems and Devices

[0033] Example systems and devices will now be described in more detail. Generally, the embodiments disclosed herein can be used in any system that includes one or more sensor systems environments. The illustrative embodiments described herein include vehicles that employ sensors, such as lidar, RADER, SONAR, cameras, etc. However, example systems can also be implemented in or take the form of other devices, such as robotic devices, industrial systems (e.g., assembly lines, etc.), or mobile communication systems or devices, among other possibilities.

[0034] The term“vehicle” is interpreted broadly herein to cover any moving object, including, for example, an aircraft, a watercraft, a spacecraft, a car, a truck, a van, a semi-trailer, a motorcycle, a golf cart, an off-road vehicle, an indoor robotic device, a warehouse transport vehicle, a forklift, a tractor or an agricultural vehicle, as well as a vehicle that travels on a track, such as a roller coaster, a trolley, a tram, or a train car, etc. Some vehicles can operate in a fully autonomous mode in which human interaction is not used for operation, in a semi-autonomous mode in which human interaction is used for operation to a lesser extent, or in a human-operated mode in which a human operates the vehicle and sensors can assist the human.

[0035] In example embodiments, an example vehicle system can include one or more processors, one or more forms of memory, one or more input devices / interfaces, one or more output devices / interfaces, and machine-readable instructions that, when executed by the one or more processors, cause the system to perform various functions, tasks, capabilities, etc., as described above. Example systems within the scope of the present disclosure will be described in more detail below.

[0036] Figure 1 is a functional block diagram illustrating a vehicle 100 according to an example embodiment. The vehicle 100 can be configured to operate, in whole or in part, in an autonomous mode and thus can be referred to as an“autonomous vehicle.” The vehicle can also be configured to be operated by a human, but with information provided to the human by the vehicle’s sensory system. For example, a computing system 111 can be capable of controlling the vehicle 100 in autonomous mode via control instructions to a control system 106 of the vehicle 100. The computing system 111 can be capable of receiving information from one or more sensor systems 104 and making one or more control processes (such as setting a direction to avoid a detected obstacle) based on the received information in an automated manner.

[0037] The autonomous vehicle 100 can be fully autonomous or partially autonomous. In a partially autonomous vehicle, some functions can optionally be manually controlled (e.g., by a driver) at some or all times. Further, a partially autonomous vehicle can be configured to switch between a fully manual mode of operation and a partially autonomous and / or fully autonomous mode of operation.

[0038] The vehicle 100 includes a propulsion system 102, a sensor system 104, a control system 106, one or more peripherals 108, a power source 110, a computing system 111, and a user interface 112. The vehicle 100 can include more or fewer subsystems, and each subsystem can optionally include multiple components. Furthermore, each subsystem and component of the vehicle 100 can be interconnected and / or in communication. Thus, one or more functions of the vehicle 100 described herein can be optionally divided among additional functional or physical components, or combined into fewer functional or physical components. In some further examples, additional functional and / or physical components can be added to the vehicle 100. Figure 1 In the illustrated example.

[0039] The propulsion system 102 can include components operable to provide powered motion to the vehicle 100. In some embodiments, the propulsion system 102 includes an engine / motor 118, an energy source 120, a transmission 122, and wheels / tires 124. The engine / motor 118 converts the energy source 120 into mechanical energy. In some embodiments, the propulsion system 102 can optionally include one or both of an engine and / or a motor. For example, a gas-electric hybrid vehicle can include both a gasoline / diesel engine and an electric motor.

[0040] The energy source 120 represents a source of energy, such as electrical and / or chemical energy, which can provide power, in whole or in part, to the engine / motor 118. That is, the engine / motor 118 can convert the energy source 120 into mechanical energy to operate the transmission. In some embodiments, the energy source 120 can include gasoline, diesel, other petroleum-based fuels, propane, other compressed-gas-based fuels, ethanol, solar panels, batteries, capacitors, flywheels, regenerative braking systems, and / or other power sources, etc. The energy source 120 can also provide energy to other systems of the vehicle 100.

[0041] The transmission 122 includes suitable gears and / or mechanical elements adapted to transfer mechanical power from the engine / motor 118 to the wheels / tires 124. In some embodiments, the transmission 122 includes a gear box, a clutch, a differential, a drive shaft, and / or an axle, etc.

[0042] The wheels / tires 124 are arranged to stably support the vehicle 100 while providing frictional traction with a surface, such as a road, over which the vehicle 100 moves. As such, the wheels / tires 124 are configured and arranged according to the nature of the vehicle 100. For example, the wheels / tires can be arranged in a unicycle, bicycle, motorcycle, tricycle, or automobile / truck four-wheel form. Other wheel / tire geometries are possible, such as geometries that include six or more wheels. Any combination of wheels / tires 124 of the vehicle 100 can be differentially rotatable relative to other wheels / tires 124. The wheels / tires 124 can optionally include at least one rigid wheel attached to the transmission 122 and at least one tire coupled to a corresponding wheel rim in contact with the driving surface. The wheels / tires 124 can include any combination of metal and rubber, and / or other materials or combinations of materials.

[0043] The sensor system 104 generally includes one or more sensors configured to detect information about the environment surrounding the vehicle 100. For example, the sensor system 104 can include a global positioning system (GPS) 126, an inertial measurement unit (IMU) 128, a radar unit 130, a laser rangefinder / lidar unit 132, a camera 134, and / or a microphone 136. The sensor system 104 can also include sensors configured to monitor internal systems of the vehicle 100 (e.g., 02 monitor, fuel gauge, engine oil temperature, wheel speed sensors, etc.). One or more of the sensors included in the sensor system 104 can be configured to be individually and / or collectively actuated in order to modify the position and / or orientation of the one or more sensors.

[0044] The GPS 126 is a sensor configured to estimate the geographic position of the vehicle 100. To this end, the GPS 126 can include a transceiver operable to provide information about the position of the vehicle 100 relative to the Earth.

[0045] The IMU 128 can include any combination of sensors (e.g., accelerometers and gyroscopes) configured to sense changes in position and orientation of the vehicle 100 based on inertial acceleration.

[0046] The radar unit 130 can represent a system that utilizes radio signals to sense objects within the local environment of the vehicle 100. In some embodiments, in addition to sensing objects, the radar unit 130 and / or the computing system 111 can be configured to sense the speed and / or direction of the objects. The radar unit 130 can include any antennas, waveguide networks, communication chips, and / or other components that facilitate radar operations.

[0047] Similarly, the laser rangefinder or lidar unit 132 can be any sensor configured to use laser light to sense objects in the environment in which the vehicle 100 is located. The laser rangefinder / lidar unit 132 can include one or more laser light sources, a laser scanner, and one or more detectors, among other system components. The laser rangefinder / lidar unit 132 can be configured to operate in a coherent (e.g., using heterodyne detection) or non-coherent detection mode.

[0048] The cameras 134 can include one or more devices configured to capture a plurality of images of the environment surrounding the vehicle 100. The cameras 134 can be still cameras or video cameras. In some embodiments, the cameras 134 can be mechanically movable, such as by rotating and / or tilting a platform on which the cameras are mounted. In this way, control processes of the vehicle 100 can be implemented to control movement of the cameras 134.

[0049] The sensor system 104 can also include microphones 136. The microphones 136 can be configured to capture sound from the environment surrounding the vehicle 100. In some cases, multiple microphones can be arranged in a single microphone array, or possibly multiple microphone arrays.

[0050] The control system 106 is configured to control operations that regulate acceleration of the vehicle 100 and its components. To effect acceleration, the control system 106 includes a steering unit 138, a throttle 140, a braking unit 142, a sensor fusion algorithm 144, a computer vision system 146, a navigation / path planning system 148, and / or an obstacle avoidance system 150, among others.

[0051] The steering unit 138 is operable to adjust the direction of the vehicle 100. For example, the steering unit can adjust the axis (or axes) of one or more wheels / tires 124 in order to effect steering of the vehicle. The throttle 140 is configured to control the operational speed of, for example, the engine / motor 118, and in turn adjust the forward acceleration of the vehicle 100 via the transmission 122 and the wheels / tires 124. The braking unit 142 decelerates the vehicle 100. The braking unit 142 can use friction to slow the wheels / tires 124. In some embodiments, the braking unit 142 inductively decelerates the wheels / tires 124 through a regenerative braking process to convert the kinetic energy of the wheels / tires 124 into an electrical current.

[0052] The sensor fusion algorithm 144 is an algorithm (or computer program product storing an algorithm) configured to accept data from the sensor system 104 as input. The data can include, for example, data representing information sensed at the sensors of the sensor system 104. The sensor fusion algorithm 144 can include, for example, a Kalman filter, a Bayesian network, etc. The sensor fusion algorithm 144 provides an assessment of the environment surrounding the vehicle based on the data from the sensor system 104. In some embodiments, the assessment can include an assessment of individual objects and / or features in the environment surrounding the vehicle 100, an estimate of a particular situation, and / or an estimate of possible interference between the vehicle 100 and features in the environment based on a particular situation (e.g., such as a predicted collision and / or impact).

[0053] The computer vision system 146 can process and analyze images captured by the cameras 134 to identify objects and / or features in the environment surrounding the vehicle 100. Detected features / objects can include traffic signals, road boundaries, other vehicles, pedestrians, and / or obstacles, etc. The computer vision system 146 can optionally employ object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and / or available computer vision techniques to enable classification and / or identification of the detected features / objects. In some embodiments, the computer vision system 146 can additionally be configured to map the environment, track perceived objects, estimate velocities of objects, etc.

[0054] The navigation and path planning system 148 is configured to determine a driving path for the vehicle 100. For example, the navigation and path planning system 148 can determine a series of velocity and directional directions to enable the vehicle to move along a path that substantially avoids perceived obstacles while generally propelling the vehicle along a road-based path leading to a final destination, which can be set according to user input via the user interface 112, for example. The navigation and path planning system 148 can additionally be configured to dynamically update the driving path based on perceived obstacles, traffic patterns, weather / road conditions, etc. as the vehicle 100 operates. In some embodiments, the navigation and path planning system 148 can be configured to incorporate data from the sensor fusion algorithm 144, the GPS 126, and one or more predetermined maps in order to determine the driving path for the vehicle 100.

[0055] The obstacle avoidance system 150 can represent a control system configured to identify, evaluate, and avoid or otherwise circumvent potential obstacles in the environment surrounding the vehicle 100. For example, the obstacle avoidance system 150 can change the navigation of the vehicle by operating one or more subsystems in the control system 106 to make a turning maneuver, a cornering maneuver, a braking maneuver, etc. In some embodiments, the obstacle avoidance system 150 is configured to automatically determine a feasible (“available”) obstacle avoidance maneuver based on surrounding traffic patterns, road conditions, etc. For example, the obstacle avoidance system 150 can be configured such that a turning maneuver is not made when other sensor systems detect a vehicle, a building obstacle, other obstacles, etc. in the vicinity of an area into which the vehicle is to turn. In some embodiments, the obstacle avoidance system 150 can automatically select a maneuver that is both available and maximizes the safety of the occupants of the vehicle. For example, the obstacle avoidance system 150 can select an obstacle avoidance maneuver that is predicted to cause the least acceleration in the passenger cabin of the vehicle 100.

[0056] The vehicle 100 also includes peripherals 108 configured to allow interaction between the vehicle 100 and external sensors, other vehicles, other computer systems, and / or users such as passengers of the vehicle 100. For example, the peripherals 108 for receiving information from passengers, external systems, etc. can include a wireless communication system 152, a touchscreen 154, a microphone 156, and / or a speaker 158.

[0057] In some embodiments, the peripherals 108 are used to receive input from a user of the vehicle 100 to interact with the user interface 112. To this end, the touchscreen 154 can both provide information to a user of the vehicle 100 and communicate information from the user indicated via the touchscreen 154 to the user interface 112. The touchscreen 154 can be configured to sense touch locations and touch gestures from a user’s finger (or stylus, etc.) via capacitive sensing, resistive sensing, optical sensing, surface acoustic wave processes, etc. The touchscreen 154 can sense finger movement in a direction parallel or planar to the touchscreen surface, in a direction perpendicular to the touchscreen surface, or in both directions, and can also sense pressure levels applied to the touchscreen surface. The passengers of the vehicle 100 can also utilize a voice command interface. For example, the microphone 156 can be configured to receive audio (e.g., voice commands or other audio input) from a user of the vehicle 100. Similarly, the speaker 158 can be configured to output audio to a user of the vehicle 100.

[0058] In some embodiments, the peripherals 108 are used to allow communication between the vehicle 100 and external systems in its surrounding environment, such as devices, sensors, other vehicles, etc., and / or with controllers, servers, etc. that are physically remote from the vehicle, that provide useful information about the vehicle's surrounding environment, such as traffic information, weather information, etc. For example, the wireless communication system 152 can be capable of wireless communication with one or more devices, either directly or via a communication network. The wireless communication system 152 can optionally use 3G cellular communication such as CDMA, EVDO, GSM / GPRS, and / or 4G cellular communication such as WiMAX or LTE. Additionally or alternatively, the wireless communication system 152 can be capable of communication with a wireless local area network (WLAN), e.g., using WiFi. In some embodiments, the wireless communication system 152 can be capable of direct communication with devices, e.g., using an infrared link, Bluetooth, and / or ZigBee. The wireless communication system 152 can include one or more dedicated short-range communication (DSRC) devices, which can include public and / or private data communication between vehicles and / or roadside stations. In the context of the present disclosure, the wireless communication system 152 can also employ other wireless protocols for delivering and receiving information embedded in signals, such as various vehicle communication systems.

[0059] As noted above, the power supply 110 can supply power to components of the vehicle 100, such as electronic devices in the peripherals 108, the computing system 111, the sensor system 104, etc. For example, the power supply 110 can include rechargeable lithium-ion or lead-acid batteries for storing and releasing electrical energy to various powered components. In some embodiments, one or more groups of batteries can be configured to provide power. In some embodiments, the power supply 110 and the energy source 120 can be implemented together, as in some all-electric vehicles.

[0060] Many or all of the functions of the vehicle 100 can be controlled via the computing system 111, which receives inputs from the sensor system 104, the peripherals 108, etc., and communicates appropriate control signals to the propulsion system 102, the control system 106, the peripherals, etc., to enable automated operation of the vehicle 100 based on its surrounding environment. The computing system 111 includes at least one processor 113 (which can include at least one microprocessor) that executes instructions 115 stored in a non-transitory computer-readable medium such as the data store 114. The computing system 111 can also represent multiple computing devices that are used to control various components or subsystems of the vehicle 100 in a distributed manner.

[0061] In some embodiments, the data store 114 contains instructions 115 (e.g., program logic) that are executable by the processor 113 to perform various functions of the vehicle 100, including those described above in connection with the Figure 1The described functionality. Data store 114 can also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of propulsion system 102, sensor system 104, control system 106, and peripherals 108.

[0062] In addition to instructions 115, data store 114 can store data, such as human road maps, path information, and other information, as map data 116. During operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes, vehicle 100 and computing system 111 can use such information to select available roads to reach a final destination, interpret information from sensor system 104, and so on.

[0063] Vehicle 100 and associated computing system 111 provide information to and / or receive input from users of vehicle 100, such as passengers in a passenger cabin of vehicle 100. User interface 112 can accordingly include one or more input / output devices within peripherals 108, such as wireless communication system 152, touchscreen 154, microphone 156, and / or speaker 158, to allow for communication between computing system 111 and vehicle passengers.

[0064] Computing system 111 controls operation of vehicle 100 based on input received from various subsystems indicative of vehicle and / or environmental conditions (e.g., propulsion system 102, sensor system 104, and / or control system 106), as well as input from user interface 112 indicative of user preferences. For example, computing system 111 can utilize input from control system 106 to control steering unit 138 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 150. Computing system 111 can be configured to control many aspects of vehicle 100 and its subsystems. Generally, however, provisions are made for manual override of automatically controller-driven operations, such as in emergency situations, or simply in response to user-activated overrides, and so on.

[0065] Components of vehicle 100 described herein can be configured to work in interconnection with other components internal or external to their respective systems. For example, when operating in autonomous mode, cameras 134 can capture a plurality of images representative of information about the environment of vehicle 100. The environment can include other vehicles, traffic lights, traffic signs, road signs, pedestrians, and so on. Computer vision system 146 can classify and / or recognize various aspects of the environment based on object recognition models pre-stored in data store 114, and / or through other techniques with sensor fusion algorithm 144, computing system 111, and so on.

[0066] Although in Figure 1The host vehicle 100 is described and shown with various components of the vehicle 100 integrated into the vehicle 100, such as the wireless communication system 152, the computing system 111, the data store 114, and the user interface 112, but one or more of these components can alternatively be installed apart from or associated with the vehicle 100. For example, the data store 114 can exist in part or in whole independently of the vehicle 100, such as in a cloud-based server. Thus, one or more functional elements of the vehicle 100 can be implemented in the form of devices elements positioned separately or together. The functional device elements that make up the vehicle 100 can generally be communicatively coupled together, in wired and / or wireless fashion.

[0067] Figure 2 An example vehicle 200 is shown that can include the features described with reference to Figure 1 the vehicle 100. Although the vehicle 200 is illustrated as a four-wheeled vehicle in Figure 2 the interest of illustration, the present disclosure is not so limited. For example, the vehicle 200 can represent a truck, a van, a semi-truck, a motorcycle, a golf cart, an off-road vehicle, or an agricultural vehicle, among others.

[0068] The example vehicle 200 includes a sensor unit 202, a first lidar unit 204, a second lidar unit 206, a first radar unit 208, a second radar unit 210, a first lidar / radar unit 212, a second lidar / radar unit 214, and two additional locations 216, 218 on the vehicle 200 at which radar units, lidar units, laser rangefinder units, and / or other types of sensors can be located. Each of the first lidar / radar unit 212 and the second lidar / radar unit 214 can take the form of a lidar unit, a radar unit, or both.

[0069] In addition, the example vehicle 200 can include any of the components described with reference to Figure 1 the vehicle 100. The first radar unit 208 and the second radar unit 210 and / or the first lidar unit 204 and the second lidar unit 206 can actively scan the surrounding environment to discover the presence of potential obstacles and can be similar to the radar units 130 and / or the laser rangefinder / lidar units 132 in the vehicle 100. In addition, the first lidar / radar unit 212 and the second lidar / radar unit 214 can actively scan the surrounding environment to discover the presence of potential obstacles and can be similar to the radar units 130 and / or the laser rangefinder / lidar units 132 in the vehicle 100.

[0070] In some examples, the lidar units can be one of two different types of lidar units. The first type of lidar unit can be a lidar capable of continuously scanning an entire region of the lidar unit’s field of view. The second type of lidar unit can be a lidar capable of scanning a particular region of the lidar unit’s field of view when steered to that particular region. The first type of lidar unit can have a shorter range than the second type of lidar unit. The second type of lidar unit can have a smaller field of view when in operation than the first lidar unit. In some examples, one or more designated lidar units of the vehicle 200 can contain one or both types of lidar units. For example, the lidar unit 204 mounted on the roof of the vehicle can contain both types of lidar units, or contain a lidar unit capable of both continuous scanning and steered scanning. In one example, the second type of lidar unit can have an operational field of view that is 5 to 15 degrees wide in the horizontal plane and 5 to 25 degrees wide in the vertical plane.

[0071] The sensor unit 202 is mounted on the roof of the vehicle 200 and includes one or more sensors configured to detect information about the environment surrounding the vehicle 200 and output an indication of that information. For example, the sensor unit 202 can include any combination of cameras, radars, lidars, range finders, acoustic sensors, and weather-related sensors such as barometers, humidity sensors, etc. The sensor unit 202 can include one or more movable mounts that are operable to adjust the orientation of one or more of the sensors in the sensor unit 202. In one embodiment, the movable mounts can include a rotating platform that can sweep the sensors in order to obtain information from every direction around the vehicle 200. In another embodiment, the movable mounts of the sensor unit 202 can move in a sweeping manner over a range of specific angles and / or azimuths. The sensor unit 202 can be mounted on the roof of a car, although other mounting locations are possible. Furthermore, the sensors of the sensor unit 202 can be distributed in different locations and need not be collocated in a single location. Some possible sensor types and mounting locations include two additional locations 216, 218. Furthermore, each sensor of the sensor unit 202 can be configured to move or sweep independently of, or in conjunction with, the other sensors of the sensor unit 202.

[0072] In an example configuration, one or more radar scanners (e.g., first radar unit 20 and second radar unit 210) can be located near the rear of vehicle 200 to actively scan the area behind vehicle 200 for the presence of radio-reflective objects. Similarly, first lidar / radar unit 212 and second lidar / radar unit 214 can be mounted near the front of the vehicle to actively scan the area in front of the vehicle. The radar scanners can be located, for example, in positions suitable for illuminating the area including the forward-moving path of vehicle 200 without being obstructed by other features of vehicle 200. For example, the radar scanners can be embedded in and / or mounted near the front bumper, headlamps, fenders, and / or hood, among others. Additionally, one or more additional radar scanning devices can be provided to actively scan the sides and / or rear of vehicle 200 for the presence of radio-reflective objects, such as by including such devices in or near the rear bumper, side panels, rocker panels, and / or undercarriage, among others.

[0073] In fact, each radar unit can scan a beam width of 90 degrees. When the radar units are placed in the corners of the vehicle, as shown by radar units 208, 210, 212, and 214, each radar unit can scan a 90-degree field of view in the horizontal plane and provide the vehicle with a radar field of view of a full 360 degrees of the area surrounding the vehicle. Additionally, the vehicle can also include two lateral radar units. The lateral radar units can be able to provide further radar imaging when other radar units are obstructed, such as when a protected right turn is made (i.e., a right turn when there is another vehicle in the lane to the left of the turning vehicle).

[0074] Although Figure 2 Although not shown in FIG. 2, vehicle 200 can include a wireless communication system. The wireless communication system can include wireless transmitters and receivers that can be configured to communicate with devices outside or inside of vehicle 200. In particular, the wireless communication system can include a transceiver configured to communicate with other vehicles and / or computing devices, such as in a vehicle communication system or a roadside station. Examples of such vehicle communication systems include Dedicated Short-Range Communication (DSRC), Radio Frequency Identification (RFID), and other communication standards proposed for intelligent transportation systems.

[0075] Vehicle 200 can include a camera, possibly at a location inside of sensor unit 202. The camera can be a light-sensitive instrument, such as a still camera, video camera, or the like, configured to capture a plurality of images of the environment of vehicle 200. To this end, the camera can be configured to detect visible light, and can additionally or alternatively be configured to detect light from other portions of the light spectrum, such as infrared or ultraviolet light. In one particular example, sensor unit 202 can contain both an optical camera (i.e., a camera that captures human-visible light) and an infrared camera. The infrared camera can be able to capture thermal images within the field of view of the camera.

[0076] The camera can be a two-dimensional detector and can optionally have sensitivity with a three-dimensional spatial range. In some embodiments, the camera can include, for example, a distance probe configured to generate a two-dimensional image indicative of distances from the camera to a plurality of points in the environment. To this end, the camera can use one or more distance probing techniques. For example, the camera can provide distance information by using structured light techniques in which the vehicle 200 illuminates an object in the environment with a predetermined light pattern, such as a grid or checkerboard pattern, and uses the camera to detect reflections of the predetermined light pattern from the surrounding environment. Based on distortions in the reflected light pattern, the vehicle 200 can determine distances to points on the object. The predetermined light pattern can include infrared light, or other suitable wavelengths of radiation for such measurements. In some examples, the camera can be mounted within a front windshield of the vehicle 200. In particular, the camera can be positioned to capture images from a front view perspective with respect to the orientation of the vehicle 200. Other mounting locations and perspectives of the camera can also be used, inside or outside the vehicle 200. Moreover, the camera can have associated optics operable to provide an adjustable field of view. Still further, the camera can be mounted to the vehicle 200 with a movable mount to vary the pointing angle of the camera, such as via a pan / tilt mechanism.

[0077] Further, the camera sensor can be configured with a rolling shutter. A rolling shutter generally iteratively samples a photosensor to capture image data. The data from the camera sensor can form one image, multiple images, or a video. For example, in a traditional image sensor, a rolling shutter can iteratively sample a photosensor one row at a time. When sampling a camera sensor with a rolling shutter, high speed objects in the sensor field of view can appear distorted. This distortion is caused by the iterative sampling. Because the cell lines are sampled repeatedly, the object being imaged moves slightly between each sample. Thus, the sampling time for each row is slightly later than the previous row. Due to the delay in sampling the respective line, objects with horizontal motion can have a horizontal skew. For example, a vehicle passing through the sensor field of view can have a horizontal skew and vertical compression (or expansion), distorting the vehicle. This skew can be troublesome for processing based on the horizontal position of objects in the image. The present system can help identify possible camera distortions caused by a rolling shutter.

[0078] Figure 3is a conceptual illustration of wireless communication between various computing systems related to autonomous vehicles in accordance with example implementations. In particular, wireless communication can occur between remote computing system 302 and vehicle 200 via network 304. Wireless communication can also occur between server computing system 306 and remote computing system 302, as well as between server computing system 306 and vehicle 200. During operation of vehicle 200, the vehicle can deliver and receive data from both server computing system 306 and remote computing system 302 to aid in the operation of vehicle 200. Vehicle 200 can communicate data related to its operation and data from its sensors to server computing system 306 and remote computing system 302. Furthermore, vehicle 200 can receive operational instructions and / or data related to objects sensed by the vehicle’s sensors from server computing system 306 and remote computing system 302.

[0079] Vehicle 200 can correspond to various types of vehicles capable of transporting passengers or objects between different locations, and can take the form of any one or more of the vehicles described above.

[0080] Remote computing system 302 can represent any type of device related to remote assistance and operations technology, including but not limited to those described herein. In examples, remote computing system 302 can represent any type of device configured to: (i) receive information related to vehicle 200; (ii) provide an interface through which a human operator or computer operator can perceive the information and input a response related to the information, in turn; and (iii) send the response to vehicle 200 or other devices. Remote computing system 302 can take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some examples, remote computing system 302 can include multiple computing devices operating together in a network configuration.

[0081] Remote computing system 302 can include one or more subsystems and components similar to or the same as those of vehicle 200. At a minimum, remote computing system 302 can include a processor configured to perform various operations described herein. In some implementations, remote computing system 302 can also include a user interface including input / output devices, such as a touchscreen and a speaker. Other examples are possible.

[0082] Network 304 represents infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, as well as between server computing system 306 and vehicle 200.

[0083] The location of the remote computing system 302 can vary in examples. For example, the remote computing system 302 can have a location that is remote from the vehicle 200, which has wireless communication via the network 304. In another example, the remote computing system 302 can correspond to a computing device within the vehicle 200 that is separate from the vehicle 200, but that a human operator utilizes to interact with a passenger or driver of the vehicle 200. In some examples, the remote computing system 302 can be a computing device with a touchscreen that is operable by a passenger of the vehicle 200.

[0084] In some implementations, the operations described herein as being performed by the remote computing system 302 can additionally or alternatively be performed by the vehicle 200 (i.e., by any system(s) or subsystem(s) of the vehicle 200). In other words, the vehicle 200 can be configured to provide a remote assistance mechanism that a driver or passenger of the vehicle is able to interact with.

[0085] The server computing system 306 can be configured to wirelessly communicate with the remote computing system 302 and the vehicle 200 via the network 304 (or possibly directly with the remote computing system 302 and / or the vehicle 200). The server computing system 306 can represent any computing device configured to receive, store, determine, and / or deliver information related to the vehicle 200 and its remote assistance. As such, the server computing system 306 can be configured to perform any operation(s) or a portion of these operations that are described herein as being performed by the remote computing system 302 and / or the vehicle 200. Some implementations of wireless communication related to remote assistance can utilize the server computing system 306, while others can not.

[0086] The server computing system 306 can include one or more subsystems and components similar to or the same as those of the remote computing system 302 and / or the vehicle 200, such as a processor configured to perform the various operations described herein, and a wireless communication interface to receive information from and provide information to the remote computing system 302 and the vehicle 200.

[0087] The various systems described above can perform various operations. These operations and related features will now be described.

[0088] In accordance with the above discussion, a computing system (e.g., the remote computing system 302, or possibly the server computing system 306, or a computing system local to the vehicle 200) can operate to use a camera to capture an image of an environment of an autonomous vehicle. Generally, at least one computing system will be able to analyze the image and possibly control the autonomous vehicle.

[0089] In some implementations, to facilitate autonomous operation, a vehicle (e.g., vehicle 200) can receive data representative of objects in the environment in which the vehicle operates (also referred to herein as“environment data”) in various ways. Sensor systems on the vehicle can provide environment data representative of objects in the environment. For example, the vehicle can have various sensors, including cameras, radar units, laser range finders, microphones, radio units, and other sensors. Each of these sensors can communicate environment data to a processor in the vehicle regarding the information received by each respective sensor.

[0090] In one example, a radar unit can be configured to transmit electromagnetic signals that reflect off one or more objects in the vicinity of the vehicle. The radar unit can then capture the reflected electromagnetic signals off the objects. The captured reflected electromagnetic signals can enable the radar system (or processing system) to make various determinations about the objects that reflected the electromagnetic signals. For example, distances and locations to various reflecting objects can be determined. In some implementations, the vehicle can have more than one radar in different orientations. In fact, the vehicle can have six different radar units. Further, each radar unit can be configured to direct a beam in one of four different sectors of the radar unit. In various examples, by scanning each of the four different sectors of the radar unit, the radar unit can be able to scan a beam over a 90 degree range. The radar system can be configured to store the captured information into memory for later processing by a processing system of the vehicle. The information captured by the radar system can be environment data.

[0091] In another example, a laser range finder (e.g., lidar unit) can be configured to transmit electromagnetic signals (e.g., light, such as light from a gas or diode laser or other possible light source) that can reflect off one or more target objects in the vicinity of the vehicle. The laser range finder can be able to capture the reflected electromagnetic (e.g., laser) signals. The captured reflected electromagnetic signals can enable the range finding system (or processing system) to determine ranges to various objects, such as objects that reflected the electromagnetic signals back to the laser range finder. The range finding system can also be able to determine the speed or velocity of the target objects and store it as environment data.

[0092] In some implementations, the processing system can be able to combine information from various sensors in order to make further determinations about the environment of the vehicle. For example, the processing system can determine whether another vehicle or pedestrian is in front of the autonomous vehicle in conjunction with data from both radar information and captured images. In other implementations, the processing system can use other combinations of sensor data to make determinations about the environment.

[0093] When operating in an autonomous mode, a vehicle can control its operation with little or no human input. For example, a human operator can input an address into the vehicle, and then the vehicle is able to travel to the specified destination without further input from the human (e.g., the human does not have to turn or touch the brake / gas pedal). In addition, while the vehicle is operating autonomously, a sensor system can receive environmental data. A processing system of the vehicle can change control of the vehicle based on the environmental data received from the various sensors. In some examples, the vehicle can change the speed of the vehicle in response to the environmental data from the various sensors. The vehicle can change the speed to avoid obstacles, obey traffic laws, etc. When the processing system in the vehicle identifies an object near the vehicle, the vehicle can be able to change speed, or change motion in another way.

[0094] When a vehicle detects an object, but is not highly confident in its detection of the object, the vehicle can be able to request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) confirming whether the object is actually present in the environment (e.g., whether a stop sign is actually present or whether a stop sign is not actually present), (ii) confirming whether the vehicle’s identification of the object is correct, (iii) correcting the identification if the identification is not correct, and / or (iv) providing supplemental instructions (or modifying current instructions) for the autonomous vehicle.

[0095] Depending on the source of the environmental data, the vehicle can detect objects of the environment in various ways. In some implementations, the environmental data can be from a camera, and can be image or video data. The vehicle can analyze the captured image or video data to identify objects in the image or video data. In other implementations, the environmental data can be from a lidar unit. The method and apparatus can be configured to monitor the image and / or video data for the presence of objects in the environment. In other implementations, the environmental data can be radar, audio, or other data. The vehicle can be configured to identify objects in the environment based on the radar, audio, or other data.

[0096] In some implementations, the techniques used by the vehicle to detect objects can be based on a set of known data. For example, data related to environmental objects can be stored in a memory located in the vehicle. The vehicle can compare the received data to the stored data to determine the object. In other implementations, the vehicle can be configured to determine the object based on the context of the data. For example, street signs related to construction typically have an orange color. Thus, the vehicle can be configured to detect orange objects, and located near the side of the road as a street sign related to construction. In addition, when the processing system of the vehicle detects an object in the captured data, it can also compute a confidence for each object.

[0097] III. Example Vehicle Sensor Fields of View

[0098] Figure 4 FIG. illustrates an example autonomous vehicle 400 with various sensor fields of view. As discussed previously with respect to FIG. 1, the vehicle 400 can include multiple sensors. The positions of the various sensors can correspond to the positions of the sensors disclosed in FIG. 2. However, in some cases, the sensors can have other positions. To simplify the figure, the positions of the sensors are omitted in FIG. 4. For each sensor unit of the vehicle 400, a corresponding field of view is shown. The field of view of a sensor can include an angular region in which the sensor can detect objects and a range corresponding to a maximum distance from the sensor at which the sensor can reliably detect objects. Figure 2 Figure 2 Figure 4 Figure 4 As discussed previously, the vehicle 400 can include six radar units. A first radar unit can be located at the front left of the vehicle and have an angular field of view corresponding to an angular portion 402A of the field of view. A second radar unit can be located at the front right of the vehicle and have an angular field of view corresponding to an angular portion 402B of the field of view. A third radar unit can be located at the back left of the vehicle and have an angular field of view corresponding to an angular portion 402C of the field of view. A fourth radar unit can be located at the back right of the vehicle and have an angular field of view corresponding to an angular portion 402D of the field of view. A fifth radar unit can be located at the left side of the vehicle and have an angular field of view corresponding to an angular portion 402E of the field of view. A sixth radar unit can be located at the right side of the vehicle and have an angular field of view corresponding to an angular portion 402F of the field of view. Each of the six radar units can be configured with a scannable beam width of, for example, 90 degrees or more. The radar beam width can be less than 90 degrees, but each radar unit can be capable of controlling the radar beam to span the entire field of view.

[0099] As discussed previously, the vehicle 400 can include six radar units. A first radar unit can be located at the front left of the vehicle and have an angular field of view corresponding to an angular portion 402A of the field of view. A second radar unit can be located at the front right of the vehicle and have an angular field of view corresponding to an angular portion 402B of the field of view. A third radar unit can be located at the back left of the vehicle and have an angular field of view corresponding to an angular portion 402C of the field of view. A fourth radar unit can be located at the back right of the vehicle and have an angular field of view corresponding to an angular portion 402D of the field of view. A fifth radar unit can be located at the left side of the vehicle and have an angular field of view corresponding to an angular portion 402E of the field of view. A sixth radar unit can be located at the right side of the vehicle and have an angular field of view corresponding to an angular portion 402F of the field of view. Each of the six radar units can be configured with a scannable beam width of, for example, 90 degrees or more. The radar beam width can be less than 90 degrees, but each radar unit can be capable of controlling the radar beam to span the entire field of view.

[0100] A first lidar unit of the vehicle 400 can be configured to scan a full 360-degree region around the vehicle or within a full 360-degree region, as shown by an angular field of view corresponding to an angular portion 404 of the field of view. A second lidar unit of the vehicle 400 can be configured to scan a region that is less than a 360-degree region around the vehicle. In one example, the second lidar unit can have a field of view of 5 to 15 degrees in the horizontal plane, as shown by an angular field of view corresponding to an angular portion 404 of the field of view.

[0101] In addition, the vehicle can also include at least one camera. The camera can be an optical camera and / or an infrared camera. The camera can have an angular field of view corresponding to an angular portion 408 of the field of view.

[0102] ​​​In addition to the field of view of each of the various sensors of the vehicle 400, each sensor can also have a corresponding range. In one example, the range of the radar units can be greater than the range of any of the lidar units, as shown by the field of view of the radar units 402A-402E extending farther than the field of view of the lidar units 404 and 406. Additionally, a first lidar unit can have a greater range than the range of a second lidar unit, as shown by the field of view 404 extending farther than the field of view 406. The camera can have a range shown by the range of the field of view 408. In various examples, the range of the camera can be greater than or less than the range of the other sensors.

[0103] It should be understood that Figure 4 The sensor fields of view, radar units, etc. in FIG. 4 are depicted as example illustrations and are not drawn to scale.

[0104] IV. Example Systems and Methods

[0105] Example systems and methods of the present disclosure will now be described in further detail.

[0106] Figure 5 FIG. 5 is a flow diagram of a method 500 in accordance with example embodiments. The method 500 can include one or more operations, functions, or actions as illustrated in one or more of blocks 502-508. Although the blocks of each method are illustrated in sequential order, these blocks can in some cases be performed in parallel, and / or in a different order than that which is described herein. Also, various blocks can be combined into fewer blocks, divided into additional blocks, and / or removed altogether, based on the desires of one desiring to implement this method.

[0107] Further, for the method 500, as well as other processes and methods disclosed herein, the flow diagrams illustrate the functionality and operations of one possible implementation of current embodiments. In this regard, each block can represent a module, a segment, a procedure, or a portion of program code, which includes one or more instructions executable by a processor for implementing specific logical functions or steps in the process. The program code can be stored on any type of computer readable medium, for example, such as a storage device including a disk or hard drive. The computer readable medium can include non-transitory computer readable medium, for example, such as computer readable media that stores data for short periods of time like register memory, processor cache and Random Access Memory (RAM). The computer readable medium can also include non-transitory media, such as secondary or persistent long term storage like read only memory (ROM), optical or magnetic disks, compact disks read-only memories (CD-ROMs). The computer readable media can also be any other volatile or non-volatile storage systems. For example, the computer readable media can be considered computer readable storage media, or a tangible storage device.

[0108] Additionally or alternatively, for the method 500, as well as other processes and methods disclosed herein, one or more of the blocks in the flowchart can represent circuitry that was wired to perform the specific logical functions in the process.

[0109] In some examples, for the method 500, as well as other processes and methods disclosed herein, the functions described in the flowcharts can be performed by a single vehicle (e.g., vehicles 100, 200, etc.), distributed across multiple vehicles, performed by a remote server / external computing system (e.g., systems 302 and 306), and / or a combination of one or more external computing systems and one or more vehicles, among other possibilities. Moreover, the functions described in the flowcharts can be performed by one or more processors of a vehicle control system and / or one or more chips that control the operation of one or more vehicle sensors.

[0110] At block 502, the method 500 involves receiving, from one or more sensors associated with an autonomous vehicle, sensor data associated with a target object in an environment of the autonomous vehicle during a first environmental condition, wherein at least one sensor of the one or more sensors is configurable to be associated with one of a plurality of operational field of view volumes, and wherein each operational field of view volume represents a space within which the at least one sensor is expected to detect objects outside of the autonomous vehicle with a minimum level of confidence.

[0111] At block 504, the method 500 involves determining, based on the sensor data, at least one parameter associated with the target object.

[0112] At block 506, the method 500 involves determining a degradation of the at least one parameter between the sensor data and past sensor data associated with the target object in the environment during a second environmental condition different from the first environmental condition.

[0113] At block 508, the method 500 includes adjusting the operational field of view volume of the at least one sensor to a different one of the plurality of operational field of view volumes based on the determined degradation of the at least one parameter.

[0114] In some embodiments, the method 500 can be repeated across multiple consecutive camera images (i.e., frames) of the target object and / or across multiple consecutive instances of lidar data capturing associated with the target object, and can be performed using the same one or more sensors or using other sensors. Such repetition can help to validate the decision of which operational field of view volume to select, and can help to check whether the determined degradation is due to a change in environmental conditions or other factors (e.g., sensor drift).

[0115] The one or more sensors involved in the method 500 can include a set of one or more lidar sensors, a set of one or more radar sensors, and / or a set of one or more cameras (operating in various wavebands including visible light and infrared), among other possible sensor types. Indeed, all sensors of a particular type can be configured to have the same operational field of view volume, such that the vehicle software receiving and processing the sensor data is configured to treat all sensors of the particular type as having that operational field of view volume. For example, in clear weather, daytime environmental conditions, all cameras of the vehicle can have an operational field of view range of 150 meters, and all lidar sensors of the vehicle can have an operational field of view range of 200 meters. In this manner, an act of adjusting the field of view volume for at least one sensor of the vehicle can involve making the same field of view adjustment to each sensor of a particular sensor type. For example, if the vehicle system determines to make an adjustment to a lidar sensor based on the operational environment of the vehicle, the vehicle system can make the adjustment to all lidar sensors of the vehicle. Other examples are possible as well. Moreover, in alternative embodiments, the operational sensor field of view volume can be configured individually, such that a sensor of a particular sensor type can be configured to have a different operational field of view volume than another sensor of the same sensor type.

[0116] The one or more sensors can include one or more sensors mounted at one or more locations relative to the vehicle. For example, in some embodiments, the one or more sensors can consist of sensors mounted on the vehicle, such as a camera mounted at one location on the vehicle and a lidar sensor mounted at a different location on the vehicle, two or more cameras mounted at different locations on the vehicle, or two or more lidar sensors mounted at different locations on the vehicle. In other embodiments, the one or more sensors can include multiple sensors, at least one of which is mounted on the vehicle and at least another of which is mounted on a different vehicle. In yet another embodiment, at least one of the one or more sensors can be mounted on a stationary object along a road on which the vehicle is traveling or will travel. The stationary object can be a utility pole, a road sign (e.g., a stop sign), a traffic light, or a building, among other possibilities. In embodiments in which at least one sensor is remote from the vehicle, a server or other computing device can be used to facilitate communication of sensor data from the remote sensor to the vehicle system. For example, sensor data obtained by a sensor on a stationary object can be transmitted to the vehicle system via the server as the vehicle approaches or passes the stationary object.

[0117] The environmental conditions can be or include sunny weather (e.g., sunny day, not overcast, no rain, no snow, or no fog), daytime (e.g., the time period from sunrise to sunset), nighttime (e.g., the time period from sunset to sunrise), rainy day, snowy day, foggy day, overcast day (e.g., more clouds and less light), and / or sensor cleanliness conditions, where the vehicle has detected that one or more sensors of the vehicle have dust, water droplets, ice / frost, insect splatter, oil, road grime, or other substances obstructing their hoods / windows or other surfaces. Other environmental conditions are also possible. Further, the environmental conditions can be or include combinations of conditions. For example, the environmental conditions can be sunny weather, a sunny daytime environmental condition (e.g., 11:00 AM and sunny). As another example, the operating environment can be a foggy daytime environmental condition, or a rainy daytime environmental condition. Further, in some examples, there can be environmental conditions with varying degrees of weather conditions, such as severe blizzards and / or winds, severe rain and / or winds, or fog densities that exceed a predetermined threshold, among other possibilities.

[0118] The target object can be an object in the environment of the vehicle that has attributes that can have an expected impact on at least one parameter determined from sensor data associated with the target object. Examples of these attributes can include one or more particular materials that the target object is made of (e.g., wood), reflectivity of the surface of the target object, color of the target object, size of the target object, shape of the target object, whether the object is static or dynamic, and whether the target object has sharp corners that can impact the brightness of the target object at different viewing angles. In some embodiments, it can be desirable for the target object to be an object made of a material with low reflectivity (e.g., wood) and have a shape that is not likely to cause a change in the at least one parameter when sensor data of the object is obtained at different viewing angles (e.g., substantially circular, with minimal or no sharp corners). For example, the target object can be a wooden utility pole that has a lower expected reflectivity and is expected to have a higher contrast relative to the horizon in sunny weather conditions. Other example target objects are also possible, such as a building, a bridge, or other man-made objects.

[0119] In some embodiments, the target object can be an object or set of objects that is intentionally placed in the environment to assist the disclosed methods. For example, the target object can be manufactured to have a black, minimally reflective surface and a simple circular design. Additionally or alternatively, the target object can be equipped with one or more fiducial markers having known properties. In other embodiments, the target object can be an object that is not intentionally placed in the environment for the purpose of assisting the disclosed methods, but rather is an object that is inferred from the environment over time, such that a profile of the object can be established. For example, as different vehicles drive past a utility pole, sensor data (e.g., images or lidar intensity data) associated with the utility pole can be acquired and used to generate a statistical profile that indicates various properties of the object and that indicates how frequently the vehicle encounters the object. For example, the statistical profile can indicate whether the object is a static object that is not expected to move in the environment, or a dynamic object that moves in a predictable or unpredictable manner. The statistical profile can also indicate one or more optical properties of the object as a function of sensor position. Other examples are also possible.

[0120] When the properties of the target object are known, changes in parameters determined from sensor data associated with the target object can be attributed with higher confidence to changing environmental conditions (e.g., weather and / or time of day) rather than to uncertainty about the object and its properties.

[0121] The past sensor data can include sensor data received from at least one of the one or more sensors and / or sensor data received from other sensors or another source. The past sensor data can be received at a time during or after a time period in which the statistical profile of the target object is being formed.

[0122] As described above, the past sensor data is associated with the target object in the environment during a second environmental condition that is different from the first environmental condition. For example, the past sensor data can include images of the target object or lidar data representing the target object during sunny weather conditions and during a particular time of day (e.g., 10:00 AM), while the sensor data can include images of the target object or lidar data representing the target object during rainy, foggy, or snowy conditions and during the same time of day (e.g., 10:00 AM) or during the same part of the day (e.g., morning, day) as the past sensor data.

[0123] The sensor data can include one or more images from one or more cameras, lidar data (e.g., 3D point cloud data, including point intensities) from one or more lidar sensors, and / or other types of sensor data from other types of sensors. Similarly, the past sensor data can include one or more past images from one or more cameras, past lidar data (e.g., 3D point cloud data, including point intensities) from one or more lidar sensors, and / or other types of past sensor data from other types of sensors.

[0124] In some embodiments, the at least one parameter determined by the vehicle system based on the sensor data can include a value comparing the target object and another region of the environment represented in the sensor data. For example, where the sensor data includes one or more images, the at least one parameter can include a contrast between the target object depicted in the one or more images and a horizon depicted in the one or more images or other value comparing the target object depicted in the one or more images and the horizon depicted in the one or more images. Using contrast can be a reliable indicator of when the field of operation should be adjusted as contrast decreases with distance in snowy or foggy weather.

[0125] Additionally or alternatively, the at least one parameter can include a value comparing an edge of the target object in the one or more images and another region of the one or more images, such as a region depicting a horizon, a road, or another object. For example, the at least one parameter can include a ratio between an edge intensity of one or more edges of the target object depicted in the one or more images and an edge intensity of one or more edges of another object or other value comparing the edge intensity of the one or more edges of the target object depicted in the one or more images and the edge intensity of the one or more edges of another object.

[0126] In other embodiments, the at least one parameter can include a lidar intensity from a laser beam reflected off the target object and / or a number of laser beams reflected off the target object. In other embodiments, the at least one parameter can include a value comparing the lidar intensity or number of laser beams reflected off the target object and a lidar intensity or number of laser beams reflected off another object in the environment, such as a road, a building, or another vehicle.

[0127] Example techniques for determining degradation of the at least one parameter will now be described in more detail with reference to Figure 6A- Figure 6B and Figure 7A- Figure 7B Example techniques for determining degradation of the at least one parameter will now be described in more detail with reference to

[0128] Figure 6A and 6BAn example image is depicted that can be used to determine a degradation in contrast or other value of a comparison target object to a horizon depicted in the image. The horizon can be a reliable indicator of degradation, although other regions in the image can be used instead.

[0129] Figure 6A Depicted are image 600 and past image 602, each depicting target object 604 and horizon 606. In particular, target object 604 in images 600, 602 can be a tree line. Image 600 depicts target object 604 and horizon 606 under daytime snowy weather conditions, and past image 602 depicts target object 604 and horizon 606 under daytime clear weather conditions.

[0130] Figure 6A Also depicted is a segmented version 608 of past image 602, in particular an image that has been divided into a plurality of rectangular azimuth / elevation angle regions. These regions are representative examples of how the image can be used to determine a degradation of at least one parameter, and it should be understood that other shapes, sizes, and numbers of regions can be used in alternative examples. For example, the vehicle system can use only regions of images 600, 602 that include at least a portion of target object 604. Furthermore, in some examples, the vehicle system can be configured to ignore or remove regions that correspond to dynamic objects in the environment, such as cars or clouds. Furthermore, in other examples, the vehicle system can select which regions to use based on predetermined map data that indicates a known location of target object 604.

[0131] In an example process for determining a degradation in contrast between target object 604 and horizon 606, the vehicle system can compute a Fourier transform 610 of image 600, and can compute a Fourier transform 612 of past image 602. Fourier transform 610 and Fourier transform 612 are each shown segmented in a similar manner to segmented version 608 of past image 602, such that each region in the Fourier transform takes the form of a power spectral density map in azimuth / elevation angle of the corresponding region of the respective original image. In some examples, each of the Fourier transforms can be cosine-tapered.

[0132] Next, the vehicle system can compute a ratio of Fourier transform 610 to Fourier transform 612. In particular, the vehicle system can compute a ratio of each of one or more rectangular regions in Fourier transform 610 to a corresponding region in Fourier transform 612, including but not limited to one or more regions that correspond to target object 604. As an example, Figure 6A Depicted is image 614 that indicates the ratio of each region in Fourier transform 610 to each corresponding region in Fourier transform 612. Figure 6B An enlarged version of image 614 is depicted.Figure 6A and Figure 6B A bounding box highlighting a particular region of interest 616 in the images 600, 602, Fourier transforms 610, 612, and image 614 is also depicted. Specifically, the region of interest 616 includes a region of interest corresponding to the intersection of the tree line with the horizon 606 in the images 600, 602, and other regions bordering the region of interest.

[0133] From the image 614, the vehicle system can be configured to identify which regions indicate the greatest difference between the image 600 and the past image 602, i.e., which regions have the highest ratio, and use that / those regions to determine how to adjust the operational field of view volume. For example, Figure 6B The region 618 shown in FIG. 6B can indicate the strongest degree of degradation (e.g., approximately four orders of magnitude of contrast - proxy degradation (-40 dB), as shown).

[0134] Based on the computed ratio(s), and further based on the distance of the target object 604 to the camera(s) from which the images 600, 602 were received, the vehicle system can determine the degradation of contrast. The range of the target object 604 can be known to the vehicle system, such as from pre-existing map data, GPS data, and / or other data, or can be estimated using data acquired with one or more sensors, such as the clear weather past image 602 and / or lidar data of one or more objects in the past image 602.

[0135] In some examples, the degradation of contrast can be represented by Equation 1, where C_o is the contrast between the target object 604 and the horizon 606 under snowing (or foggy) conditions (e.g., C_o = 1 for a black target object), k is the extinction length, and d is the distance to the target object 604.

[0136] C = C_o * exp(-k * d) (Equation 1)

[0137] Thus, for each region, the natural logarithm of the ratio between C and C_o can provide k * d for that region, from which k can be estimated using a known value of d. For example, if the region 618 (-40 dB) is approximately 1000 meters below the road, k can be determined as shown in Equations 2-5.

[0138] 20 * log10(C / C_o) = 40 (Equation 2)

[0139] C / C_o = 100 (Equation 3)

[0140] ln(C / C_o) = ln(100) = 4.6 = k * d (Equation 4)

[0141] k = 4.6 / (1000 meters) = 0.0046 (Equation 5)

[0142] In other embodiments, the Fourier transforms 610, 612 can be compared in alternative ways other than a ratio.

[0143] Figure 6A- Figure 6B Some regions (e.g., pixels) in the illustrated images can correspond to respective distances between the object and the one or more cameras (e.g., pixel 250 can correspond to the ground approximately 50-70 meters away from the one or more cameras). Thus, in another example process for determining a degradation of at least one parameter, the vehicle system can be configured to average values across multiple regions, each corresponding to approximately the same distance, i.e., the distance at which the target object 604 is estimated or known to be located.

[0144] To facilitate this, for example, the vehicle system can be configured to compute a plurality of ratios of (i) a first plurality of regions in the Fourier transform 610 corresponding to the target object 604 and further corresponding to other portions of the environment located approximately the same distance from the one or more cameras as the distance of the target object 604 to the one or more cameras and (ii) a second plurality of regions in the Fourier transform 612 corresponding to the target object 604 and further corresponding to the other portions of the environment. Both the first plurality of regions and the second plurality of regions can have the same number of regions. For example, as illustrated, regions 618 and 620 can be used. Figure 6B

[0145] For example, for ease of approximation, the vehicle system can then compute an average ratio by summing the plurality of ratios and dividing the sum by the number of regions. For example, the vehicle system can take an average signal-to-noise ratio of all regions containing pixel 250 (e.g., regions in the same row as region 618, such as region 620). Based on the average ratio, and further based on the distance of the target object 604 to the one or more cameras from which the images 600, 602 were received, the vehicle system can determine a degradation of contrast, such as by using the techniques described above with respect to Equation 1.

[0146] In some examples, the vehicle system can use multiple regions having target objects at different ranges. For example, the vehicle system can compute k for region 618, and can compute k for region 622, where region 622 can correspond to another target object in the environment, such as another tree within a known range approximately equal to the distance to the target object 604. If the snow (or fog, etc. in other cases) is substantially uniform in density, the computed value of k for region 622 can be approximately the same as the computed value of k for region 618.

[0147] ​As a supplement or alternative to using the above techniques, in some examples the vehicle system can implement an edge detection technique to determine the degradation of the at least one parameter.

[0148] In an example process of using an edge detection technique to determine the degradation of the contrast between the target object 604 and the horizon 606, the vehicle system can convolve a two-dimensional edge detection kernel (e.g., an edge detection kernel with a Difference of Gaussians (DoG) filter) with the one or more images to determine a first edge strength in the one or more images, and can convolve the two-dimensional edge detection kernel with the one or more past images to determine a second edge strength in the one or more past images. Figure 7A The convolved image 700 and past image 702 are shown, with the image 700 depicting the target object 604 and the horizon 606 in a snowy weather condition during the day, and the past image 702 depicting the target object 604 and the horizon 606 in a clear weather condition during the day. As in the previous example, the target object 604 is a tree line, although other objects can be used in other examples. Figure 7A An example reference past image 704 before convolution is also depicted.

[0149] Next, the vehicle system can sum the first edge strength in each of the one or more first regions corresponding to the target object in the one or more images, and can sum the second edge strength in each of the one or more second regions corresponding to the target object in the one or more past images. For example, Figure 7A A segmented version 706 of the image 700 and a segmented version 708 of the past image 702 are depicted. In the illustrated example, each image 706, 708 has been divided into a plurality of rectangular azimuth / elevation regions, and the edge strength in each region has been summed. It will be appreciated that other shapes, sizes, and numbers of regions can be used in alternative examples.

[0150] Figure 7A And Figure 7B A bounding box highlighting a particular region of interest 710 is also depicted. In particular, the region of interest 710 includes the region of interest corresponding to the intersection of the tree line (including the target object 604) with the horizon 606 in the images 700, 702, as well as other regions bordering the region of interest. In some examples, the vehicle system can sum the edge strength only in the particular region of interest of the image, rather than summing over the entire image.

[0151] The vehicle system can then calculate one or more ratios of the summed first edge strength in each of the one or more first regions to the summed second edge strength in each of the one or more second regions. As one example, Figure 7AAn image 712 depicting the ratio of each region in the image 708 to each corresponding region in the image 706. Figure 7B An enlarged version of the image 712 is depicted.

[0152] From the image 712, the vehicle system can be configured to identify which regions indicate the greatest difference between the image 700 and the past image 702, i.e., which regions have the highest ratios, and use that / those region(s) to determine how to adjust the operational field of view volume. For example, Figure 7B The regions 714 shown can indicate the strongest degree of degradation.

[0153] Based on the computed ratio(s), and further based on the distance of the target object 604 to the camera(s) from which the images 700, 702 were received, the vehicle system can determine a degradation in contrast, such as by using the techniques described above with respect to Equation 1.

[0154] In other embodiments, the images 706, 708 can be compared in alternative ways, in addition to the ratios.

[0155] In some examples, the vehicle system can be configured to average over multiple regions of the images 706 and 708, each corresponding to approximately the same distance, i.e., the distance at which the target object 604 is estimated or known to be located.

[0156] To facilitate this, for example, the vehicle system can be configured to: (i) sum the first edge intensities in each of a first plurality of regions of the image 700, the first plurality of regions corresponding to the target object 604 and further corresponding to other portions of the environment located at approximately the same distance to the camera(s) as the target object 604 to the camera(s); and (ii) sum the second edge intensities in each of a second plurality of regions of the past image 702, the second plurality of regions corresponding to the target object 604 and further corresponding to the other portions of the environment. The first and second pluralities of regions can have the same number of regions.

[0157] The vehicle system can then compute a plurality of ratios of the aggregate first edge intensities in the first plurality of regions to the aggregate second edge intensities in the second plurality of regions. For example, the vehicle system can compute the ratio of at least a portion of the regions shown in the image 706 (each representing the sum of the first edge intensities in that region of the image 700) to at least a portion of the corresponding regions in the image 708 (each representing the sum of the second edge intensities in that region of the image 712).

[0158] The vehicle system can then compute an average ratio as a sum of the plurality of ratios divided by the number of regions. For example, the vehicle system can take an average signal-to-noise ratio of all regions containing pixel 250 (e.g., regions in the same row as region 710). Based on the average ratio, and further based on a distance of the target object 604 to the one or more cameras from which the images 700, 702 were received, the vehicle system can determine a degradation in contrast, such as by using the techniques described above with respect to Equation 1.

[0159] In some embodiments, the vehicle system can be configured to determine a degradation in edge strength between one or more regions of one or more images (e.g., one or more regions of image 700 representing target object 604) and one or more corresponding regions of one or more past images (e.g., one or more regions of past image 702 representing target object 604) without performing operations to determine a degradation in contrast. The vehicle system can then use the determined degradation in edge strength as a basis for adjusting an operational field of view volume of at least one sensor of the vehicle.

[0160] In some embodiments, as described above, the at least one parameter can include lidar intensity of a laser beam reflected from a target object in the environment. The vehicle system can use lidar data and past lidar data to determine a degradation in lidar intensity of the laser beam reflected from the target object between the lidar data and the past lidar data (e.g., a degradation expressed in dB / meter). The vehicle system can then use the determined degradation in lidar intensity as a basis for adjusting a field of view volume of at least one of the one or more lidar sensors of the vehicle. In some examples, for a given degradation in lidar intensity, a maximum operational field of view range of a lidar sensor can be reduced by a factor equal to the square root of the degradation.

[0161] As described above, a degradation parameter (or parameters) determined from camera images can be used as a basis for adjusting an operational field of view volume of at least one lidar sensor. Similarly, a degradation parameter (or parameters) determined from lidar data can be used as a basis for adjusting an operational field of view of at least one camera. One reason for this can be that the contrast between a target object and the horizon degrades in a similar manner when light returns from the target object. For example, in some cases, a distance from a lidar sensor at which approximately 95% of lidar intensity is lost due to exponential decay can be approximately the same as a distance at which contrast between a target object on the horizon (ideally, a black target object) exponentially decays to approximately 5%.

[0162] Accordingly, the act of adjusting the field of view volume of each of the one or more sensors based on the determined degradation of at least one parameter can involve adjusting the field of view volume of each of at least one of the one or more cameras and each of the one or more lidar sensors based on the determined degradation of lidar intensity between the lidar data and the past lidar data. Additionally or alternatively, the act of adjusting the field of view volume of each of the one or more sensors based on the determined degradation of at least one parameter can include adjusting the field of view volume of each of at least one of the one or more cameras and each of the one or more lidar sensors based on the determined degradation of contrast, edge intensity, and / or another parameter value between the one or more images and the one or more past images.

[0163] As an example, once k is calculated for contrast degradation using camera images as described above, the vehicle system can calculate how much the signal from the one or more lidar sensors is attenuated, as k for camera visible wavelengths can be similar to k for lidar wavelengths. For example, in conditions of snow or rain, the intensity of the received light can degrade by a factor of exp(-2k*d), where d is the distance to the target object, compared to clear weather conditions where k is approximately zero. The vehicle system can then use this degradation to infer the degraded field of view volume of the one or more lidar sensors in the direction of the target object.

[0164] To facilitate the act of adjusting the operational field of view volume of the at least one sensor to a different one of a plurality of operational field of view volumes based on the determined degradation of at least one parameter, the vehicle system can store in memory (e.g., data store 114) a table or other form of data that maps each of one or more parameter values (e.g., contrast, lidar intensity, edge intensity, etc.) to each of a respective finite / predetermined number of operational field of view volumes. Accordingly, when the vehicle system determines the degradation, the vehicle system can select a different one of the plurality of operational field of view volumes, in particular one that corresponds to the degraded parameter value. As another example, the vehicle system can determine a new range, azimuth, and / or elevation of the operational field of view volume based on the determined degradation. Further, the table or other form of data can also map each of the finite / predetermined number of operational field of view volumes and / or each of the corresponding one or more parameter values to a label that identifies an environmental condition (e.g., snow, fog, or clear) or combination of environmental conditions (e.g., daytime and snow, or daytime and fog). The memory in which the table or other data is stored and from which the table or other data is accessed can be local to the vehicle (e.g., in memory on the vehicle) or remote from the vehicle (e.g., a database accessible by a server).

[0165] In some embodiments, the corresponding finite / predetermined number of operational field of view volumes can include multiple sets of operational field of view volumes corresponding to a given one of the one or more parameter values, each set having a finite / predetermined number of operational field of view volumes associated with a given type of sensor (e.g., lidar sensor) or a particular one of the one or more sensors (e.g., a lidar sensor on the left side of the vehicle). For example, to facilitate the action of adjusting the operational field of view volume of a first type of sensor based on a degradation of at least one parameter determined from sensor data of a second sensor of a different type, the parameter values can be mapped to two sets of operational field of view volumes, one set for the first type of sensor (e.g., camera) and the other set for the second type of sensor (e.g., lidar sensor).

[0166] In some embodiments, the sensor data used to determine the degradation can not be sensor data acquired by a sensor mounted on the vehicle, but the determined degradation can be used as a basis for adjusting the operational field of view volume of at least one sensor mounted on the vehicle. For example, a camera can be mounted to another vehicle or a traffic light at an intersection, and can acquire an image, which a computing device other than the vehicle system (e.g., a server of another vehicle or the vehicle system) can use to determine a decrease in contrast. The computing device can then send a signal to the vehicle system indicating the determined degradation, so that the vehicle system can then use the determined degradation as a basis for adjusting the operational field of view of one or more cameras and / or lidar sensors mounted on the vehicle. Alternatively, the computing device can send the image itself to the vehicle system, and the vehicle system can determine the degradation. Other examples are also possible.

[0167] In adjusting the operational field of view volume of the at least one sensor, the vehicle system can control the vehicle to operate using the at least one sensor having the adjusted operational field of view volume. That is, the vehicle system can control the vehicle while operating in an autonomous mode to use the at least one sensor to acquire sensor data based on the adjusted operational field of view volume. In some embodiments, to facilitate this, the local computing system on the vehicle can set itself to disregard sensor data readings acquired during operation of the vehicle that exceed the respective range, azimuth, and / or elevation angle associated with the adjusted operational field of view volume of each of the at least one sensor. Additionally or alternatively, the remote system can send instructions to the local computing system of the vehicle that, when received by the local computing system, cause the local computing system to control the vehicle to operate in an autonomous mode in which the local computing system disregards sensor data readings that exceed the respective range, azimuth, and / or elevation angle associated with the adjusted operational field of view volume of each of the at least one sensor. Other examples are possible as well. As noted above, the at least one sensor having the adjusted operational field of view volume can include at least one sensor mounted on the vehicle and / or at least one sensor remote from the vehicle but still used by the vehicle system to facilitate operation of the vehicle.

[0168] In some embodiments, even if a particular operating field of view volume for a given sensor can be less than the maximum operating field of view value for that sensor and parameter at a given point in time, the sensor can still be configured to acquire and send to the vehicle system (e.g., to a processor configured to process sensor data) sensor data corresponding to ranges, azimuths, and / or elevations that are outside of the respective ranges, azimuths, and / or elevations associated with that particular operating field of view volume. In such embodiments, the vehicle system can ignore (e.g., discard or store but not use as a basis for determining the vehicle’s environment, such as object detection) sensor data corresponding to ranges, azimuths, and / or elevations that are greater than the respective ranges, azimuths, and / or elevations associated with the particular operating field of view. For example, if the range of a lidar sensor has been reduced from 200 meters to 150 meters, the vehicle system can ignore sensor data corresponding to ranges that are more than 150 meters from the vehicle. Other examples are also possible. Additionally or alternatively, the vehicle system can identify (e.g., flag or otherwise store in memory an indication that the data can be suspect) sensor data corresponding to parameter values that are greater than the maximum parameter value for the particular operating field of view volume. In alternative embodiments, such a sensor can be configured such that the sensor is able to set itself to not acquire sensor data corresponding to ranges, azimuths, and / or elevations that are outside of the respective ranges, azimuths, and / or elevations associated with the particular operating field of view volume. Additionally or alternatively, the sensor can be configured to acquire sensor data corresponding to ranges, azimuths, and / or elevations that are outside of the respective ranges, azimuths, and / or elevations associated with the particular operating field of view, but also configured to discard such sensor data in order to reduce the amount of data sent from the sensor to other computing devices of the vehicle system.

[0169] In some embodiments, a sensor of a vehicle and an associated computing device, such as a chip (e.g., microchip) that controls operation of the one or more sensors, can perform operations prior to the sensor sending acquired sensor data to an on-board computer or a remote computer, which can influence how the on-board computer or the remote computer controls operation of the vehicle. In particular, such a sensor chip can perform one or more operations of the method 500. In such a context, the act of adjusting the operational field of view volume can involve the sensor chip ignoring or flagging sensor data corresponding to ranges, azimuths, and / or elevations that are greater than respective ranges, azimuths, and / or elevations associated with the adjusted operational field of view volume. Additionally or alternatively, the act of adjusting the operational field of view volume can involve the sensor chip (i) adjusting a power level of laser pulses sent by the one or more lidar sensors from a first power level to an adjusted power level that is different from the first power level when acquiring the sensor data, and / or (ii) acquiring the sensor data by sending one or more laser pulses at the adjusted power level associated with the adjusted operational field of view volume. Other examples are possible as well.

[0170] The terms “substantially,” “approximately,” or “about” as used herein mean that the recited characteristic, parameter, or value need not be achieved exactly, but that deviations or variations, including tolerances, measurement error, measurement accuracy limitations, and other factors known to those of skill in the art, can occur in the quantity to the extent that such deviations do not preclude the effect the characteristic was intended to provide.

[0171] While various example aspects and example embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art based on the disclosure herein. The various example aspects and example embodiments disclosed herein are for purposes of illustration and are not intended to limit the true scope and spirit of the disclosure. The appended claims are intended to cover all such aspects and embodiments.

Claims

1. A method comprising: Sensor data associated with a target object in the environment of the autonomous vehicle during a first environmental condition is received from one or more sensors associated with the autonomous vehicle, wherein at least one of the one or more sensors is configured to be associated with one of a plurality of operational field-of-view volumes, wherein each operational field-of-view volume represents a space in which at least one sensor is expected to detect an object outside the autonomous vehicle at a minimum confidence level, and wherein a particular sensor of the at least one sensor is currently associated with a first operational field-of-view volume of the plurality of operational field-of-view volumes; Based on sensor data, determine at least one parameter associated with the target object; Determine the degradation of at least one parameter between sensor data and past sensor data, wherein the past sensor data was associated with a target object in the environment during a second environmental condition different from the first environmental condition; and Based on the degradation of at least one determined parameter, the operating field of view volume of at least one sensor is adjusted to a different one of a plurality of operating field of view volumes, wherein adjusting the operating field of view volume of at least one sensor includes: Based on the degradation of at least one determined parameter, a second operational field of view volume is selected from a plurality of operational field of view volumes for a specific sensor, wherein the second operational field of view volume is different from the first operational field of view volume; and Adjust the operating field volume of a specific sensor to the second operating field volume.

2. The method according to claim 1, wherein, The degradation is determined based on sensor data from a first sensor among one or more sensors, and In this context, a specific sensor is the second sensor among one or more sensors.

3. The method according to claim 2, wherein, The first sensor is a camera, and The second sensor is a lidar sensor.

4. The method according to claim 2, wherein, The first sensor is a lidar sensor, and The second sensor is a camera.

5. The method according to claim 2, wherein, The first and second sensors are the same type of sensors located at different positions relative to the vehicle.

6. The method according to claim 1, wherein, The first environmental condition is one or more of the following: rain, fog, or snow. The second environmental condition is clear weather.

7. The method according to claim 1, wherein, Sensor data and past sensor data are each associated with the target object at the same time of day or during the same part of the day.

8. The method according to claim 1, wherein, One or more sensors include one or more cameras. The sensor data includes one or more images. The past sensor data includes one or more past images, and At least one parameter includes the value of comparing the target object with the horizon depicted in one or more images.

9. The method according to claim 8, wherein, The value mentioned is the contrast ratio.

10. The method of claim 9, further comprising: Calculate the first Fourier transform of one or more images; Calculate the second Fourier transform of one or more past images; as well as Calculate the ratio of one or more first regions corresponding to the target object in the first Fourier transform to one or more second regions corresponding to the target object in the second Fourier transform. Determining the degradation of at least one parameter between sensor data and past sensor data includes determining the degradation of contrast between the target object and the horizon based on one or more calculated ratios and further based on the distance of the target object to one or more cameras.

11. The method of claim 9, further comprising: Calculate the first Fourier transform of one or more images; Calculate the second Fourier transform of one or more past images; Calculate the ratio of (i) a first plurality of regions in a first Fourier transform that correspond to the target object and further to other parts of the environment located at distances to one or more cameras that are approximately equal to the distances from the target object to one or more cameras, to (ii) a second plurality of regions in a second Fourier transform that correspond to the target object and further to said other parts of the environment, wherein the first plurality of regions and the second plurality of regions have the same number of regions; and The average ratio is calculated as the sum of multiple ratios divided by the number of regions. Determining the degradation of at least one parameter between sensor data and past sensor data includes determining the degradation of contrast between the target object and the horizon based on the calculated average ratio and further based on the distance of the target object to one or more cameras.

12. The method of claim 9, further comprising: A two-dimensional edge detection kernel is convolved with one or more images to determine the first edge intensity in one or more images; The two-dimensional edge detection kernel is convolved with one or more past images to determine the intensity of a second edge in one or more past images; Sum the first edge intensities in each of one or more first regions corresponding to the target object in one or more images; Sum the second edge intensities in each of one or more second regions corresponding to the target object in one or more past images; as well as Calculate one or more ratios of the total first edge intensity in each of one or more first regions to the total second edge intensity in each of one or more second regions. Determining the degradation of at least one parameter between sensor data and past sensor data includes determining the degradation of contrast between the target object and the horizon based on one or more calculated ratios and further based on the distances from one or more cameras to the target object.

13. The method of claim 9, further comprising: A two-dimensional edge detection kernel is convolved with one or more images to determine the first edge intensity in one or more images; The two-dimensional edge detection kernel is convolved with one or more past images to determine the intensity of a second edge in one or more past images; The first edge intensities in each of the first plurality of regions of one or more images are summed, the first plurality of regions corresponding to the target object and further corresponding to other parts of the environment located at a distance to one or more cameras that is approximately the same as the distance from the target object to one or more cameras; The second edge intensities in each of the second plurality of regions of one or more past images are summed, the second plurality of regions corresponding to the target object and further to the other parts of the environment, wherein the first plurality of regions and the second plurality of regions have the same number of regions; Calculate multiple ratios between the total first edge intensity in the first plurality of regions and the total second edge intensity in the second plurality of regions; and The average ratio is calculated as the sum of multiple ratios divided by the number of regions. Determining the degradation of at least one parameter between sensor data and past sensor data includes determining the degradation of contrast between the target object and the horizon based on the calculated average ratio and further based on the distance of the target object to one or more cameras.

14. The method according to claim 1, wherein, One or more sensors include one or more cameras. The sensor data includes one or more images. The past sensor data includes one or more past images, and At least one parameter includes a value that compares the edge of a target object in one or more images with another region of one or more images.

15. The method according to claim 1, wherein, One or more sensors include one or more lidar sensors. The sensor data includes lidar data. This includes past sensor data, and At least one parameter includes the lidar intensity from the laser beam reflected from the target object.

16. The method according to claim 15, wherein, One or more sensors may also include one or more cameras. Specifically, determining the degradation of at least one parameter between the sensor data and past sensor data includes determining the degradation of the lidar intensity of the laser beam reflected from the target object between the lidar data and past lidar data. Adjusting the field of view volume of at least one sensor includes adjusting the field of view volume of at least one of one or more cameras and at least one of one or more lidar sensors based on the determined degradation of lidar intensity.

17. The method according to claim 1, wherein, One or more sensors associated with the autonomous vehicle include sensors mounted on the vehicle.

18. The method according to claim 1, wherein, At least one sensor is mounted on the autonomous vehicle, and one or more sensors associated with the autonomous vehicle also include at least one sensor mounted on different autonomous vehicles.

19. The method according to claim 1, wherein, One or more sensors associated with an autonomous vehicle include sensors mounted on a stationary object along a road on which the autonomous vehicle is traveling or will travel.

20. The method according to claim 1, wherein, In the past, sensor data was received from one or more sensors.

Images