Real-time adjustment of vehicle sensor field of view volume

Abstract

To provide a system and a method available for adjusting field of view volumes of one or more sensors of an autonomous vehicle.SOLUTION: In a system and a method, each sensor in one or more sensors is configured to operate in accordance with a field of view volume equal to or less than a maximum field of view volume. The system and the method include determining an operation environment of an autonomous vehicle. The system and the method also include based on the determined operation environment of the autonomous vehicle, adjusting a field of view volume of at least one sensor in the one or more sensors from a first field of view volume to an adjusted field of view volume which is different from the first field of view volume. In addition, the system and the method include controlling the autonomous vehicle so as to operate using at least one sensor having the adjusted field of view volume.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] Cross - reference to Related Applications This disclosure claims the benefit of U.S. Non - Provisional Application No. 17 / 002,092, filed Aug. 25, 2020, which claims the benefit of U.S. Provisional Application No. 62 / 952,879, filed Dec. 23, 2019, the contents of which are hereby incorporated by reference in their entirety.

Background Art

[0002] Vehicles can be configured to operate in an autonomous mode in which the vehicle navigates through an environment with little or no input from a driver. Such autonomous vehicles can include one or more systems (e.g., sensors and associated computing devices) configured to detect information about the environment in which the vehicle operates. The vehicle and its associated computer - implemented controller use the detected information to navigate through the environment. For example, if the system detects that the vehicle is approaching an obstacle, the controller adjusts the vehicle's direction control according to a determination by the computer - implemented controller to cause the vehicle to avoid the obstacle and proceed.

[0003] For example, an autonomous vehicle can include lasers, sonars, radars, cameras, thermal imaging devices, and other sensors that scan and / or record data about the vehicle's surroundings. Sensor data from one or more of these devices can be used to detect objects and their respective characteristics (position, shape, direction of travel, speed, etc.). This detection and identification is useful for the operation of the autonomous vehicle.

Summary of the Invention

[0004] In one example, this disclosure provides a system. The system includes one or more sensors, each of which is configured to operate in accordance with an increase in field of view volume, where the field of view volume represents the space around an autonomous vehicle in which the sensors are expected to detect an object at a confidence level higher than a predetermined confidence threshold. The system also includes one or more processors coupled to the one or more sensors. The system also includes memory coupled to the one or more processors, which, when executed by the one or more processors, stores instructions causing the one or more processors to perform an operation. The operation includes identifying a plurality of operational design domains (ODDs) of the autonomous vehicle, where each ODD includes at least one of environmental conditions, geographical conditions, temporal conditions, traffic conditions, or road conditions, and where each ODD is associated with a predetermined field of view volume of at least one of the sensors. The operation also includes associating the autonomous vehicle with a first ODD of the plurality of ODDs. The operation also includes detecting a change in the operating environment of the autonomous vehicle. The operation also includes, in response to the detection, associating the autonomous vehicle with a second ODD of the plurality of ODDs. The operation also includes operating at least one sensor using a predetermined field of view volume associated with the second ODD, depending on whether the autonomous vehicle is associated with the second ODD.

[0005] Some examples of this disclosure provide a method performed by a computing device configured to control the operation of an autonomous vehicle. The method includes identifying a plurality of operational design domains (ODDs) of the autonomous vehicle, each ODD comprising at least one of environmental conditions, geographical conditions, temporal conditions, traffic conditions, or road conditions, and each ODD being associated with a predetermined field of view volume of at least one sensor of one or more sensors, each of which is configured to operate according to the field of view volume, where the field of view volume represents the space around the autonomous vehicle in which the sensor is expected to detect an object at a confidence level higher than a predetermined confidence threshold. The method also includes associating the autonomous vehicle with a first ODD of the plurality of ODDs. The method also includes detecting a change in the operating environment of the autonomous vehicle. The method also includes, in response to the detection, associating the autonomous vehicle with a second ODD of the plurality of ODDs. The method also includes, in response to the autonomous vehicle being associated with the second ODD, as including operating at least one sensor using a predetermined field of view volume associated with the second ODD.

[0006] Some examples of this disclosure provide a non-temporary computer-readable storage medium in which program instructions causing one or more processors to perform an operation are stored when executed by one or more processors. The operation includes identifying a plurality of operational design domains (ODDs) of an autonomous vehicle, each ODD comprising at least one of environmental conditions, geographical conditions, temporal conditions, traffic conditions, or road conditions, and each ODD being associated with a predetermined field of view volume of at least one sensor of one or more sensors, each sensor of the one or more sensors being configured to operate according to the field of view volume, where the field of view volume represents the space around the autonomous vehicle in which the sensor is expected to detect an object at a confidence level higher than a predetermined confidence threshold. The operation also includes associating the autonomous vehicle with a first ODD of the plurality of ODDs. The operation also includes detecting a change in the operating environment of the autonomous vehicle. The operation also includes, in response to the detection, associating the autonomous vehicle with a second ODD of the plurality of ODDs. The operation also includes, in response to the autonomous vehicle being associated with the second ODD, associating at least one sensor using a predetermined field of view volume associated with the second ODD.

[0007] Some examples of this disclosure provide a system. The system includes one or more sensors, each of which is configured to operate according to a field of view volume. The system also includes one or more processors coupled to the one or more sensors. The system also includes a memory coupled to the one or more processors, which, when executed by the one or more processors, stores instructions causing the one or more processors to perform an operation. The operation includes determining the operating environment of the autonomous vehicle. The operation also includes adjusting the field of view volume of at least one of the one or more sensors from a first field of view volume to a modified field of view volume different from the first field of view volume, based on the determined operating environment of the autonomous vehicle.

[0008] Some examples of the present disclosure provide a method performed by a computing device configured to control the operation of an autonomous vehicle. The method includes determining the operating environment of the autonomous vehicle, wherein the autonomous vehicle is equipped with one or more sensors, and each of the one or more sensors is configured to operate according to a field of view volume. The method also includes adjusting the field of view volume of at least one of the one or more sensors from a first field of view volume to a modified field of view volume different from the first field of view volume, based on the determined operating environment of the autonomous vehicle. The method also includes controlling the autonomous vehicle to operate using at least one sensor having the modified field of view volume.

[0009] Some examples of the present disclosure provide a non-temporary computer-readable storage medium in which program instructions causing one or more processors to perform an operation are stored when executed by one or more processors. The operation includes determining the operating environment of an autonomous vehicle, wherein the autonomous vehicle is equipped with one or more sensors, and each of the one or more sensors is configured to operate according to a field of view volume. The operation also includes adjusting the field of view volume of at least one of the one or more sensors from a first field of view volume to a modified field of view volume different from the first field of view volume, based on the determined operating environment of the autonomous vehicle. The operation also includes controlling the autonomous vehicle to operate using at least one sensor having the modified field of view volume.

[0010] Some examples of this disclosure provide a method for operating a sensor mounted on an autonomous vehicle. This method includes operating the sensor of the autonomous vehicle according to a first field of view volume, the first field of view volume being associated with a first operating environment of the autonomous vehicle. The method also includes receiving data indicating a second operating environment of the autonomous vehicle, the second operating environment being associated with environmental conditions within the environment of the autonomous vehicle. The method also includes automatically adjusting the operation of the sensor to operate according to a second field of view volume, the second field of view volume being associated with a second operating environment.

[0011] Some examples of this disclosure provide a system for operating sensors mounted on an autonomous vehicle. The system includes a sensor and one or more processors coupled to the sensor. The system also includes a memory coupled to the one or more processors, which, when executed by the one or more processors, stores instructions causing the one or more processors to perform an operation. An operation includes operating the sensor of the autonomous vehicle according to a first field of view volume, the first field of view volume being associated with a first operating environment of the autonomous vehicle. An operation also includes receiving data indicating a second operating environment of the autonomous vehicle, the second operating environment being associated with environmental conditions within the environment of the autonomous vehicle. An operation also includes automatically adjusting the operation of the sensor to operate according to a second field of view volume, the second field of view volume being associated with a second operating environment.

[0012] Some examples of this disclosure provide a non-temporary computer-readable storage medium in which program instructions causing one or more processors to perform an operation are stored when executed by one or more processors. The operation includes operating a sensor mounted on an autonomous vehicle according to a first field of view volume, the first field of view volume being associated with a first operating environment of the autonomous vehicle. The operation also includes receiving data indicating a second operating environment of the autonomous vehicle, the second operating environment being associated with environmental conditions within the environment of the autonomous vehicle. The operation also includes automatically adjusting the operation of the sensor to operate according to a second field of view volume, the second field of view volume being associated with a second operating environment.

[0013] These and other embodiments, advantages, and alternatives will become apparent to those skilled in the art by reading the following detailed description and, as appropriate, referring to the accompanying drawings. [Brief explanation of the drawing]

[0014] [Figure 1] This is a functional block diagram illustrating an exemplary configuration of an autonomous vehicle. [Figure 2] An example of an autonomous vehicle's external view is shown. [Figure 3] This is a conceptual diagram of wireless communication between various computing systems related to autonomous vehicles. [Figure 4] An example of a sensor field of view setup is shown. [Figure 5] An exemplary method is shown. [Figure 6] This shows an example of mapping sensor field of view parameters (range) to the operating environment. [Figure 7] This example shows a communication system in which a vehicle communicates with local weather station servers and global weather station servers. [Figure 8] Another exemplary method is shown. [Figure 9] Another exemplary method is shown. [Modes for carrying out the invention]

[0015] Exemplary methods, devices, and systems are described herein. The terms “example” and “exemplary” are used herein to mean “serving as an example, case, or illustration.” No embodiment or feature described herein as “example” or “exemplary” should necessarily be construed as being preferable or advantageous to any other embodiment or feature. Other embodiments may be utilized and other modifications made without departing from the scope of the subject matter presented herein.

[0016] Therefore, the exemplary embodiments described herein are not intended to be limiting. The aspects of this disclosure can be arranged, replaced, combined, separated, and designed in a wide variety of different configurations, as generally described herein and illustrated in the figures, all of which are assumed herein.

[0017] Furthermore, unless otherwise specifically suggested by the context, the features illustrated in each figure can be used in combination with each other. Therefore, the figures should generally be considered as partial aspects of one or more overall embodiments, with the understanding that not all illustrated features are necessarily required for each embodiment.

[0018] I. Overview Many vehicles include various sensing systems to assist with vehicle navigation and control. Some vehicles may operate in a fully autonomous mode where human interaction is not used for operation, a semi-autonomous mode where human interaction is rarely used for operation, or a human-operated mode where a human operates the vehicle and sensors can assist the human. Sensors may be used to provide information about the area around the vehicle. This information can be used to identify features of the roadway and other objects near the vehicle (other vehicles, pedestrians, etc.).

[0019] A vehicle's sensor system can include, for example, a Light Detection and Ranging (LIDAR) system and a radar system. LIDAR uses laser pulses to measure the distance and velocity to an object that reflects the laser light. Radar uses radio waves to measure the distance and velocity to an object that reflects the radio waves. Data from the LIDAR and radar systems can be used, along with data from other sensors of the vehicle's sensor system, such as a camera in some cases, to determine where objects are located in the environment around the vehicle. A particular LIDAR sensor, radar sensor, and / or camera may each have a certain field of view. The field of view of a sensor can include one or more angular (or other shaped) regions where the sensor can detect an object, and an associated range corresponding to the maximum distance from the sensor at which the sensor can reliably detect an object within its field of view. In some cases, the associated range may vary depending on the various azimuth / elevation angles within the field of view. The values of the parameters that define this field of view, such as the range, azimuth, and elevation values, together form a volume that can be referred to as the field of view volume.

[0020] The field of view volume of a particular sensor can be considered as an accurate representation of the space where the particular sensor can consider an object to be detected with high reliability. In other words, one or more processors of the vehicle system (e.g., a chip that controls the operation of the sensor or a processor of the vehicle's control system) can be configured to confidently rely on sensor data obtained within the space defined by the sensor's field of view volume. For example, a processor associated with a particular sensor may be configured to associate a higher reliability level (e.g., higher than a pre-defined reliability threshold level) to an object or other information detected at a range, azimuth, and / or elevation within the field of view volume of that sensor, and a lower reliability level (e.g., below a pre-defined reliability threshold level) to an object or other information detected at a range, azimuth, and / or elevation outside of that field of view volume.

[0021] Vehicles can be exposed to various conditions during operation, such as changes in weather (fog, rain, snow, etc.), time of day, speed limits, terrain or other geographical conditions, changes in the place of residence (e.g., city, suburbs, countryside), changes in the number of other vehicles or objects proximate to the vehicle, other changes outside the vehicle, and / or changes within the vehicle's system (e.g., sensor errors, cleanliness of the sensor surface, malfunctions of vehicle subsystems, etc.). At any given time, one or more of these or other conditions may exist in the vehicle's operating environment. In the context of this disclosure, the "operating environment" of a vehicle is one or more internal and / or external conditions of the vehicle that can change over time, including but not limited to the conditions described above and other conditions described elsewhere in this disclosure. Thus, when one or more of such conditions change, the operating environment of the vehicle may change.

[0022] In some cases, the operating environment may be associated with a particular geographical location and / or constrained by some geographical limitation. For example, a first operating environment may be associated with a first route between two locations, and a second, different operating environment may be associated with a second route between the same two locations. Thus, when traveling on the first route, the vehicle may operate according to the first operating environment, and when traveling on the second route, the vehicle may operate according to the second operating environment. As another example, a first operating environment may be associated with a first portion of a route between a pick-up or loading location and a drop-off or unloading location, and a second operating environment may be associated with a second portion of the route between the pick-up or loading location and the drop-off or unloading location. As another example, the operating environment may be limited to a geographically defined area, such as the boundaries of an airport, a university, or a private residential community. Other examples are similarly possible.

[0023] This disclosure relates to a system and method for adjusting the field of view volume of one or more sensors of a vehicle based on the vehicle's operating environment (for example, based on the vehicle system detecting an operating environment or detecting a change from one operating environment to another). This may occur in real time or near real time when the vehicle system detects an operating environment or detects a change in the operating environment.

[0024] As described herein, the act of adjusting the field of view volume of a particular sensor can be performed in a variety of ways. For example, the act of adjusting the field of view volume of a particular sensor may be performed in response to the detection of a change in the vehicle's operating environment from a first operating environment to a second operating environment, and thus the vehicle system may include adjusting the sensor's field of view volume from a first field of view volume corresponding to the first operating environment to a second adjusted field of view volume corresponding to the second operating environment. As another example, the vehicle system may decide to actively switch itself from an operating mode associated with one operating environment to an operating mode associated with another operating environment, and the switch may involve, or cause, a switch from a state in which each of one or more vehicle sensors operates according to its respective field of view volume to a state in which each operates with a different respective field of view volume. As yet another example, the act of adjusting the field of view volume of a particular sensor may include the vehicle system initializing the field of view volume of that sensor, for example, by selecting a predetermined field of view volume based on the detected operating environment of the vehicle, or by determining the field of view volume using other techniques. In some cases, there may be a finite / predetermined number of operating environments to which a vehicle can be designed to operate, or to which a vehicle can be otherwise associated, and such operating environments, when detected, may be used to determine the field of view volume of one or more of the vehicle's sensors.

[0025] According to this disclosure, sensors may be configured to operate according to a field of view volume less than or equal to the maximum field of view volume. For example, a LiDAR sensor may be configured to operate with a maximum field of view of 200 meters, a maximum azimuth angle of 210 degrees, and a maximum elevation angle of 20 degrees above the horizon and 75 degrees below the horizon, and the maximum distance from the LiDAR sensor at which the LiDAR sensor data is reliably treated as having detected an object is 200 meters within these azimuth and elevation angle ranges. In one example, a vehicle may be traveling along a road, operating with one of its LiDAR sensors using a maximum field of view, such as 200 meters. However, the vehicle may encounter (or is expected to encounter) dense fog, which may degrade the sensor capabilities of the LiDAR. Therefore, the vehicle system (e.g., one or more processors, computing devices, etc.) may determine that the vehicle is operating in (or will soon switch to operating in) foggy weather conditions and, accordingly, adjust the field of view volume of at least one LIDAR sensor (e.g., all LIDAR sensors) to a lower field of view range, such as 70 meters, within the same azimuth and elevation range, or within a different azimuth and elevation range. Thus, the vehicle then operates so that the maximum distance from a LIDAR sensor where the LIDAR sensor data is reliably treated as having detected an object is 70 meters. Once the fog clears or the vehicle leaves the foggy area, the vehicle system may readjust the field of view volume of the LIDAR sensors (e.g., increase it). Other examples are similarly possible.

[0026] In some embodiments, to determine which field of view volume is most appropriate for a particular sensor to fit the vehicle's current operating environment, the vehicle system may store a mapping in memory (e.g., in the form of a lookup table) between several different operating environments of the vehicle and corresponding field of view volumes (or specific volume parameters, e.g., ranges) such as one or more sensors or sensor types of the vehicle. These corresponding field of view volumes can be predetermined using various techniques (e.g., machine learning of sensor databases, physics-based computation, etc.). In addition, or alternatively, the field of view volume may be determined in real time in response to the determination of the vehicle's operating environment, and / or a given field of view volume may be compared with newly acquired sensor data to determine whether the given field of view volume still accurately represents the degree to which the sensor should be trusted for the determined operating environment.

[0027] In some embodiments, the field of view volume of at least one of the vehicle's sensors may be adjusted based on the vehicle's operational design area (ODD). The ODD is defined by or includes conditions under which a given vehicle or other automated driving system or its functions are specifically designed to operate. Such conditions include, but are not limited to, environmental, geographical, and temporal constraints, and / or the presence or absence of specific traffic or road features required. A vehicle may have multiple ODDs, each including at least one of geographical conditions, temporal conditions, traffic conditions, or road conditions, and each may be associated with a predetermined field of view volume of at least one of the vehicle's sensors. Thus, a vehicle system may associate the vehicle with a first ODD (e.g., clear weather conditions), but in response to detecting a change in the vehicle's operating environment, it may instead associate the vehicle with a second different ODD (e.g., foggy, rainy, or snowy weather conditions), and therefore the vehicle system may activate at least one of the vehicle's sensors using a predetermined field of view volume of the sensor associated with the second ODD.

[0028] This disclosure also provides systems and methods that help determine the operating environment of a vehicle, particularly a weather-based operating environment. In some examples, a vehicle system may receive weather data from one or more servers or other computing devices associated with one or more weather stations. The weather data may, among other possibilities, identify weather conditions associated with a particular area (e.g., present in a particular area), such as fog, rain, or snow. Based on this weather data, the vehicle system can determine the operating environment of the vehicle and, therefore, how to adjust the sensor field of view volume depending on the area to which the weather data is associated. In one example, a global or local computing system may collect the weather data and distribute the weather data to various vehicle systems, possibly along with other information such as recommended sensor adjustments for a particular area, in light of the weather conditions present in that area. Other examples are similarly possible.

[0029] The disclosed systems and methods, advantageously, enable vehicle systems to adjust received sensor data in real time to dynamically adapt to changing conditions during movement, and enable the vehicle to accurately and reliably detect objects in its environment continuously. Similarly, the disclosed systems and methods, advantageously, enable vehicle systems to prioritize sensor systems that are more likely to behave more reliably than other sensor systems, and / or sensor data that are more likely to be reliable than other sensor data, and thus enable vehicle systems to control the vehicle using the most reliable sensor data. For example, a vehicle system may prioritize power to a particular sensor system or sensor function over other systems, or enable more power to that system, or prioritize or enable computing resources for processing particular sensor data over other sensors.

[0030] II. Exemplary Systems and Devices Next, exemplary systems and devices will be described in more detail. In general, embodiments disclosed herein can be used in any system that includes one or more sensors for scanning the system environment. Exemplary embodiments described herein include vehicles employing sensors such as LiDAR, radar, sonar, and cameras. However, examples of systems may also be implemented as robotic devices, industrial systems (e.g., assembly lines), or other devices, or in the form of other devices, among other possibilities.

[0031] The term “vehicle” is used herein to broadly encompass all moving objects, including, for example, aircraft, ships, spacecraft, automobiles, trucks, vans, semi-trailer trucks, motorcycles, golf carts, off-road vehicles, warehouse transport vehicles, tractors, or agricultural vehicles, as well as rail-riding transport vehicles such as roller coasters, trolleys, trams, or railway vehicles. Some vehicles may operate in a fully autonomous mode in which human interaction is not used for operation, a semi-autonomous mode in which human interaction is not used for operation at all, or a human-operated mode in which a human operates the vehicle and sensors may assist the human.

[0032] In exemplary embodiments, an example of a vehicle system may include one or more processors, one or more forms of memory, one or more input devices / interfaces, one or more output devices / interfaces, and machine-readable instructions that, when executed by one or more processors, cause the system to perform the various functions, tasks, capabilities, etc. described above. Exemplary systems within the scope of this disclosure are described in more detail below.

[0033] Figure 1 is a functional block diagram showing a vehicle 100 according to an exemplary embodiment. The vehicle 100 may be configured to operate in a fully or partially autonomous mode and may therefore be referred to as an “autonomous vehicle.” The vehicle may also be operated by a human but may be configured to provide information to the human through the vehicle’s sensing systems. For example, a computing system 111 may control the vehicle 100 during autonomous mode via control commands to the vehicle’s control system 106. The computer system 111 may receive information from one or more sensor systems 104 and automatically make one or more control processes based on the received information (e.g., setting the orientation to avoid detected obstacles).

[0034] The autonomous vehicle 100 may be fully autonomous or partially autonomous. In a partially autonomous vehicle, some functions may be optionally, temporarily or permanently, controlled manually (e.g., by a driver). Furthermore, a partially autonomous vehicle may be configured to switch between a fully manual operating mode and a partially autonomous and / or fully autonomous operating mode.

[0035] Vehicle 100 includes a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computing system 111, and a user interface 112. Vehicle 100 may include more or fewer subsystems, each subsystem of which may optionally include multiple components. Furthermore, each subsystem and component of Vehicle 100 may be interconnected and / or communicate with one another. Thus, one or more functions of Vehicle 100 described herein may optionally be divided among additional functional or physical components, or combined into fewer functional or physical components. In some further examples, additional functional and / or physical components may be added to the example illustrated by Figure 1.

[0036] The propulsion system 102 may include components that can operate to provide power motion to the vehicle 100. In some embodiments, the propulsion system 102 includes an engine / motor 118, an energy source 120, a transmission 122, and wheels / tires 124. The engine / motor 118 converts the energy source 120 into mechanical energy. In some embodiments, the propulsion system 102 may optionally include either or both an engine and / or a motor. For example, a gas-electric hybrid vehicle may include both a gasoline / diesel engine and an electric motor.

[0037] The energy source 120 represents an energy source such as electrical energy and / or chemical energy, which may, in whole or in part, power the engine / motor 118. That is, the engine / motor 118 may be configured to convert the energy source 120 into mechanical energy to operate the transmission. In some embodiments, the energy source 120 may include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, capacitors, flywheels, regenerative braking systems, and / or other power sources. The energy source 120 may also supply energy to other systems of the vehicle 100.

[0038] The transmission 122 includes suitable gears and / or mechanical elements for transmitting mechanical power from the engine / motor 118 to the wheels / tires 124. In some embodiments, the transmission 122 includes a gearbox, clutch, differential, drive shaft, and / or axle, etc.

[0039] The wheels / tires 124 are positioned to stably support the vehicle 100 while providing frictional traction to the surface on which the vehicle 100 travels, such as a road. Therefore, the wheels / tires 124 are configured and positioned according to the nature of the vehicle 100. For example, the wheels / tires may be positioned as a unicycle, bicycle, motorcycle, tricycle, or four-wheeled configuration for a car / truck. Other wheel / tire configurations are also possible, such as those including six or more wheels. Any combination of wheels / tires 124 on the vehicle 100 may be operable to rotate differentially with respect to other wheels / tires 124. The wheels / tires 124 may optionally include at least one wheel securely mounted to the transmission 122 and at least one tire coupled to the rim of the corresponding wheel that contacts the running surface. The wheels / tires 124 may include any combination of metal and rubber, and / or other materials or combinations of materials.

[0040] The sensor system 104 generally includes one or more sensors configured to detect information about the surrounding environment of the vehicle 100. For example, the sensor system 104 may include a global positioning system (GPS) 126, an inertial measurement unit (IMU) 128, a radar unit 130, a rangefinder / LIDAR unit 132, a camera 134, and / or a microphone 136. The sensor system 104 may also include sensors configured to monitor the internal systems of the vehicle 100 (e.g., an O2 monitor, a fuel gauge, engine oil temperature, wheel speed sensors, etc.). One or more sensors included in the sensor system 104 may be configured to operate separately and / or collectively to correct the position and / or orientation of one or more sensors.

[0041] GPS126 is a sensor configured to estimate the geographical location of vehicle 100. For this purpose, GPS126 may include a transceiver that can operate to provide information about the position of vehicle 100 relative to the Earth.

[0042] The IMU128 may include any combination of sensors (e.g., accelerometers and gyroscopes) configured to sense changes in the position and orientation of the vehicle 100 based on inertial acceleration.

[0043] The radar unit 130 may represent a system that utilizes radio signals to sense objects in the local environment of the vehicle 100. In some embodiments, in addition to sensing objects, the radar unit 130 and / or the computer system 111 may be further configured to sense the velocity and / or orientation of objects. The radar unit 130 may include any antennas, waveguide networks, communication chips, and / or other components that facilitate radar operation.

[0044] Similarly, the laser rangefinder or LiDAR unit 132 may be any sensor configured to sense objects in the environment in which the vehicle 100 is located using a laser. The laser rangefinder / LIDAR unit 132 may include, among other system components, one or more laser sources, a laser scanner, and one or more detectors. The laser rangefinder / LIDAR unit 132 may be configured to operate in coherent or incoherent detection mode (for example, using heterodyne detection).

[0045] The camera 134 may include one or more devices configured to capture multiple images of the surrounding environment of the vehicle 100. The camera 134 may be a still camera or a video camera. In some embodiments, the camera 134 may be mechanically movable, such as by rotating and / or tilting the platform on which the camera is mounted. Thus, the control process of the vehicle 100 can be implemented to control the movement of the camera 134.

[0046] The sensor system 104 may also include a microphone 136. The microphone 136 may be configured to capture sound from the surrounding environment of the vehicle 100. In some cases, multiple microphones may be arranged as a microphone array, or possibly as multiple microphone arrays.

[0047] The control system 106 is configured to control the operation of the vehicle 100 and its components to regulate acceleration. To bring about acceleration, the control system 106 includes a steering unit 138, a throttle 140, a brake unit 142, a sensor fusion algorithm 144, a computer vision system 146, a navigation / routing system 148, and / or an obstacle avoidance system 150, among others.

[0048] The steering unit 138 is operable to adjust the direction of travel of the vehicle 100. For example, the steering unit may adjust the axis (or more axes) of one or more wheels / tires 124 to effectively change the direction of the vehicle. The throttle 140 is configured to control, for example, the operating speed of the engine / motor 118, and then adjust the forward acceleration of the vehicle 100 via the transmission 122 and the wheels / tires 124. The brake unit 142 decelerates the vehicle 100. The brake unit 142 may decelerate the wheels / tires 124 using friction. In some embodiments, the brake unit 142 inductively decelerates the wheels / tires 124 by a regenerative braking process, converting the kinetic energy of the wheels / tires 124 into an electric current.

[0049] The sensor fusion algorithm 144 is an algorithm (or a computer program product that stores the algorithm) configured to accept data from the sensor system 104 as input. The data may include, for example, data representing information sensed by the sensors of the sensor system 104. The sensor fusion algorithm 144 may include, for example, a Kalman filter, a Bayesian network, etc. Based on the data from the sensor system 104, the sensor fusion algorithm 144 provides an assessment of the vehicle's surrounding environment. In some embodiments, the assessment may include an assessment of individual objects and / or features in the environment surrounding the vehicle 100, an assessment of a particular situation, and / or an assessment of potential interference between the vehicle 100 and features in the environment based on a particular situation (e.g., prediction of collision and / or impact).

[0050] The computer vision system 146 can process and analyze images captured by the camera 134 to identify objects and / or features in the environment surrounding the vehicle 100. Detected features / objects may include traffic signals, road boundaries, other vehicles, pedestrians, and / or obstacles. The computer vision system 146 may optionally employ object recognition algorithms, multi-view three-dimensional reconstruction (SFM: Structure From Motion) algorithms, video tracking, and / or available computer vision technologies to classify and / or identify the detected features / objects. In some embodiments, the computer vision system 146 may be additionally configured to map the environment, track perceived objects, estimate the velocity of objects, and so on.

[0051] The navigation and routing system 148 is configured to determine the route of the vehicle 100. For example, the navigation and routing system 148 can be configured, for example, according to user input via the user interface 112, and can determine a set of speeds and indicated directions of travel in order to move the vehicle along a route that substantially avoids perceived obstacles, while generally moving the vehicle along a road-based route to the final destination. The navigation and routing system 148 may also be configured to dynamically update the route while the vehicle 100 is in motion, based on perceived obstacles, traffic patterns, weather / road conditions, etc. In some embodiments, the navigation and routing system 148 may be configured to incorporate a sensor fusion algorithm 144, GPS 126, and data from one or more predetermined maps in order to determine the route of the vehicle 100.

[0052] The obstacle avoidance system 150 may represent a control system configured to identify, evaluate, and avoid or otherwise pass through potential obstacles in the environment surrounding the vehicle 100. For example, the obstacle avoidance system 150 may alter the vehicle's navigation by operating one or more subsystems within the control system 106 to perform lane change maneuvers, direction changes, braking maneuvers, etc. In some embodiments, the obstacle avoidance system 150 is configured to automatically determine feasible ("available") obstacle avoidance maneuvers based on surrounding traffic patterns, road conditions, etc. For example, the obstacle avoidance system 150 may be configured not to perform a lane change maneuver if other sensor systems detect a vehicle, construction barricade, other obstacle, etc., in the area adjacent to the vehicle to which it is to change lanes. In some embodiments, the obstacle avoidance system 150 may automatically select an available maneuver that takes maximum consideration of the safety of the occupants in the vehicle. For example, the obstacle avoidance system 150 may select an avoidance maneuver that is expected to minimize the acceleration inside the vehicle 100.

[0053] The vehicle 100 also includes peripheral devices 108 configured to enable interaction between the vehicle 100 and users such as external sensors, other vehicles, other computer systems, and / or the occupants of the vehicle 100. For example, peripheral devices 108 for receiving information from occupants, external systems, etc. may include a wireless communication system 152, a touchscreen 154, a microphone 156, and / or a speaker 158.

[0054] In some embodiments, peripheral device 108 functions to receive input for the user of vehicle 100 to interact with user interface 112. For this purpose, touchscreen 154 is capable of both providing information to the user of vehicle 100 and transmitting information from the user indicated via touchscreen 154 to user interface 112. Touchscreen 154 may be configured to sense both touch position and touch gesture from the user's finger (or stylus, etc.) via capacitive sensing, resistive sensing, optical sensing, surface acoustic wave processes, etc. Touchscreen 154 can sense finger movement in a direction parallel or planar to the touchscreen surface, in a direction normal to the touchscreen surface, or both, and can also sense the level of pressure applied to the touchscreen surface. Occupants of vehicle 100 may also utilize a voice command interface. For example, microphone 156 may be configured to receive voice (e.g., voice commands or other voice input) from the user of vehicle 100. Similarly, speaker 158 may be configured to output voice to the user of vehicle 100.

[0055] In some embodiments, peripheral devices 108 function to enable communication between the vehicle 100 and external systems such as devices, sensors, and other vehicles in the vehicle's surrounding environment, and / or external systems such as controllers and servers located physically away from the vehicle that provide useful information about the vehicle's surroundings, such as traffic information and weather information. For example, the wireless communication system 152 can wirelessly communicate with one or more devices directly or via a communication network. The wireless communication system 152 may optionally use 3G cellular communication such as CDMA, EVDO, GSM / GPRS, or 4G cellular communication such as WiMAX or LTE. In addition or alternatively, the wireless communication system 152 may communicate with a wireless local area network (WLAN) using, for example, WiFi. In some embodiments, the wireless communication system 152 may communicate directly with devices using, for example, infrared link, Bluetooth, and / or ZigBee. The wireless communication system 152 may include one or more dedicated narrow-area communication (DSRC) devices that can include public and / or private data communication between vehicles and / or roadside gas stations. Other wireless protocols for transmitting and receiving information embedded in signals, such as various vehicle communication systems, may also be employed by the wireless communication system 152 within the context of this disclosure.

[0056] As described above, the power supply 110 can supply power to components of the vehicle 100, such as electronic devices included in peripheral equipment 108, a computing system 111, a sensor system 104, and so on. The power supply 110 may include, for example, a lithium-ion battery or a lead-acid battery that is rechargeable to store electrical energy and discharge it to various powered components. In some embodiments, one or more banks of batteries may be configured to provide power. In some embodiments, the power supply 110 and the energy source 120 may be implemented together, as in some all-electric vehicles.

[0057] Many or all of the functions of the vehicle 100 may be controlled via a computer system 111 that receives inputs from sensor systems 104, peripheral devices 108, etc., and communicates appropriate control signals to propulsion systems 102, control systems 106, peripheral devices, etc., thereby bringing about automatic operation of the vehicle 100 based on its surroundings. The computing system 111 includes at least one processor 113 (which may include at least one microprocessor) that executes instructions 115 stored in a non-temporary computer-readable medium such as data storage 114. The computing system 111 may also represent multiple computing devices that function to control individual components or subsystems of the vehicle 100 in a distributed manner.

[0058] In some embodiments, the data storage 114 includes instructions 115 (e.g., program logic) that can be executed by the processor 113 to perform various functions of the vehicle 100, including those described above in relation to Figure 1. The data storage 114 may also include additional instructions, including instructions for transmitting data to, receiving data to, interacting with, and / or controlling one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0059] In addition to the instruction 115, the data storage 114 may store data such as road maps and route information as map data 116, among other information. Such information may be used by the vehicle 100 and the computer system 111 during the operation of the vehicle 100 in autonomous mode, semi-autonomous mode, and / or manual mode, for example, to select available roads to the final destination or to interpret information from the sensor system 104.

[0060] The vehicle 100 and its associated computer system 111 provide information to and / or receive input from users of the vehicle 100, such as occupants inside the vehicle 100. The user interface 112 may, accordingly, include one or more input / output devices within a set of peripherals 108, such as a wireless communication system 152, a touchscreen 154, a microphone 156, and / or a speaker 158, to enable communication between the computer system 111 and the occupants of the vehicle.

[0061] The computing system 111 controls the operation of the vehicle 100 based on inputs received from various subsystems indicating vehicle and / or environmental conditions (e.g., the propulsion system 102, the sensor system 104, and / or the control system 106), as well as inputs received from the user interface 112 indicating user preferences. For example, the computing system 111 may use inputs from the control system 106 to control the steering unit 138 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 150. The computing system 111 may be configured to control many aspects of the vehicle 100 and its subsystems. However, generally, provisions are made for manually overriding automated, controller-driven operation, such as in emergencies or in response to overrides initiated by the user.

[0062] The components of the vehicle 100 described herein may be configured to operate in conjunction with other components within or outside their respective systems. For example, the camera 134 may capture multiple images representing information about the environment of the vehicle 100 while operating in autonomous mode. The environment may include other vehicles, traffic lights, road signs, road markings, pedestrians, etc. The computer vision system 146 may classify and / or recognize various aspects of the environment in cooperation with the sensor fusion algorithm 144, the computer system 111, etc., based on object recognition models pre-stored in the data storage 114 and / or by other techniques.

[0063] Vehicle 100 is described as having various components, such as a wireless communication system 152, a computing system 111, data storage 114, and a user interface 112, integrated into the vehicle 100, as illustrated in Figure 1. However, one or more of these components can be optionally mounted or associated separately from the vehicle 100. For example, data storage 114 may exist partially or completely separately from the vehicle 100, such as in a cloud-based server. Thus, one or more of the functional elements of vehicle 100 may be implemented in the form of device elements that are arranged separately or together. The functional device elements constituting vehicle 100 can generally be coupled together to communicate with each other via wired and / or wireless means.

[0064] Figure 2 shows an exemplary vehicle 200 which may include some or all of the features described in relation to vehicle 100 with reference to Figure 1. Although vehicle 200 is shown in Figure 2 as a four-wheeled vehicle for illustrative purposes, the disclosure is not limited in this way. For example, vehicle 200 may represent a truck, van, semi-trailer truck, motorcycle, golf cart, off-road vehicle, or agricultural vehicle, etc.

[0065] Vehicle example 200 includes a sensor unit 202, a first LiDAR unit 204, a second LiDAR unit 206, a first radar unit 208, a second radar unit 210, a first LiDAR / radar unit 212, a second LiDAR / radar unit 214, and two additional locations 216, 218 on the vehicle 200 where radar units, LiDAR units, laser rangefinder units, and / or one or more sensors of other types may be located. Each of the first LiDAR / radar unit 212 and the second LiDAR / radar unit 214 may take the form of a LiDAR unit, a radar unit, or both.

[0066] Furthermore, the exemplary vehicle 200 may include any of the components described in relation to vehicle 100 in Figure 1. The first and second radar units 208, 210 and / or the first and second LIDAR units 204, 206 may be similar to the radar unit 130 and / or laser rangefinder / LIDAR unit 132 of vehicle 100, and may be similar to the radar unit 130 and / or laser rangefinder / LIDAR unit 132 of vehicle 100, and may be similar to the radar unit 130 and / or laser rangefinder / LIDAR unit 132 of vehicle 100, and may be similar to the first and second LIDAR / radar units 212 and / or laser rangefinder / LIDAR unit 214, and may be similar to the radar unit 130 and / or laser rangefinder / LIDAR unit 132 of vehicle 10.

[0067] In some examples, a LIDAR unit may be one of two different types of LIDAR units. A first type of LIDAR unit may be a rotating LIDAR capable of continuously scanning the entire field of view of the LIDAR unit. A second type of LIDAR unit may be a rotatable LIDAR capable of steering to scan a specific area of ​​the field of view of the LIDAR unit. A first type of LIDAR unit may have a smaller range than a second type of LIDAR unit. A second type of LIDAR unit may have a narrower field of view in operation compared to a first type of LIDAR unit. In some examples, one or more of the designated LIDAR units of a vehicle 200 may include one or both types of LIDAR units. For example, a LIDAR unit 204 mounted on top of a vehicle may include both types of LIDAR units. In one example, a second type of LIDAR unit may have a field of view in operation with a width of 8 degrees in the horizontal plane and a width of 15 degrees in the vertical plane.

[0068] The sensor unit 202 is mounted on top of the vehicle 200 and includes one or more sensors configured to detect information about the surrounding environment of the vehicle 200 and output an indicator of that information. For example, the sensor unit 202 may include any combination of cameras, radar, LiDAR, range finders, and acoustic sensors. The sensor unit 202 may include one or more movable mounts that can be operated to adjust the orientation of one or more sensors within the sensor unit 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to obtain information from each direction around the vehicle 200. In another embodiment, the movable mount of the sensor unit 202 may be movable in a scanning manner within a specific range of angles and / or orientations. The sensor unit 202 may be mounted on the roof of the vehicle, although other mounting locations are also conceivable. In addition, the sensors of the sensor unit 202 may be distributed in various locations and do not need to be juxtaposed in a single location. Some conceivable sensor types and mounting locations include two additional locations 216, 218. Furthermore, each sensor in the sensor unit 202 may be configured to move or scan in conjunction with other sensors in the sensor unit 202, or independently of other sensors.

[0069] In one configuration example, one or more radar scanners (e.g., first and second radar units 208, 210) may be located near the rear of the vehicle 200 to actively scan the area behind the vehicle 200 for the presence of radar-reflecting objects. Similarly, a first LIDAR / radar unit 212 and a second LIDAR / radar unit 214 may be mounted near the front of the vehicle to actively scan the area in front of the vehicle. The radar scanners may be positioned in a location suitable for illuminating an area including the forward path of the vehicle 200 without being obstructed by other features of the vehicle 200. For example, the radar scanners may be positioned to be embedded in and / or mounted in or near the front bumper, front headlights, cowl, and / or hood. Furthermore, one or more additional radar scanning devices may be positioned to actively scan the sides and / or rear of the vehicle 200 to check for the presence or absence of radar-reflective objects, for example, by including such devices in the rear bumper, side panels, rocker panels, and / or chassis.

[0070] In practice, each radar unit may be able to scan across a 90-degree beamwidth. If the radar units are positioned at the corners of the vehicle, as shown by radar units 208, 210, 212, and 214, each radar unit may be able to scan a 90-degree field of view in the horizontal plane, providing the vehicle with a full 360-degree radar field of view around the vehicle. Furthermore, the vehicle may also include two side-facing radar units. Side-facing radar units may be able to provide additional radar imaging when other radar units are obstructed, such as when making a right turn without an arrow signal (i.e., a right turn when there is another vehicle in the lane to the left of the turning vehicle).

[0071] Although not shown in Figure 2, the vehicle 200 may include a wireless communication system. The wireless communication system may include a radio transmitter and a radio receiver that can be configured to communicate with devices outside or inside the vehicle 200. Specifically, the wireless communication system may include, for example, a transceiver configured to communicate with other vehicles and / or computing devices in a vehicle communication system or road station. Examples of such vehicle communication systems include dedicated short-range communication (DSRC), radio frequency identification (RFID), and other communication standards proposed for intelligent transport systems.

[0072] The vehicle 200 may, in some cases, include a camera located inside the sensor unit 202. The camera may be a photosensitive device such as a still camera or a video camera, configured to capture multiple images of the environment of the vehicle 200. For this purpose, the camera may be configured to detect visible light, and in addition or alternatively, to detect light from other parts of the spectrum, such as infrared or ultraviolet light. In one particular example, the sensor unit 202 may include both an optical camera (i.e., a camera that captures human visible light) and an infrared camera. The infrared camera may be capable of capturing thermal images within the camera's field of view.

[0073] The camera may be a two-dimensional detector and may optionally have a sensitivity range in three-dimensional space. In some embodiments, the camera may include a range detector configured to produce a two-dimensional image showing the distance from the camera to several points in the environment. For this purpose, the camera may use one or more range detection techniques. For example, the camera may provide range information by using a structured light technique, in which the vehicle 200 illuminates objects in the environment with a predetermined light pattern, such as a grid or checkerboard pattern, and the camera is used to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, the vehicle 200 may determine the distance to a point on the object. The predetermined light pattern may consist of infrared light or radiation of other wavelengths suitable for such measurements. In some examples, the camera may be mounted inside the windshield of the vehicle 200. Specifically, the camera may be positioned to capture an image from a forward view relative to the orientation of the vehicle 200. Other mounting locations and field of view angles of the camera may also be used, and may be inside or outside the vehicle 200. Furthermore, the camera may have associated optical elements that are operable to provide an adjustable field of view. Additionally, the camera may be mounted on the vehicle 200 using a movable mount to change the camera's directional angle, such as via a pan / tilt mechanism.

[0074] Furthermore, the camera sensor may consist of a rolling shutter. A rolling shutter generally captures image data by iteratively sampling the light sensor. The data from the camera sensor can form an image, multiple images, or video. For example, in a conventional image sensor, a rolling shutter may iteratively sample one row of cells of the light sensor at a time. When sampling a camera sensor with a rolling shutter, fast-moving objects within the sensor's field of view may appear distorted. Such distortion is caused by iterative sampling. Because the lines of cells are sampled iteratively, the object being imaged moves slightly between each sampling. Thus, each line is sampled with a slight delay compared to the previous line. Due to the delay in sampling each line, objects with horizontal movement may have horizontal skew. For example, a vehicle moving across the sensor's field of view may have horizontal skew and vertical compression (or expansion), which distorts the vehicle. This skew can be problematic for processing based on the horizontal position of objects in the image. This system may help identify camera distortion that may be caused by a rolling shutter.

[0075] Figure 3 is a conceptual diagram of wireless communication between various computing systems related to an autonomous vehicle, based on an example implementation. In particular, wireless communication can occur between the remote computing system 302 and the vehicle 200 via the network 304. Wireless communication can also occur between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200. While the vehicle 200 is operating, it can send and receive data from both the server computing system 306 and the remote computing system 302 to support its operation. The vehicle 200 can communicate data related to its operation and data from its sensors to the server computing system 306 and the remote computing system 302. In addition, the vehicle 200 can receive operation commands and / or data related to objects sensed by the vehicle's sensors from the server computing system 306 and the remote computing system 302.

[0076] Vehicle 200 can correspond to various types of vehicles capable of transporting passengers or objects between locations, and can take any one or more forms of the vehicles considered above.

[0077] The remote computing system 302 may represent any type of device relating to remote assistance and operation technologies, including but not limited to those described herein. In examples, the remote computing system 302 may represent any type of device configured to (i) receive information relating to the vehicle 200, (ii) provide an interface from which a human operator or computer operator can then perceive the information and input a response relating to the information, and (iii) transmit the response to the vehicle 200 or to another device. The remote computing system 302 may take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some examples, the remote computing system 302 may include a number of computing devices operating together in a network configuration.

[0078] The remote computing system 302 may include one or more subsystems and components that are similar to or identical to those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some implementations, the remote computing system 302 may also include a user interface, including input / output devices such as a touchscreen and speakers. Other examples are similarly possible.

[0079] Network 304 represents the infrastructure that enables wireless communication between the remote computing system 302 and the vehicle 200. Network 304 also enables wireless communication between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0080] The location of the remote computing system 302 can vary within the scope of the example. For example, the remote computing system 302 may be located remotely from the vehicle 200, which has wireless communication via the network 304. In another example, the remote computing system 302 may correspond to a computing device within the vehicle 200, separate from the vehicle 200, but capable of interacting with the passengers or driver of the vehicle 200 by a human operator. In some examples, the remote computing system 302 may be a computing device with a touchscreen that can be operated by the passengers of the vehicle 200.

[0081] In some implementations, the operations described herein, performed by the remote computing system 302, may be performed, in addition or instead, by the vehicle 200 (i.e., by any system or subsystem of the vehicle 200). In other words, the vehicle 200 may be configured to provide a remote assistance mechanism that can be interacted with by the vehicle's driver or passengers.

[0082] The server computing system 306 may be configured to communicate wirelessly with (or, optionally, directly with) the remote computing system 302 and the vehicle 200 via the network 304. The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information about the vehicle 200 and its remote assistance. Thus, the server computing system 306 may be configured to perform any or any part of any operation described herein as being performed by the remote computing system 302 and / or the vehicle 200. The server computing system 306 may be available in some implementations of the wireless communication related to remote assistance, but not in other implementations.

[0083] The server computing system 306 may include one or more subsystems and components that are similar to or identical to those of the remote computing system 302 and / or vehicle 200, such as a processor configured to perform the various operations described herein, and a wireless communication interface for receiving and providing information to the remote computing system 302 and the vehicle 200.

[0084] The various systems described above can perform a variety of operations. These operations and their associated characteristics are described below.

[0085] In line with the above discussion, a computing system (e.g., a remote computing system 302, or possibly a server computing system 306, or a computing system local to the vehicle 200) could operate to use cameras to capture images of the autonomous vehicle's environment. Generally, at least one computing system could analyze the images and, if possible, control the autonomous vehicle.

[0086] In some implementations, to facilitate autonomous operation, a vehicle (e.g., vehicle 200) may receive data (also referred to herein as “environmental data”) representing objects in the environment in which the vehicle operates, in various ways. The vehicle’s sensor system may provide environmental data representing objects in the environment. For example, a vehicle may have a variety of sensors, including cameras, radar units, laser rangefinders, microphones, radio units, and other sensors. Each of these sensors may transmit environmental data regarding the information it receives to a processor within the vehicle.

[0087] In one example, a radar unit may be configured to transmit an electromagnetic signal that reflects off one or more objects near the vehicle. The radar unit can then capture the electromagnetic signal reflected by the objects. The captured reflected electromagnetic signal can enable the radar system (or processing system) to make various determinations about the object that reflected the electromagnetic signal. For example, the distance and position to various reflective objects can be determined. In some implementations, a vehicle may have two or more radars in different orientations. In practice, a vehicle may have six different radar units. In addition, each radar unit may be configured to steer its beam to one of four different sectors of that radar unit. In various examples, a radar unit may be able to scan its beam over a 90-degree range by scanning each of the four different sectors of the radar unit. The radar system may be configured to store the captured information in memory for subsequent processing by the vehicle's processing system. The information captured by the radar system may be environmental data.

[0088] In another example, a laser rangefinder (e.g., a LiDAR unit) may be configured to transmit an electromagnetic signal (e.g., light from a gas or diode laser, or other possible light source) that can be reflected by one or more objects near the vehicle. The laser rangefinder may be able to capture the reflected electromagnetic (e.g., laser) signal. The captured reflected electromagnetic signal may allow a ranging system (or processing system) to determine the range to various objects, for example, the object that reflected the electromagnetic signal back to the laser rangefinder. The ranging system may also be able to determine the velocity or speed of the object and store it as environmental data.

[0089] In some implementations, the processing system may be able to combine information from various sensors to further determine the vehicle's environment. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous vehicle. In other implementations, the processing system may use other combinations of sensor data to make environmental determinations.

[0090] While operating in autonomous mode, a vehicle can control its movements with little to no human input. For example, if a human operator enters an address into the vehicle, the vehicle may be able to drive to the designated destination without further human input (e.g., without the human needing to operate or touch the brake / accelerator pedals). Furthermore, while the vehicle is operating autonomously, sensor systems may receive environmental data. The vehicle's processing system may modify the vehicle's control based on the environmental data received from various sensors. In some examples, the vehicle may change its speed in response to environmental data from various sensors. The vehicle may change its speed to avoid obstacles, comply with traffic laws, etc. If the vehicle's processing system identifies an object near the vehicle, the vehicle may be able to change its speed or otherwise alter its movement.

[0091] If a vehicle detects an object but is not confident in its detection, the vehicle may request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object actually exists in the environment (e.g., whether there is actually a stop sign or not), (ii) verifying whether the vehicle's identification of the object is correct, (iii) correcting the identification if it was incorrect, and / or (iv) providing supplementary instructions to the autonomous vehicle (or correcting the current instructions).

[0092] Depending on the source of the environmental data, a vehicle may detect objects in the environment in various ways. In some implementations, the environmental data may come from a camera and be image or video data. The vehicle may analyze the captured image or video data to identify objects within the image or video data. In other implementations, the environmental data may come from a LIDAR unit. Methods and apparatus may be configured to monitor image and / or video data for the presence of objects in the environment. In other implementations, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in the environment based on radar, audio, or other data.

[0093] In some implementations, the techniques a vehicle uses to detect objects may be based on a set of known data. For example, data related to environmental objects may be stored in memory located within the vehicle. The vehicle can then determine an object by comparing the received data with the stored data. In other implementations, the vehicle may be configured to determine objects based on the context of the data. For example, street signs related to construction may generally be orange. Therefore, the vehicle may be configured to detect orange objects located near the side of the road as construction-related street signs. In addition, when the vehicle's processing system detects an object in the captured data, it may also calculate the confidence level of each object.

[0094] III. Examples of Field of View for Vehicle Detection Figure 4 shows an example of an autonomous vehicle 400 having various sensor fields of view. As previously mentioned with respect to Figure 2, the vehicle 400 may include multiple sensors. The positions of the various sensors may correspond to the sensor positions disclosed in Figure 2. However, in some cases, sensors may be in other positions. For the sake of simplicity in the drawing, the sensor positions have been omitted from Figure 4A. Figure 4A shows the respective fields of view of each sensor unit of the vehicle 400. The field of view of a sensor may include the angular range in which the sensor can detect an object, and the range corresponding to the maximum distance from the sensor at which the sensor can reliably detect an object.

[0095] As described above, the vehicle 400 may include six radar units. The first radar unit may be located on the left front of the vehicle and may have an angular field of view corresponding to the angular portion of the field of view 402A. The second radar unit may be located on the right front of the vehicle and may have an angular field of view corresponding to the angular portion of the field of view 402B. The third radar unit may be located on the left rear of the vehicle and may have an angular field of view corresponding to the angular portion of the field of view 402C. The fourth radar unit may be located on the right rear of the vehicle and may have an angular field of view corresponding to the angular portion of the field of view 402D. The fifth radar unit may be located on the left side of the vehicle and may have an angular field of view corresponding to the angular portion of the field of view 402E. The sixth radar unit may be located on the right side of the vehicle and may have an angular field of view corresponding to the angular portion of the field of view 402F. Each of the six radar units may be configured to have a scannable beam width of 90 degrees. The radar beam width may be less than 90 degrees, but each radar unit may be able to steer the radar beam over a 90-degree field of view.

[0096] A first LiDAR unit of the vehicle 400 may be configured to scan a full 360-degree area around the vehicle, as indicated by an angular field of view corresponding to an angular portion of the field of view 404. A second LiDAR unit of the vehicle 400 may be configured to scan an area smaller than a 360-degree area around the vehicle. In one example, the second LiDAR unit may have an 8-degree field of view in the horizontal plane, as indicated by an angular field of view corresponding to an angular portion of the field of view 404.

[0097] In addition, the vehicle may also include at least one camera. The camera may be an optical camera and / or an infrared camera. The camera may have an angular field of view corresponding to the angular portion of the field of view 408.

[0098] In addition to the respective fields of view of each of the various sensors on vehicle 400, each sensor may also have a corresponding range. For example, the range of a radar unit may be greater than the range of any LIDAR unit, as shown by the fields of view of radar units 402A-402E, which extend further than the fields of view of LIDAR units 404 and 406. In addition, the first LIDAR unit may have a larger range than the second LIDAR unit, as shown by the field of view 404, which extends further than field of view 406. The camera may have a range indicated by the width of its field of view 408. In various examples, the camera's range may be larger or smaller than the range of the other sensors. Please understand that the sensor fields of view, radar units, etc. in Figure 4 are shown as illustrative examples and are not to scale.

[0099] IV. Examples of Systems and Methods Examples of the systems and methods described herein are described in more detail below.

[0100] Figure 5 is a flowchart of Method 500 according to an exemplary embodiment. Method 500 may include one or more operations, functions, or actions, as illustrated by one or more of blocks 502-506. Although the blocks of each method are shown in a sequential order, these blocks may, in some cases, be executed in parallel and / or in an order different from that described herein. Also, the various blocks may be combined into fewer blocks, divided into further blocks, and / or removed, based on the desired implementation.

[0101] In addition, with respect to Method 500 and other processes and methods disclosed herein, flowcharts illustrate the function and operation of one possible implementation of these embodiments. In this regard, each block may represent a module, segment, part of a manufacturing or operation process, or part of program code, which includes one or more instructions executable by a processor to implement a particular logical function or step in the process. The program code may be stored on any type of computer-readable medium, such as a storage device including a disk or hard drive. The computer-readable medium may include non-temporary computer-readable mediums, such as computer-readable mediums that store data for short periods, such as register memory, processor cache, and random access memory (RAM). The computer-readable medium may also include non-temporary computer-readable mediums, such as auxiliary storage or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, or compact disk read-only memory (CD-ROM). The computer-readable medium may also be any other volatile or non-volatile storage system. The computer-readable medium may be thought of as, for example, a computer-readable storage medium or a tangible storage device.

[0102] In addition or alternatively, with respect to Method 500 and other processes and methods disclosed herein, one or more blocks in the flowchart may represent circuits wired to perform a particular logical function within the process.

[0103] In some examples, with respect to Method 500 and other processes and methods disclosed herein, the functions represented in the flowchart may, among other possibilities, be performed by a single vehicle (e.g., vehicles 100, 200, etc.), distributed among multiple vehicles, by a remote server / external computing system (e.g., systems 302 and 306), and / or by a combination of one or more external computing systems and one or more vehicles.

[0104] In block 502, method 500 includes determining the operating environment of an autonomous vehicle. The autonomous vehicle is equipped with one or more sensors, each of which is configured to operate according to the field of view volume.

[0105] In block 504, this method includes adjusting the field of view volume of at least one of one or more sensors from a first field of view volume to a modified field of view volume different from the first field of view volume, based on a determined operating environment of the autonomous vehicle.

[0106] In block 506, this method includes controlling the autonomous vehicle to operate using at least one sensor having a regulated field of view volume.

[0107] As described above, one or more sensors in an autonomous vehicle may include, among other possible sensor types, a set of one or more LiDAR sensors, a set of one or more radar sensors, and / or a set of one or more cameras (operating in various wavelength bands, including visible and infrared). In practice, all sensors of a particular type may be configured to have the same maximum field of view volume, and vehicle software that receives and processes sensor data may be configured to treat all sensors of that particular type as having that maximum field of view volume. For example, the maximum field of view of all radar sensors in a vehicle may be 220 meters, and the maximum field of view of all LiDAR sensors in a vehicle may be 200 meters. In accordance with these, the act of adjusting the field of view volume of at least one of the vehicle's sensors may include making the same field of view adjustment for each sensor of a particular sensor type. For example, if a vehicle system decides to make adjustments to LiDAR sensors based on the vehicle's operating environment, the vehicle system may make those adjustments to all LiDAR sensors in the vehicle. Other examples are similarly possible. Furthermore, in alternative embodiments, sensor field of view volumes may be configured individually, so that a sensor of a particular sensor type may have a different maximum field of view volume than another sensor of the same sensor type.

[0108] At a given point in time, a vehicle may be configured to operate in one of several different operating environments, each of which may include one or more conditions inside and / or outside the vehicle. (Hereinafter, the terms “state” or “operating state” refer to an operating environment defined by at least one condition and may be used synonymously with the term “operating environment.”) For example, the operating environment of a vehicle may include: (i) default state (e.g., a state defined to include an operating environment different from other environments listed herein, in which the vehicle operates according to predetermined default parameters such as speed, steering, navigation, and / or sensor field of view); (ii) clear weather conditions (e.g., sunny, not cloudy, and without rain, snow, or fog); (iii) daytime operating conditions (e.g., the period from sunrise to sunset); (iv) nighttime operating conditions (e.g., the period from sunset to sunrise); (v) rainy weather conditions; (vi) snowy weather conditions; (v) foggy weather conditions; (viii) conditions for a particular type of road on which the autonomous vehicle is traveling (e.g., street, suburban road, country road, local road, highway, gravel, brick, asphalt, and / or (x)

[0109] As described above, the operating environment may be a combination of conditions or include a combination of conditions. For example, the operating environment may be sunny, clear, and daytime (e.g., 11:00 AM, sunny). Another example is that the operating environment may be sunny, daytime, uphill terrain, asphalt road, and heavy traffic. In some examples, the default state may consist of one or more other conditions, such as sunny weather and a vehicle driving on asphalt or concrete. Furthermore, in some examples, the default state may have the maximum field of view volume for each sensor type as the corresponding field of view volume. In addition, in some examples, there may be operating environments with varying degrees of weather conditions, such as severe blizzard and / or wind, heavy rain and / or wind, or fog density exceeding a predetermined threshold, among other possibilities.

[0110] Depending on the circumstances, operating environment conditions that affect the field of view volume of one type of sensor may affect, or may not affect, the field of view volume of another type of sensor. For example, a vehicle system may reduce the field of view of a LiDAR sensor in foggy nighttime conditions, and by using fog / nighttime operating environment information, it may reduce the field of view in which one or more of the vehicle's cameras can detect objects with a certain contrast (e.g., a gray car in the fog). Another example is when a vehicle system adjusts the field of view of a LiDAR sensor in a particular way due to a large reflection from a highway sign (retroreflector), but the vehicle system may not adjust the field of view in which the cameras can detect objects due to the large reflection. Other examples are similarly possible.

[0111] To facilitate the act of adjusting the sensor field of view volume of at least one of one or more sensors, the vehicle system may store in memory (e.g., data storage 114) a plurality of operating environments and a mapping between each of the plurality of operating environments and the corresponding adjusted field of view volume of at least one of the one or more sensors. In such embodiments, the act of adjusting the field of view volume based on the operating environment of the autonomous vehicle may include selecting an adjusted field of view volume corresponding to a determined operating environment of the autonomous vehicle. In some embodiments, an adjusted field of view volume corresponding to a particular operating environment may be part of a set of adjusted field of view volumes that correspond to that operating environment and include their respective adjusted field of view volumes for each of a plurality of sensor types (e.g., LIDAR, radar, and / or cameras). The stored operating environments and mappings may take various forms, such as tables.

[0112] Figure 6 shows an exemplary mapping of sensor field-of-view parameters (i.e., range in this example) to exemplary operating environments in the form of Table 600. As illustrated, for each of the eight representative examples of operating environments, Table 600 includes the respective field-of-view ranges corresponding to each of the three sensor types (LIDAR, radar, and camera). In some examples, Table 600 can be used for a vehicle system to detect foggy weather, determine that the vehicle is operating in a foggy weather operating environment, and accordingly select a field-of-view range value mapped to the foggy weather operating environment for use. Specifically, a vehicle system having a set of LIDAR sensors, a set of radar sensors, and a set of cameras may have all LIDAR sensors with a field-of-view range of 50 meters, all radar sensors with a field-of-view range of 70 meters, and all cameras with a field-of-view range of 120 meters. In foggy weather during daytime operation, cameras may be more reliable than LIDAR sensors, but at night, LIDAR sensors may be more reliable because headlights are rearscattered around the vehicle.

[0113] As another example, a vehicle system may detect an error in at least one camera of a set of cameras and, based on that detection, select one or more field-of-view volume parameters (e.g., range, azimuth, and / or elevation). For example, a vehicle system may detect a camera error based on the judgment that one or more images acquired by at least one camera fall below expected accuracy, roughness, and / or other measurements of the ability to discern details beyond a certain distance (e.g., it is unable to discern in the image a reference road sign that is expected to be visible with high contrast from a distance of more than 50 meters). Thus, the vehicle system may, accordingly, switch to a judgment that the operating environment of the vehicle system is an error state in which all readings from all cameras are negligible for the vehicle system. Alternatively, there may be variations in the operating environment in which the sensors are less reliable, and instead of ignoring all readings from such sensors, the vehicle system may use reduced field-of-view range, azimuth, and / or elevation for such sensors. For example, if the vehicle system estimates that the camera image is low contrast beyond approximately 10 meters and the LIDAR readings are abnormal beyond 50 meters, the vehicle system may reduce the field of view of all cameras to 70 meters and further reduce the field of view of all LIDAR sensors to 50 meters. To facilitate this example and other examples, there may be sensor error operating environments in addition to, or instead of, the sensor error operating environments shown in Figure 6. Other examples are similarly possible.

[0114] In alternative embodiments, the vehicle system may use only the field-of-view volume parameter values ​​from the mapping for a subset of specific types of sensors. For example, in foggy weather, the vehicle system may use a first field-of-view range of 50 meters for the vehicle's first LIDAR unit shown in Figure 4, but a default field-of-view range of 200 meters for the vehicle's second LIDAR unit. In other alternative embodiments, the vehicle system may use only the sensor field-of-view volume parameter values ​​for a subset of sensor types. For example, in foggy weather, the vehicle system may use reduced field-of-view ranges for LIDAR and camera sensors, but the radar field-of-view range may remain unchanged (e.g., the default 220 meters). Other examples are possible, including mappings for more or fewer sensor types.

[0115] In addition to the sensor type being mapped to a corresponding adjusted field of view volume parameter value, or alternatively, the above mapping or a separate mapping stored in memory may map the sensor type to other information related to the sensor field of view and affecting the sensor data that the sensor may acquire. For example, a stored mapping may map the sensor type to a power level that a particular sensor may use when acquiring sensor data.

[0116] As a more specific example, in the case of a LiDAR sensor, a stored mapping may map the LiDAR sensor type to the power level of the laser pulses transmitted by the LiDAR sensor when acquiring LiDAR data. In embodiments where the mapping is stored and / or in other embodiments where the stored mapping may not be used, the act of adjusting the field of view volume of at least one sensor from a first field of view volume to a modified field of view volume different from the first field of view volume may include adjusting the power level of the laser pulses transmitted by the LiDAR sensor when acquiring sensor data from a first power level to a modified power level associated with the modified field of view volume, different from the first power level. Furthermore, the act of controlling an autonomous vehicle to operate using at least one sensor having a modified field of view volume may include controlling the LiDAR sensor to acquire sensor data by transmitting one or more laser pulses having a modified power level associated with the modified field of view volume. For example, when illuminating relatively large retroreflector targets with a laser after they have been detected, the power level for transmitting the laser pulses may be reduced. As another example, in foggy conditions, for example, when a vehicle system is observing a narrow field of view, the power level may be increased. Other examples are similarly possible. A specific regulated power level can be associated with a regulated field of view volume in various ways. For example, a mapping stored by a vehicle system may map the corresponding regulated power level of each transmission that may occur to acquire (or acquire and ignore, discard, identify, etc.) sensor data corresponding to a specific regulated range, azimuth, or elevation of the regulated field of view volume. Other examples are similarly possible.

[0117] As another specific example, in the case of a radar sensor, the stored mapping may map the radar sensor type to specific radio wave characteristics (e.g., shape, amplitude, bandwidth, duration) of the radio waves transmitted by the radar sensor when acquiring radar data. For illustrative purposes, the transmit power of the transmitted radio waves and the transmit or receive beamforming performed by the vehicle's radar system can also be considered radio wave characteristics. In embodiments in which such stored mappings are stored, and / or in other embodiments in which stored mappings may not be used, the act of adjusting the field of view volume of at least one sensor from a first field of view volume to a modified field of view volume different from the first field of view volume may include adjusting the radio wave characteristics of the radio waves transmitted by the radar sensor when acquiring sensor data, for example, by adjusting a certain characteristic (e.g., transmit power) from a first value to a modified value different from the first value, or by adjusting the characteristic in another way. Furthermore, the act of controlling an autonomous vehicle to operate using at least one sensor having a modified field of view volume may include controlling the radar sensor to acquire sensor data by transmitting one or more radio waves having modified radio wave characteristics associated with the modified field of view volume. Other examples are similarly possible. Specific tuned radio wave characteristics can be associated with tuned field-of-view volumes in various ways. For example, a mapping stored by a vehicle system may map the corresponding tuned radio wave characteristics of transmissions that may occur to acquire (or acquire and ignore, discard, identify, etc.) sensor data corresponding to a specific tuned range, azimuth, or elevation angle of a tuned field-of-view volume. Other examples are similarly possible.

[0118] In more specific examples of how a vehicle system adjusts the field of view volume of a radar sensor, in a heavy rain operating environment, the vehicle system (or the sensor chip controlling the radar) may reduce the field of view volume by reducing the radar's azimuth scanning angle, and may also direct more antenna gain forward to improve penetration in the rain. Another example is in a densely populated urban environment, where the vehicle system (or the sensor chip controlling the radar) may cause the radar sensor to transmit less power at certain angles corresponding to objects such as large retroreflectors, thereby improving the radar sensor's ability to detect smaller objects near those large retroreflectors. Yet another example is when there are water droplets on the vehicle's radome; the vehicle system (or the sensor chip controlling the radar) may cause the radar sensor to transmit more power, thereby compensating for the water droplets and reaching a range that the radar sensor would reach if the radome were dry. Other examples are similarly possible.

[0119] In some embodiments, the range, azimuth, and / or elevation of a sensor's field of view volume may be adjusted to a value lower than the maximum field of view of that sensor and parameter. Nevertheless, the sensor may be configured to acquire sensor data corresponding to ranges, azimuth, and / or elevations exceeding the respective ranges, azimuth, and / or elevations associated with the adjusted field of view volume and transmit it to the vehicle system (e.g., a processor configured to process the sensor data). In such embodiments, for example, if the vehicle is controlled to operate using a sensor with an adjusted field of view volume, the vehicle system may ignore sensor data corresponding to ranges, azimuth, and / or elevations greater than the respective ranges, azimuth, and / or elevations associated with the adjusted field of view volume (e.g., discarding or storing it, but not making decisions about the vehicle environment, such as object detection, based on it). For example, if the range of a LiDAR sensor is reduced from 200 meters to 150 meters, the vehicle system may ignore sensor data corresponding to distances from the vehicle greater than 150 meters. Other examples are similarly possible. In addition or alternatively, the vehicle system may identify sensor data corresponding to parameter values ​​greater than the maximum parameter value of the adjusted field of view volume (e.g., by flagging it or otherwise storing in memory an indication that the data may be suspicious). In an alternative embodiment, such a sensor may be configured to set itself not to acquire sensor data corresponding to ranges, azimuths, and / or elevations exceeding the respective ranges, azimuths, and / or elevations associated with the adjusted field of view volume. In addition or alternatively, the sensor may be configured to acquire sensor data corresponding to ranges, azimuths, and / or elevations exceeding the respective ranges, azimuths, and / or elevations associated with the adjusted field of view volume, but may be further configured to discard such sensor data in order to reduce the amount of data transmitted from the sensor to other computing devices in the vehicle system.

[0120] The act of controlling a vehicle to operate using at least one sensor having a calibrated field of view volume may include controlling the vehicle to operate in autonomous mode using at least one sensor having a calibrated field of view volume, i.e., when operating in autonomous mode, controlling the vehicle to acquire sensor data using at least one sensor based on the calibrated field of view volume. To facilitate this in some embodiments, a local computing system on board the vehicle may configure itself to ignore sensor data readings acquired during the vehicle's operation that exceed the respective range, azimuth, and / or elevation angles associated with each calibrated field of view volume of at least one sensor. In addition or alternatively, a remote system may send a command to the vehicle's local computing system, upon reception by the local computing system, causing the local computing system to control the vehicle to operate in autonomous mode in which the local computing system ignores sensor data readings that exceed the respective range, azimuth, and / or elevation angles associated with each calibrated field of view volume of at least one sensor. Other examples are similarly possible.

[0121] In some embodiments, the vehicle's sensors and associated computing devices, such as a chip (e.g., a microchip) that controls the operation of one or more sensors, may perform actions before the sensors transmit acquired sensor data to an onboard computer or remote computer, which may affect how the onboard computer or remote computer controls the vehicle's operation. In particular, such a sensor chip may perform one or more actions of Method 500 (or Method 800, which will be described in more detail later herein). For example, the sensor chip may be configured to determine the vehicle's operating environment based on acquired sensor data (in the same or similar manner as described in more detail below) and adjust the field of view volume of one or more sensors accordingly. In this regard, the act of adjusting the field of view volume may include the sensor chip ignoring or flagging sensor data corresponding to ranges, azimuths, and / or elevations greater than the respective ranges, azimuths, and / or elevations associated with the adjusted field of view volume. In addition or alternatively, the act of adjusting the field of view volume may include the sensor chip (i) adjusting the power level of laser pulses transmitted by one or more LIDAR sensors at the time of acquiring sensor data from a first power level to an adjusted power level different from the first power level, and / or (ii) acquiring sensor data by transmitting one or more laser pulses at the adjusted power level associated with the adjusted field of view volume. In addition or alternatively, the act of adjusting the field of view volume may include the sensor chip (i) adjusting the characteristics of radio waves transmitted by one or more radar sensors at the time of acquiring sensor data (e.g., from a first value to an adjusted value different from the first value), and / or (ii) acquiring sensor data by transmitting one or more radio waves having adjusted radio wave characteristics associated with the adjusted field of view volume. Other examples are similarly possible.

[0122] Actions of a vehicle system that determine the operating environment of a vehicle can occur in various forms. Generally, these actions may include the vehicle system receiving information related to the vehicle's surrounding environment (e.g., detected road objects, weather data detected by various sensors), information related to the running operation of the vehicle and its components (e.g., sensor error codes), and / or information entered by the user (e.g., the vehicle's driver) via the user interface 112. For example, one or more of the vehicle's sensors may acquire sensor data, and the vehicle system may use this sensor data to determine the weather conditions at a particular location along the vehicle's travel path. The vehicle system can then determine the vehicle's operating environment based on the determined weather conditions. As a more specific example, one or more of the vehicle's sensors may acquire sensor data that the vehicle system is configured to interpret as indicating sunlight (and therefore daytime) and rainy weather, and in response to the acquisition and analysis of that sensor data, the vehicle system may determine that the vehicle is operating in rainy weather conditions. For example, a vehicle system may be configured to determine weather conditions by detecting backscattered light from a LiDAR laser pulse striking raindrops, snowflakes, or fog droplets. Similarly, a vehicle's radar sensor may acquire sensor data, from which the vehicle system can infer the type of rain conditions present in the environment (e.g., light, moderate, heavy) based on the amount / distribution / Doppler shift of backscattered radar energy from droplets in the air. Another example of determining the operating environment is a vehicle camera that may acquire one or more images, from which the vehicle system can infer the fog conditions (e.g., dense, fine) based on the amount of contrast reduction to known objects (e.g., prior information stored in the vehicle) within a certain range.

[0123] As another specific example, the driver, remote assistant, or passenger of a vehicle may know that a blizzard will soon begin (for example, based on a weather forecast) and provide input data indicating a command for the vehicle to begin operating in snowy weather conditions (for example, via a touchscreen GUI installed in the vehicle). Thus, in response to the receipt of the input data, the vehicle system may control the vehicle to begin operating in snowy weather conditions, which may take the form of the vehicle system determining that the vehicle is operating in snowy weather conditions. In a more specific variation of this example, the vehicle system may have access to a predetermined 3D map of the vehicle's surrounding environment, for example, a 3D map showing a stop sign that the vehicle is approaching. The vehicle system may be configured to compare the predetermined 3D map with sensor data acquired in real time by the vehicle's sensors and determine the vehicle's operating environment based on the comparison. For example, the 3D map may clearly show a stop sign, but the vehicle's camera or LIDAR sensor may acquire sensor data in which the stop sign is not as clearly visible. The vehicle system may be configured to interpret the nature of the difference determined in the comparison as a specific type of weather condition, such as snow. Other examples are similarly possible.

[0124] As yet another specific example, as described above, a vehicle system may receive sensor data acquired by one or more of the vehicle's sensors and, based on that sensor data, determine a sensor error in at least one of the vehicle's sensors (for example, an abnormal sensor reading outside a predetermined threshold sensor reading range). Using the sensor error, the vehicle system may determine that the vehicle is operating in a particular sensor error state. Other examples are similarly possible.

[0125] In some embodiments, a vehicle system may be configured to determine the operating environment of the vehicle based on weather conditions associated with specific locations along the vehicle's travel route. To facilitate this, the vehicle system may receive weather data indicating the weather conditions at specific locations along the vehicle's travel route, and based on the weather conditions indicated by the received weather data, the vehicle system may determine the operating environment of the vehicle. Weather conditions can take the form of any one or more weather conditions and / or other possible weather conditions described herein. Specific locations can be represented in the weather data in various forms. Generally, a location can be dynamic (e.g., the vehicle's current location along the travel route) or static (e.g., the vehicle's destination or a location on its way to a destination). Furthermore, a location can be a circular area with a specific radius centered on a specific landmark (e.g., a circular area with a radius of 8 kilometers centered on the city center of a certain city). Other boundaries of the area are also possible, for example, a city and its boundaries shown on a given map.

[0126] In some embodiments, the vehicle system may receive weather data from a weather station server or other types of servers. The weather station server may be a local weather station server for a specific location, i.e., a weather station server dedicated to a specific location, configured to acquire weather data corresponding to that location and transmit the weather data to one or more vehicle systems. In addition or alternatively, the weather station server may be a global weather station server configured to acquire weather data corresponding to multiple locations, such as an entire state, an entire county, or an entire country. The global weather station server may also operate as a server configured to collect weather data from multiple local weather station servers and transmit the collected weather data to one or more vehicle systems. In some embodiments, the weather station server may be configured to estimate weather conditions in various ways and to include various types of information in the weather data. For example, the weather station server may estimate weather conditions in the form of “donut” or other shaped representations of fog, mist, snow, and / or rain, in the form of the distribution, density, and diameter of water droplets in clouds, fog, and mist, and / or other forms. Such weather condition estimation activities may include a weather station server (or vehicle) monitoring and analyzing indicators of donut quality such as fog, haze, and rain. Other possible examples of functions of local or global weather station servers are also possible.

[0127] To facilitate the reception of weather data from weather station servers, a vehicle system can select a weather station server from several possible options before determining the vehicle's operating environment, and then send a query for weather data to the selected weather station server. In response to the query, the vehicle system can then receive weather data from the selected weather station. The vehicle system may be configured to select a weather station server based on various criteria. In some examples, the vehicle system may select a weather station server within a threshold distance (e.g., within 16 kilometers) of the vehicle's current location. In other examples, the vehicle system may select a weather station server or other weather station data transmitter within a threshold distance of the vehicle's estimated future location (e.g., if the vehicle is en route to a city, the vehicle system may select a weather station server within 8 kilometers of the city's boundary). Other examples are similarly possible.

[0128] In some embodiments, the weather station server may be configured to deliver up-to-date weather data for a specific location to a fleet of vehicles (e.g., multiple different vehicle systems associated with multiple different vehicles) or to multiple individual vehicles. Furthermore, the weather station server may be configured to transmit weather data to vehicle systems in response to receiving queries for weather data from vehicle systems and / or without being specifically requested by the vehicle systems (e.g., configured to deliver up-to-date weather data for a specific location every 30 minutes).

[0129] In any of the examples provided herein, weather data may be time-stamped so that a vehicle system can use the timestamp as a reference when determining the operating environment of the vehicle. For example, if a timestamp indicates that a vehicle encountered fog in a particular area 35 minutes ago, another vehicle approaching the same area may determine, based on the timestamp providing a time exceeding a predetermined threshold (e.g., 30 minutes), that fog conditions are no longer present, indicating a high threshold possibility. Thus, the other vehicle may not adjust its sensor field of view volume to account for such fog conditions, or it may adjust the accepted sensor field of view volume after one or more other signals supporting the possibility of fog conditions.

[0130] Figure 7 shows an exemplary communication system in which exemplary vehicles communicate with a local weather station server 700 and a global weather station server 702. In particular, Figure 7 shows a first vehicle 704 and a second vehicle 706 on a travel route 708 (e.g., a road). Furthermore, Figure 7 shows an exemplary region of interest 710 to which meteorological data may correspond. Although Figure 7 shows vehicles 704 and 706 communicating directly with servers 700 and 702, such communication may also be performed via one or more other computing devices, such as the remote computing system 302 in Figure 3, in addition to or instead.

[0131] In one example, since region 710 is along the travel path 708 of the first vehicle 704, the first vehicle 704 can query one or more of the servers 700, 702 for weather data corresponding to region 710. As described above, the first vehicle 704 can determine that the local weather station server 700 is within a threshold distance from the current location of the first vehicle 740 and / or within a threshold distance from region 710, and accordingly select the local weather station server 700 as the destination for querying weather data corresponding to region 710. In addition or alternatively, the second vehicle 706 may acquire sensor data indicating at least one weather condition present in region 710 and transmit the weather data indicating the weather condition directly to the first vehicle 704, or the first vehicle 704 may receive the weather data via another computing device, such as one or more of the servers 700, 702.

[0132] By configuring the vehicle system and weather station server as described above, the latest weather information can be efficiently provided to the vehicle, allowing the vehicle to quickly adjust the sensor field of view volume to adapt to changing weather conditions.

[0133] In addition to, or instead of, weather station servers configured to acquire, collect, manage, and transmit weather data, vehicle systems can receive weather data directly or indirectly from other vehicle systems. For example, one vehicle system may transmit weather data to another vehicle system. As another example, a global system (e.g., server computing system 306 or remote computing system 302) may be configured to receive weather data associated with various regions and transmit it to a fleet of vehicles. In this way, vehicles can usefully inform each other about weather conditions in areas where other vehicles are currently driving or plan to drive, and thus continue to inform each other in real time, and quickly adjust their sensor field of view volume accordingly.

[0134] Accordingly, in some embodiments, a vehicle may be configured to operate as a weather station, collecting weather data and transmitting it to other vehicles (e.g., other vehicles configured to operate in autonomous mode), a weather station server, and / or another backend server (e.g., server computing system 306 in Figure 3). In addition or alternatively, a vehicle may be configured to operate as a weather station server, facilitating and / or collecting weather data communication between vehicles, weather stations, and / or other servers. For brevity, a vehicle configured to operate as a weather station and / or weather station server is referred to as a weather station vehicle. In some examples, a fleet of weather station vehicles may share weather data among the vehicles. This can be helpful in determining the location of weather data, as the location of each other may be shared among the vehicles in a fleet and / or notified by a backend server. An example of the advantages of weather station vehicles is that they reduce the reliance of autonomous vehicles on other weather stations or other resources.

[0135] A vehicle configured to operate as a weather station server may consist of one or more of the sensors described herein, and may use such sensors to acquire weather data. In addition or alternatively, such sensors may be specially modified to allow them to acquire additional weather information, or weather information that is more detailed than usual, for example.

[0136] In addition, or instead, such a vehicle may include additional sensors, components, and computing devices that enable the vehicle to acquire and provide meteorological data of types that a weather station might provide, but which would not normally be used during the vehicle's standard operation (e.g., when the vehicle is not operating as a weather station server). For example, the vehicle may include sensors configured to determine the air quality of the air in the vehicle's environment, sensors configured to determine the humidity outside the vehicle, solar sensors (e.g., for determining the solar load on the vehicle's equipment and for calculating expected temperature changes caused by the sun that may affect the vehicle and its equipment), temperature sensors, and / or rain sensors. One or more of the sensors described herein (e.g., LIDAR, radar, camera, thermal, humidity, air quality, solar, rain) may be physically different sensors or may be integrated as a single sensor configured to be used for acquiring meteorological data to facilitate the vehicle's operation as a weather station server. Furthermore, such sensors may be mounted in various locations inside and outside the vehicle, such as on the roof, windshield, mirrors, etc.

[0137] In some embodiments, a vehicle configured to operate as a weather station server may be assigned the role of acquiring weather data for a specific geographical location, such as a city or suburb. In addition, or instead, such a vehicle may acquire weather data for one or more specific weather services, such as local, national, or global radio, television, or online weather services.

[0138] In some embodiments, in addition to, or instead of, a vehicle system that adjusts sensor field of view volume, the vehicle system, or other computing system such as the remote computing system 302, may be configured to change the vehicle's route based on weather conditions. For example, the second vehicle 706 in Figure 7 may determine that there is adverse weather in area 710 and that the vehicle system of the second vehicle 706 has reduced at least one sensor field of view volume beyond a threshold due to the adverse weather, and report this to the remote computing system 302 or one of the servers 700, 702. The threshold may be a value that indicates a field of view volume that at least one sensor reduced above that value may not be reliable to use when navigating the adverse weather. Thus, if the remote computing system 302, or other computing system associated with the first vehicle 704 (including the vehicle system of the first vehicle), determines that the reduction has exceeded a threshold, the remote computing system 302 may accordingly change the course of the first vehicle 704 to take an alternative route (and not reach area 710). Other examples are possible, such as scenarios where it may be desirable for the vehicle to avoid mild adverse weather.

[0139] The decision to change a vehicle's route may also be based on other factors, such as the size of the area where the weather conditions are occurring. For example, if the area of ​​fog is determined to be smaller than a threshold size (e.g., a circular area with a radius of 5 kilometers or less), the vehicle may change its route to avoid that area, as the area is small enough that changing the vehicle's route may not significantly increase the vehicle's estimated travel time. Other examples are similarly possible.

[0140] In some situations, it may be desirable for the vehicle system to proactively perform at least one of the disclosed actions before reaching a specific location along the travel path (e.g., an area with rain, a section of a busy road) associated with a determined operating environment. For example, the vehicle system may be configured to perform an action to adjust the field of view volume of at least one sensor within a threshold period (e.g., 5 minutes) before the vehicle is estimated to reach a particular location. In addition or alternatively, the vehicle system may be configured to perform an action to adjust the field of view volume of at least one sensor within a threshold distance (e.g., 1.6 kilometers) before the vehicle is estimated to reach a particular location. To facilitate these decisions, the vehicle system may store and continuously update data representing the estimated travel path and estimated time to one or more points along the travel path, including the final destination and / or one or more intermediate points. Other examples are similarly possible.

[0141] Figure 8 is a flowchart of another method 800 according to an exemplary embodiment. Method 800 may include one or more operations, functions, or actions, as illustrated by one or more of blocks 802-806.

[0142] In block 802, method 800 includes operating sensors of an autonomous vehicle according to a first field of view volume, where the first field of view volume is associated with a first operating environment of the autonomous vehicle.

[0143] In block 804, method 800 includes receiving data indicating a second operating environment of the autonomous vehicle, the second operating environment being associated with environmental conditions within the autonomous vehicle's environment.

[0144] In block 806, method 800 includes automatically adjusting the operation of a sensor to operate according to a second field of view volume, the second field of view volume being associated with a second operating environment.

[0145] In some embodiments, the sensor may be one of LiDAR, radar, and a camera.

[0146] In some embodiments, the environmental conditions may be meteorological conditions associated with one of fog, rain, and snow.

[0147] In some embodiments, environmental conditions may be associated with solar conditions, and solar conditions may be associated with a time of day, such as daytime, twilight, or nighttime.

[0148] In some embodiments, method 800 may also include controlling the autonomous vehicle using data from sensors operating in a second field of view volume. The act of controlling the autonomous vehicle using data from sensors operating in a second field of view volume may include adjusting the speed of the autonomous vehicle.

[0149] In some embodiments, the autonomous vehicle may be a first autonomous vehicle, and the act of receiving data indicating a second operating environment for the autonomous vehicle may include receiving data indicating the operating environment for the second autonomous vehicle. The second autonomous vehicle may be operated on a planned route of the first autonomous vehicle.

[0150] In some embodiments, the act of receiving data indicating a second operating environment of an autonomous vehicle may include receiving data on environmental conditions in the environment of the second autonomous vehicle.

[0151] In some embodiments, the act of automatically adjusting the sensor's operation to operate according to a second field of view volume may include discarding data associated with a field of view that exceeds the maximum field of view associated with the second field of view volume.

[0152] In some embodiments, the act of automatically adjusting the operation of the sensor to operate according to a second field of view volume may include adjusting the azimuth or elevation angle associated with the sensor to the respective azimuth or elevation angle associated with the second field of view volume, based on the direction of travel of the autonomous vehicle.

[0153] As described above, the field of view volume of at least one of the vehicle's sensors may be adjusted based on the vehicle's ODD. A vehicle may have multiple ODDs, each ODD may include at least one of environmental conditions, geographical conditions, time conditions, traffic conditions, or road conditions. Furthermore, each ODD may be associated with a predetermined field of view volume of at least one of one or more sensors that represent the space around the vehicle where at least one sensor is expected to detect an object at a certain level of confidence. In some examples, the vehicle system may store in memory the identifier of each ODD, as well as, for each ODD, the conditions included in the ODD, and the predetermined field of view volume of at least one of the one or more sensors.

[0154] Figure 9 is a flowchart of another method 900 according to an exemplary embodiment. Method 900 may include one or more operations, functions, or actions, as illustrated by one or more of blocks 902 to 906.

[0155] In block 902, method 900 includes identifying a plurality of ODDs of an autonomous vehicle, where each ODD includes at least one of environmental conditions, geographical conditions, time conditions, traffic conditions, or road conditions, and each ODD is associated with a predetermined field of view volume of at least one of one or more sensors. In some examples, the act of identifying a plurality of ODDs may include referring to a location in memory where the vehicle's ODDs are stored.

[0156] In block 904, method 900 includes associating an autonomous vehicle with a first ODD among a plurality of ODDs. In some examples, the vehicle system may use one or more sensors from the vehicle and / or one or more sensors located away from the vehicle (e.g., sensors from another vehicle, or sensors mounted on road structures such as stop signs, median strips, traffic lights, etc.) to detect one or more conditions of the vehicle's environment and select one of the plurality of ODDs, i.e., the ODD containing the detected conditions. The act of associating the vehicle with a first ODD may include the vehicle system configuring itself, or at least one of its subsystems, to operate using one or more parameters associated with the first ODD. Thus, in response to associating the vehicle with a first ODD, the vehicle system may operate at least one of the vehicle's sensors using a predetermined field of view volume associated with the first ODD.

[0157] In block 906, method 900 includes detecting a change in the operating environment of an autonomous vehicle. The act of detecting a change may include detecting the change using one or more of the vehicle's sensors. In addition or alternatively, the act of detecting a change may include detecting the change using an external sensor different from the vehicle's sensors. In particular, the external sensor may be located away from the vehicle itself, but may be directly linked to the vehicle system, for example, through a wireless communication interface, or may be indirectly linked to the vehicle system, for example, when the vehicle system receives data from the external sensor via a server or other intermediate computing device. In some examples, the external sensor may be mounted on a road structure such as a stop sign, traffic light, median strip, bridge, curb, or reflector, among other possible structures.

[0158] In block 908, method 900 includes associating an autonomous vehicle with a second ODD among a plurality of ODDs in response to a detection. In some examples, the act of associating a vehicle with a second ODD may include configuring at least one subsystem of the vehicle system itself or its subsystems to operate using one or more parameters associated with the second ODD.

[0159] In block 910, method 900 includes operating at least one sensor using a predetermined field of view volume associated with a second ODD, depending on whether the autonomous vehicle is associated with a second ODD.

[0160] In some implementations, operating at least one sensor with a predetermined field of view volume associated with a second ODD may include operating at least one sensor with a smaller field of view volume than when operating at least one sensor with a predetermined field of view volume associated with a first ODD. For example, the first ODD may include environmental conditions of clear weather, and the second ODD may include one or more environmental conditions of rain, fog, or snow, thus making it beneficial to use a smaller field of view volume for the second ODD. In other implementations, operating at least one sensor with a predetermined field of view volume associated with a second ODD may include operating at least one sensor with a larger field of view volume than when operating at least one sensor with a predetermined field of view volume associated with a first ODD.

[0161] As described herein, the first ODD and the second ODD may include other conditions. For example, the first ODD may include a first time condition of a first time (e.g., morning), and the second ODD may include a second time condition of a second time (e.g., evening or night). Another example is that the first ODD may include a first traffic condition (e.g., light or no traffic), and the second ODD may include a second traffic condition (e.g., heavy or congested traffic). Yet another example is that the first ODD may include a first speed limit on the road the vehicle is traveling on, and the second ODD may include a second speed limit that is higher or lower than the first speed limit. Yet another example is that the first ODD may include a first geographically defined area (e.g., public road or private land) the vehicle is traveling on, and the second ODD may include a second geographically defined area (e.g., highway or public road) the vehicle is traveling on. Other examples are similarly possible.

[0162] As used herein, the terms “substantially,” “approximately,” or “about” mean that the listed characteristics, parameters, values, or geometric planarity do not need to be achieved exactly, but deviations or variations may occur, including, for example, tolerances, measurement errors, limits of measurement accuracy, and other factors known to those skilled in the art, in an amount that does not preclude the effect that the characteristic is intended to provide.

[0163] While various exemplary embodiments and models are disclosed herein, other embodiments and models will be apparent to those skilled in the art. The various exemplary embodiments and models disclosed herein are for illustrative purposes only and are not intended to limit the scope and spirit of the invention, as set forth by the following claims.

Claims

1. It is a system, One or more sensors, each of which is configured to operate according to a field of view volume, wherein the field of view volume represents the space around the autonomous vehicle in which the sensor is expected to detect an object at a confidence level higher than a predetermined confidence threshold, One or more processors coupled to one or more of the aforementioned sensors, The system comprises a memory connected to one or more processors and in which instructions are stored, and when an instruction is executed by one or more processors, the instructions are sent to the one or more processors. To determine the operating environment of the autonomous vehicle which may affect the field of view volume of one or more of the sensors, A system that, based on the determined operating environment of the autonomous vehicle, performs an operation which includes adjusting the field of view volume of at least one of the one or more sensors from a first field of view volume to an adjusted field of view volume different from the first field of view volume, and if the at least one sensor outputs sensor data corresponding to a field of view volume exceeding the adjusted field of view volume, ignoring the sensor data corresponding to the portion exceeding the adjusted field of view volume.

2. The memory stores a plurality of operating environments for the autonomous vehicle, and a mapping between each of the plurality of operating environments and the corresponding adjusted field of view volume of at least one of the one or more sensors. The system according to claim 1, wherein adjusting the field of view volume based on the operating environment of the autonomous vehicle includes selecting the adjusted field of view volume corresponding to the determined operating environment of the autonomous vehicle.

3. The system according to claim 2, wherein the plurality of operating environments of the autonomous vehicle include two or more of the following: (i) default state, (ii) sunny weather state, (iii) daytime operating state, (iv) nighttime operating state, (v) rainy weather state, (vi) snowy weather state, (v) foggy weather state, (viiii) state for a particular type of road on which the autonomous vehicle is traveling, (ix) state in which at least a threshold number of vehicles are present on the road within a threshold distance from the autonomous vehicle, or (x) state in which at least one of the one or more sensors has a sensor error.

4. The corresponding adjusted field of view volume is part of a corresponding adjusted field of view volume set, which includes each adjusted field of view volume for each of the multiple sensor types. The system according to claim 2, wherein the plurality of sensor types include two or more of LIDAR sensors, radar sensors, or cameras.

5. The one or more sensors include a plurality of sensor sets, each sensor set having a respective sensor type. The system according to claim 1, wherein adjusting the field of view volume of at least one sensor includes adjusting the field of view volume of all sensors in at least one sensor set of the sensor set.

6. The system according to claim 5, wherein the plurality of sensor sets include two or more of the following: a set of LIDAR sensors, a set of radar sensors, or a set of cameras.

7. The aforementioned operation, The system according to claim 1, further comprising controlling the autonomous vehicle to operate using the at least one sensor having the adjusted field of view volume.

8. The system according to claim 7, wherein controlling the autonomous vehicle to operate using the at least one sensor having the adjusted field of view volume includes identifying sensor data corresponding to a parameter value greater than the maximum parameter value of the adjusted field of view volume.

9. The at least one sensor includes a LIDAR sensor, The system according to claim 7, wherein controlling the autonomous vehicle to operate using the at least one sensor having the adjusted field of view volume includes controlling the LIDAR sensor to acquire sensor data by transmitting one or more laser pulses having adjusted power levels associated with the adjusted field of view volume.

10. The at least one sensor includes a radar sensor, The system according to claim 7, wherein controlling the autonomous vehicle to operate using the at least one sensor having the adjusted field of view volume includes controlling the radar sensor to acquire sensor data by transmitting one or more radio waves having adjusted radio wave characteristics associated with the adjusted field of view volume.

11. A method performed by a computing device configured to control the operation of an autonomous vehicle, wherein the method is Determining the operating environment of the autonomous vehicle which may affect the field of view volume of one or more sensors provided by the autonomous vehicle, wherein each of the one or more sensors is configured to operate according to the field of view volume, and the field of view volume represents the space around the autonomous vehicle in which the sensor is expected to detect an object at a confidence level higher than a predetermined confidence threshold, Based on the determined operating environment of the autonomous vehicle, the field of view volume of at least one of the one or more sensors is adjusted from a first field of view volume to a modified field of view volume different from the first field of view volume. A method for controlling the autonomous vehicle to operate using the at least one sensor having the adjusted field of view volume, the method comprising: ignoring the sensor data corresponding to the portion of the field of view volume that exceeds the adjusted field of view volume if the at least one sensor outputs sensor data corresponding to the field of view volume that exceeds the adjusted field of view volume.

Images