Reducing auto-exposure latency

Abstract

To reduce auto-exposure latency.SOLUTION: An example embodiment includes a method of reducing auto-exposure latency. The method includes determining, by a processor, a first setting of an exposure parameter for a first frame to be captured by an image sensor. The first setting of the exposure parameter is determined based at least in part on characteristics of a previous frame captured by the image sensor. The first setting of the exposure parameter is determined during a first frame period associated with capturing the first frame. The method also includes initiating, by the processor, a first frame exposure operation based on the first setting of the exposure parameter. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

Description

[Background technology]

[0001] Unless otherwise stated herein, the statements in this section are not prior art to the claims of this application and should not be admitted as prior art by inclusion in this section.

[0002] An autonomous vehicle may include one or more image sensors that capture images of the surrounding environment. The image sensors typically have different exposure settings that are selectively used to improve the quality of the images. As a non-limiting example, if the surrounding environment is relatively dark, the exposure time used to capture the image with the image sensor may be adjusted to have a relatively long duration, which in turn may increase the illumination characteristics of the image (e.g., brighten the image). However, if the surrounding environment is relatively bright, the exposure time used to capture the image with the image sensor may be adjusted to have a relatively short duration, which in turn may decrease the illumination characteristics of the image (e.g., darken the image). Typically, there is a delay between when the exposure setting is adjusted and when the image sensor captures the image with the adjusted exposure setting. For example, the image sensor may have a single frame delay or a multiple frame delay between when the exposure setting is changed and when the frame acquired with the new setting is read. As a result, during the frame delay, the image sensor may capture one or more overexposed and / or underexposed frames, which take up storage space and provide little (or no) benefit to the operation of the autonomous vehicle. Summary of the Invention

[0003] The techniques described herein reduce latency (e.g., frame delay) associated with auto-exposure operations. In particular, the techniques described herein enable target exposure parameters, such as exposure time, analog gain, or digital gain, to be determined and executed in a single frame period. For example, according to one embodiment, during a particular frame period, the on-chip processor may analyze a previous frame captured by the image sensor to determine target exposure parameters for a "current" frame to be captured by the image sensor during the particular frame period. As another example, according to one embodiment, during a particular frame period, the on-chip processor may sample a subset of pixels on the image sensor to generate a "pre-exposure" frame and analyze the pre-exposure frame to determine target exposure parameters. In response to determining the target exposure parameters (e.g., based on a previous frame or based on the pre-exposure frame), the image sensor may capture a current frame using the target exposure parameters such that there is no frame delay between determining the target exposure parameters and reading an image frame that is captured using the target exposure parameters.

[0004] The system includes an image sensor and a processor coupled to the image sensor. The processor is configured to determine a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of a previous frame captured by the image sensor. The first setting of exposure parameters is determined during a first frame period associated with capturing the first frame. The image sensor is configured to perform a first frame exposure operation during the first frame period. The first frame exposure operation is based on the first setting of exposure parameters.

[0005] The method includes determining, by the processor, a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of a previous frame captured by the image sensor. The first setting of exposure parameters is determined during a first frame period associated with capturing the first frame. The method also includes initiating, by the processor, a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

[0006] The non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform operations. The operations include determining a first setting of exposure parameters for a first frame captured by an image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of a previous frame captured by the image sensor. The first setting of exposure parameters is determined during a first frame period associated with capturing the first frame. The operations also include initiating a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

[0007] The system includes an image sensor and a processor coupled to the image sensor. The processor is configured to receive a first pre-exposure frame from the image sensor during a first frame period associated with capture of the first frame. The processor is also configured to determine a first setting of exposure parameters for the first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of the first pre-exposure frame, the first setting of exposure parameters being determined during the first frame period. The image sensor is configured to perform a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

[0008] The method includes receiving, by the processor, a first pre-exposure frame from the image sensor during a first frame period associated with capture of the first frame. The method also includes determining, by the processor, a first setting of exposure parameters for the first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of the first pre-exposure frame, the first setting of exposure parameters being determined during the first frame period. The method further includes initiating, by the processor, a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

[0009] The non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform operations. The operations include receiving a first pre-exposure frame from an image sensor during a first frame period associated with capturing the first frame. The operations also include determining a first setting of exposure parameters for the first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of the first pre-exposure frame, the first setting of exposure parameters being determined during the first frame period. The operations further include initiating a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

[0010] These and other aspects, advantages, and alternatives will become apparent to those skilled in the art from a reading of the following detailed description, appropriately taken from the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a functional block diagram illustrating a vehicle, in accordance with an exemplary embodiment. [Figure 2A] FIG. 2A is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2B] FIG. 2B is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2C] FIG. 2C is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2D] FIG. 2D is an illustrative diagram of a vehicle's physical configuration in accordance with an exemplary embodiment. [Figure 2E] FIG. 2E is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2F] FIG. 2F is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2G] FIG. 2G is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2H] FIG. 2H is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2I] FIG. 2I is an illustrative diagram of a vehicle's physical configuration, according to an exemplary embodiment. [Figure 2J] FIG. 2J is an illustrative diagram of the fields of view of various sensors in accordance with an example embodiment. [Figure 2K] FIG. 2K is an illustrative diagram of beam steering relative to a sensor in accordance with an example embodiment. [Diagram 3] FIG. 3 is a conceptual illustration of wireless communication between various computing systems associated with an autonomous or semi-autonomous vehicle, in accordance with an example embodiment. [Figure 4] FIG. 4 is a system operable to reduce auto-exposure latency, according to an example embodiment. [Diagram 5] FIG. 5 is a process for reducing auto-exposure latency based on frame characteristics of previously captured frames according to an example embodiment. [Figure 6] FIG. 6 is a process for reducing auto-exposure latency based on frame characteristics of a pre-exposure frame according to an example embodiment. [Figure 7] FIG. 7 is another process for reducing auto-exposure latency, according to an example embodiment. [Figure 8] FIG. 8 is a method for reducing auto-exposure latency according to an example embodiment. [Figure 9] FIG. 9 is another method for reducing auto-exposure latency, according to an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Exemplary methods and systems are contemplated herein. Any exemplary embodiment or example feature described herein should not necessarily be construed as preferred or advantageous over other embodiments or features. Moreover, the exemplary embodiments described herein are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein. In addition, the specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments can include more or less of each element shown in a given figure. Additionally, some of the illustrated elements can be combined or omitted. Still further, exemplary embodiments can include elements not illustrated in the figures.

[0013] The techniques described herein reduce the latency associated with auto-exposure operations. In particular, the techniques described herein allow for determining and performing (e.g., adjusting) exposure parameters for a frame, such as exposure time, during a frame period associated with capturing the frame. As a result, instead of having a delay (e.g., a single frame delay or a multi-frame delay) between when a target exposure parameter is determined and when a captured frame is read from an image sensor using the target exposure parameter, the techniques described herein can reduce (or eliminate) the delay such that a frame having a target exposure parameter is read during the same frame period in which the target exposure parameter is determined. By reducing the above-mentioned latency, the techniques described herein can reduce the number of overexposed frames captured and can reduce the number of underexposed frames captured.

[0014] As used herein, the "frame period" of a current frame corresponds to the period of time that begins immediately after the capture of the previous frame and ends immediately after the capture of the current frame. Thus, when the shutter closes to complete the capture of the previous frame, the frame period of the previous frame ends and the frame period of the current frame begins. Additionally, when the shutter closes to complete the capture of the current frame, the frame period of the current frame ends. The length of time that defines a frame period can be defined as the inverse of the frame rate, which is specific to the associated camera system.

[0015] To reduce latency as described above, according to some implementations, during a frame period of a "current frame" (e.g., a frame period associated with the capture of the current frame), the processor may dynamically modify the exposure parameters used to capture the current frame based on characteristics of the previously captured frame. For illustrative purposes, during the frame period of the current frame, the processor may analyze the previous captured frame and determine the setting of the exposure parameters based on the analysis. As a non-limiting example, if during the frame period of the current frame, the illumination metrics of the previous frame did not meet a lower illumination threshold (e.g., the previous frame was too dark), the processor may determine that the exposure time of the current frame should be extended and implement an extended exposure time at the image sensor. Conversely, if during the frame period of the current frame, the illumination metrics of the previous frame did not meet an upper illumination threshold (e.g., the previous frame was too bright), the processor may determine that the exposure time of the current frame should be shortened and implement a shortened exposure time at the image sensor. Of course, in addition to analyzing the previous frame to determine the exposure parameters for the current frame, the processor may also analyze other data, such as one or more additional previously captured frames, histograms of data associated with the previously captured frames, external inputs indicating data collected from other sensors, etc. After the exposure parameters have been determined, the image sensor may perform a frame exposure operation using the exposure parameters to capture the current frame during the frame period of the current frame.

[0016] According to another implementation, to reduce latency as described above, during the frame period of the current frame, the processor may dynamically modify the exposure parameters used to capture the current frame based on characteristics of a "pre-exposure" frame. In this implementation, at the beginning of the frame period of the current frame, the processor may sample (for a short period of time) a subset of image pixels on the image sensor to generate a pre-exposure frame. The subset of image pixels may correspond to a particular region of interest or may correspond to a distributed region across the image sensor. The pre-exposure frame may be used by the processor to determine how or whether the exposure parameters should be adjusted in capturing the current frame. After determining how the exposure parameters should be adjusted, during the frame period of the current frame, the image sensor may perform a frame exposure operation using the adjusted exposure parameters to capture the current frame.

[0017] Although the exposure parameters may be adjusted based on the pre-exposure frame, in some implementations the exposure parameters may additionally be adjusted based on data from one or more other sensors. As a non-limiting example, at the start of a frame period of the current frame, the processor may also receive data (e.g., images) from one or more other sensors. In these implementations, the processor may adjust the exposure parameters of the current frame based on the pre-exposure frame and based on data (e.g., images) from the one or more other sensors.

[0018] The following description and the accompanying drawings clarify features of various exemplary embodiments. The embodiments provided are by way of example and are not intended to be limiting. Thus, dimensions of the drawings are not necessarily drawn to scale.

[0019] Certain embodiments are described herein with reference to the drawings. In the description, common features are indicated by common reference numbers throughout the drawings. In some figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically different, the same reference number is used for each, and the different instances are distinguished by the addition of a letter to the reference number. When features as a group or type are referred to herein (e.g., when no particular one of the features is referred to), the reference number is used without the identifying letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the identifying letter. For example, with reference to FIG. 4, multiple frames are illustrated and associated with reference numbers 450A, 450B, etc. When referring to a particular one of these frames, such as frame 450A, the identifying letter "A" is used. However, when referring to any one of these frames or these frames as a group, the reference number 450 is used without the identifying letter.

[0020] An exemplary system within the scope of the present disclosure will now be described in more detail. The exemplary system may be implemented in or take the form of an automobile. Additionally, the exemplary system may also be implemented in or take the form of a variety of vehicles, such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor trailers, etc.), motorcycles, buses, airplanes, helicopters, drones, lawnmowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment or vehicles, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, trams, golf carts, trains, trolleys, walkway transport vehicles, robotic devices, and the like. Other vehicles are possible as well. Furthermore, in some embodiments, the exemplary system may not include a vehicle.

[0021] Referring now to the figures, FIG. 1 is a functional block diagram illustrating an example vehicle 100 that may be configured to operate fully or partially in an autonomous mode. More specifically, the vehicle 100 may operate in the autonomous mode without human interaction through receiving control instructions from a computing system. As part of the operation in the autonomous mode, the vehicle 100 may use sensors to detect and possibly identify objects in the surrounding environment to enable safe navigation. Additionally, the example vehicle 100 may operate in a partially autonomous (i.e., semi-autonomous) mode in which some functions of the vehicle 100 are controlled by a human driver of the vehicle 100 and some functions of the vehicle 100 are controlled by a computing system. For example, the vehicle 100 may also include subsystems that enable the driver to control the operation of the vehicle 100, such as steering, acceleration, and braking, while the computing system performs assistance functions, such as lane departure warning / lane keeping assist or adaptive cruise control, based on other objects (e.g., vehicles) in the surrounding environment.

[0022] As described herein, in a partially autonomous driving mode, the vehicle assists with one or more driving operations (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), and emergency braking), but the human driver is expected to provide situational awareness of the vehicle 100's surroundings and oversee the assisted driving operations. Here, while the vehicle 100 may perform all driving tasks in certain situations, the human driver is expected to be responsible for assuming control as necessary.

[0023] For simplicity and brevity, the various systems and methods are described below in conjunction with autonomous vehicles, but these or similar systems and methods may be used in various driver assistance systems that do not reach the level of a fully autonomous driving system (i.e., a partially autonomous driving system). In the United States, the Society of Automotive Engineers (SAE) defines different levels of automated driving behavior to indicate how much or how little control the vehicle has over the driving, but different organizations in the United States or other countries may classify the levels differently. More specifically, the disclosed systems and methods may be used in SAE Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, etc., and other driver support. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems that are capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles that use SAE Level 4 automated driving systems, which operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such systems, accurate lane estimation is performed automatically without driver input or control (e.g., while the vehicle is moving), resulting in improved reliability of vehicle positioning and navigation, and overall safety of autonomous, semi-autonomous, and other driver assistance systems. As noted above, in addition to the manner in which the SAE classifies levels of autonomous driving operation, other organizations in the United States or other countries may classify levels of autonomous driving operation differently. Without being limited thereto, the systems and methods disclosed herein may be used for driver assistance systems defined by the levels of autonomous driving operation of these other organizations.

[0024] As shown in FIG. 1, the vehicle 100 may include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power source 110, a computer system 112 (which may also be referred to as a computing system) having data storage 114, and a user interface 116. In other examples, the vehicle 100 may include more or fewer subsystems, each of which may include multiple elements. The subsystems and components of the vehicle 100 may be interconnected in various ways. In addition, the functions of the vehicle 100 described herein may be divided into additional functional or physical components or combined into fewer functional or physical components within an embodiment. For example, the control system 106 and the computer system 112 may be combined into a single system that operates the vehicle 100 according to various operations.

[0025] Propulsion system 102 may include one or more components operable to provide powered motion for vehicle 100 and may include an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121, among other possible components. For example, engine / motor 118 may be configured to convert energy source 119 into mechanical energy and may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine, among other possible options. For example, in some embodiments, propulsion system 102 may include multiple types of engines and / or motors, such as gasoline engines and electric motors.

[0026] Energy source 119 represents an energy source that may fully or partially power one or more systems (e.g., engine / motor 118) of vehicle 100. For example, energy source 119 may correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, energy source 119 may include a combination of a fuel tank, a battery, a capacitor, and / or a flywheel.

[0027] The transmission 120 may transfer mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. As such, the transmission 120 may include a gearbox, a clutch, a differential, and a drive shaft, among other possible components. The drive shaft may include an axle that connects to one or more of the wheels / tires 121.

[0028] The wheels / tires 121 of the vehicle 100 may have a variety of configurations within the exemplary embodiment. For example, the vehicle 100 may exist in the form of a unicycle, a bicycle / motorcycle, a tricycle, or four wheels of a car / truck, among other possible configurations. Thus, the wheels / tires 121 may be connected to the vehicle 100 in a variety of ways and may exist in different materials, such as metal and rubber.

[0029] The sensor system 104 may include various types of sensors, such as a global positioning system (GPS) 122, an inertial measurement unit (IMU) 124, a radar 126, a lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, among other possible sensors. In some embodiments, the sensor system 104 includes sensors configured to monitor internal systems of the vehicle 100 (e.g., O 2 monitors, fuel gauge, engine oil temperature, and brake wear).

[0030] The GPS 122 may include a transceiver operable to provide information regarding the position of the vehicle 100 relative to the Earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes and may sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 while the vehicle 100 is stationary or in motion.

[0031] Radar 126 may represent one or more systems configured to sense objects in the environment surrounding vehicle 100 using radio signals, including the speed and orientation of the objects. Thus, radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, radar 126 may correspond to a mountable radar configured to obtain measurements of the environment surrounding vehicle 100.

[0032] The LIDAR 128 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components, and may operate in a coherent mode (e.g., using heterodyne detection, etc.) or in an incoherent detection mode (i.e., time-of-flight mode). In some embodiments, the one or more detectors of the LIDAR 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes). In some examples, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Furthermore, such photodetectors may be arranged in an array (e.g., as in silicon photomultipliers (SiPMs)) (e.g., through serial electrical connections). In some examples, the one or more photodetectors are devices that operate in Geiger mode, and the LIDAR includes subcomponents designed for such Geiger mode operation.

[0033] Camera 130 may include one or more devices (eg, a still camera, a video camera, a thermal imaging camera, a stereo camera, and a night vision camera) configured to capture images of the environment surrounding vehicle 100.

[0034] The steering sensor 123 may sense the steering angle of the vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representative of the angle of the steering wheel. In some embodiments, the steering sensor 123 may measure the angle of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the forward axle of the vehicle 100. The steering sensor 123 may also be configured to measure a combination (or a subset) of the steering wheel angle, the electrical signal representative of the steering wheel angle, and the angles of the wheels of the vehicle 100.

[0035] The throttle / brake sensor 125 may detect the position of either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angle of both the accelerator pedal (throttle) and the brake pedal, or may measure an electrical signal representative of, for example, the accelerator pedal (throttle) angle and / or the brake pedal angle. The throttle / brake sensor 125 may also measure the angle of a throttle body of the vehicle 100, which may include part of the physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., butterfly valves and carburetors). Additionally, the throttle / brake sensor 125 may measure the pressure of one or more brake pads on a rotor of the vehicle 100, or a combination (or subset) of the accelerator pedal (throttle) and brake pedal angles, an electrical signal representative of the accelerator pedal (throttle) and brake pedal angles, the throttle body angle, and the pressure that at least one brake pad exerts on a rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure pressure applied to a vehicle pedal, such as a throttle or brake pedal.

[0036] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / pathfinding system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the heading of the vehicle 100, and the throttle 134 may control the operating speed of the engine / motor 118 to control the acceleration of the vehicle 100. The braking unit 136 may decelerate the vehicle 100, which may involve slowing the wheels / tires 121 using friction. In some embodiments, the braking unit 136 may convert the kinetic energy of the wheels / tires 121 into electrical current for subsequent use by one or more systems of the vehicle 100.

[0037] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithms capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide an assessment based on the incoming sensor data, such as an assessment of individual objects and / or features, an assessment of a particular situation, and / or an assessment of possible influences within a given situation.

[0038] The computer vision system 140 may include hardware and software (e.g., a general-purpose processor such as a central processing unit (CPU), a special-purpose processor such as a graphics processing unit (GPU) or a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) operable to process and analyze images to determine moving objects (e.g., other vehicles, pedestrians, bicyclists, or animals) and non-moving objects (e.g., traffic signals, roadway boundaries, speed bumps, or potholes). Thus, the computer vision system 140 may use object recognition, structure-from-motion (SFM), video tracking, and other algorithms used in computer vision, for example, to recognize objects, map the environment, track objects, estimate the speed of objects, etc.

[0039] The navigation / routing system 142 may determine a driving path for the vehicle 100, which may involve dynamically adjusting navigation during operation. Thus, the navigation / routing system 142 may use data from sensor fusion algorithms 138, GPS 122, and maps, among other sources, to navigate the vehicle 100. The obstacle avoidance system 144 may evaluate potential obstacles based on the sensor data and cause systems of the vehicle 100 to avoid or otherwise navigate the potential obstacles.

[0040] 1, the vehicle 100 may also include peripherals 108, such as a wireless communication system 146, a touchscreen 148, an internal microphone 150, and / or a speaker 152. The peripherals 108 may provide controls or other elements for a user to interact with a user interface 116. For example, the touchscreen 148 may provide information to a user of the vehicle 100. The user interface 116 may also accept input from a user via the touchscreen 148. The peripherals 108 may also enable the vehicle 100 to communicate with devices, such as other vehicle devices.

[0041] The wireless communication system 146 may communicate with one or more devices directly or wirelessly through a communication network. For example, the wireless communication system 146 may use 3G cellular communications such as Code Division Multiple Access (CDMA), Evolution Data Optimized (EVDO), Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or cellular communications such as 4G Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE), or 5G. Alternatively, the wireless communication system 146 may communicate with a wireless local area network (WLAN) using WIFI or other possible connections. The wireless communication system 146 may also communicate directly with devices using, for example, an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are possible within the context of the present disclosure. For example, the wireless communication system 146 may include one or more Dedicated Short Range Communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0042] Vehicle 100 may include a power source 110 for providing power to the components. Power source 110 may include a rechargeable lithium ion or lead acid battery in some embodiments. For example, power source 110 may include one or more batteries configured to provide power. Vehicle 100 may also use other types of power sources. In an exemplary embodiment, power source 110 and energy source 119 may be integrated into a single energy source.

[0043] Vehicle 100 may also include a computer system 112 for performing operations such as those described therein. Thus, computer system 112 may include at least one processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored in a non-transitory computer-readable medium, such as data storage 114. In some embodiments, computer system 112 may represent multiple computing devices that may function to control individual components or subsystems of vehicle 100 in a distributed manner.

[0044] In some embodiments, data storage 114 may include instructions 115 (e.g., program logic) executable by processor 113 to perform various functions of vehicle 100, including those described above in connection with Figure 1. Data storage 114 may also include additional instructions, including instructions to transmit data to, receive data from, interact with, and / or control one or more of propulsion system 102, sensor system 104, control system 106, and peripherals 108.

[0045] In addition to instructions 115, data storage 114 may store data such as road maps, route information, among other information. Such information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0046] Vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of vehicle 100. User interface 116 may control or allow for control of content and / or the layout of interactive images that may be displayed on touchscreen 148. Additionally, user interface 116 may include one or more input / output devices in the set of peripherals 108, such as wireless communication system 146, touchscreen 148, microphone 150, and speaker 152.

[0047] The computer system 112 may control functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, or the control system 106) and from the user interface 116. For example, the computer system 112 may utilize inputs from the sensor system 104 to estimate outputs generated by the propulsion system 102 and the control system 106. Depending on the embodiment, the computer system 112 may be operable to monitor many aspects of the vehicle 100 and its subsystems. In some embodiments, the computer system 112 may disable some or all functions of the vehicle 100 based on signals received from the sensor system 104.

[0048] The components of vehicle 100 may be configured to function in an interconnected manner with other components within their respective systems or externally. For example, in an exemplary embodiment, camera 130 may capture multiple images that may represent information about the state of the surrounding environment of vehicle 100 operating in an autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, computer vision system 140 may be capable of recognizing the slope (gradient) or other features based on multiple images of the road. Additionally, a combination of GPS 122 and the features recognized by computer vision system 140 may be used along with map data stored in data storage 114 to determine specific road parameters. Furthermore, radar 126 and / or lidar 128, and / or some other environmental mapping, range, and / or positioning sensor systems may also provide information about the vehicle's surroundings.

[0049] In other words, a combination of various sensors (which may be referred to as input indicator sensors and output indicator sensors) and computer system 112 may interact to provide an indication of an input or an indication of the vehicle's surroundings that is provided to control the vehicle.

[0050] In some embodiments, computer system 112 may make decisions regarding various objects based on data provided by systems other than a wireless system. For example, vehicle 100 may have laser or other optical sensors configured to sense objects within the field of view of the vehicle. Computer system 112 may use output from the various sensors to determine information regarding objects within the field of view of the vehicle and may determine distance and direction information to the various objects. Computer system 112 may also determine whether an object is desirable or undesirable based on output from the various sensors.

[0051] 1 illustrates various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) as being integrated into vehicle 100, one or more of these components may be separately mounted or associated with vehicle 100. For example, data storage 114 may exist partially or completely separate from vehicle 100. Thus, vehicle 100 may be provided in the form of device elements that may be located separately or together. The device elements that make up vehicle 100 may be communicatively coupled together in a wired and / or wireless manner.

[0052] 2A-2E illustrate an example vehicle 200 (e.g., a fully autonomous vehicle or a semi-autonomous vehicle) that may include some or all of the features described in connection with vehicle 100 with reference to FIG. 1. Vehicle 200 is illustrated in FIGS. 2A-2E as a van with side mirrors for illustrative purposes, but the disclosure is not so limited. For example, vehicle 200 may represent a truck, an automobile, a semi-trailer truck, a motorcycle, a golf cart, an off-road vehicle, an agricultural vehicle, or any other vehicle described elsewhere herein (e.g., a bus, a boat, an airplane, a helicopter, a drone, a lawn mower, a bulldozer, a submarine, an all-terrain vehicle, a snowmobile, an aircraft, a recreational vehicle, an amusement park vehicle, farm equipment, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, a tram, a train, a trolley, a walkway transport vehicle, and a robotic device).

[0053] Exemplary vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidars, one or more radars, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones, and sonar devices), or one or more other sensors configured to sense information about the environment surrounding vehicle 200. In other words, any sensor system now known or hereafter created may be coupled to vehicle 200 and / or utilized in conjunction with various operations of vehicle 200. As an example, lidar may be utilized for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of vehicle 200. Additionally, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent a combination of the sensors described herein (e.g., one or more lidar and radar, one or more lidar and cameras, one or more cameras and radar, one or more lidar, cameras, and radar).

[0054] 2A-E are intended as non-limiting examples of the locations, numbers, and types of such sensor systems on an autonomous or semi-autonomous vehicle. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to accommodate vehicle size, shape, aerodynamics, fuel economy, aesthetics, or other conditions to reduce cost or to suit a particular environmental or application situation). For example, the sensor systems (e.g., 202 and 204) may be disposed in various other locations on the vehicle (e.g., at location 216) and may have a field of view that corresponds to the interior and / or surrounding environment of vehicle 200.

[0055] The sensor system 202 may include one or more sensors mounted on top of the vehicle 200 and configured to detect information about the environment surrounding the vehicle 200 and output an indication of the information. For example, the sensor system 202 may include any combination of cameras, radar, lidar, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones, and sonar devices). The sensor system 202 may include one or more movable mounts that may be operable to adjust the orientation of one or more sensors in the sensor system 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to obtain information from each direction around the vehicle 200. In another embodiment, the movable mount of the sensor system 202 may be movable in a scanning manner within a certain range of angles and / or azimuth and / or elevation angles. The sensor system 202 may be mounted on the roof of the vehicle, although other mounting locations are also contemplated.

[0056] Additionally, the sensors of sensor system 202 may be distributed at various locations and need not be co-located at a single location. Further, each sensor of sensor system 202 may be configured to be moved or scanned independently of the other sensors of sensor system 202. Additionally or alternatively, multiple sensors may be mounted at one or more of sensor locations 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two lidar devices mounted at a sensor location, and / or there may be one lidar device and one radar mounted at a sensor location.

[0057] One or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more LIDAR devices. For example, a LIDAR device may include multiple light emitter devices arranged over a range of angles relative to a given plane (e.g., an xy plane). For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot about an axis perpendicular to the given plane (e.g., a z-axis) to illuminate the environment surrounding the vehicle 200 with light pulses. Based on detection of various aspects of the reflected light pulses (e.g., elapsed time of flight, polarization, and intensity), information about the surrounding environment may be determined.

[0058] In an exemplary embodiment, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide respective point cloud information that may be associated with physical objects within the surrounding environment of vehicle 200. While vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are illustrated as including certain features, it will be understood that other types of sensor systems are contemplated within the scope of the present disclosure. Additionally, exemplary vehicle 200 may include any of the components described in connection with vehicle 100 of FIG.

[0059] In an exemplary configuration, one or more radars may be located on the vehicle 200. Similar to the radar 126 described above, the one or more radars may include an antenna configured to transmit and receive radio waves (e.g., electromagnetic waves having frequencies between 30 Hz and 300 GHz). Such radio waves may be used to determine the distance and / or speed of one or more objects in the surrounding environment of the vehicle 200. For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some examples, one or more radars may be located near the rear of the vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of the vehicle 200 for the presence of radio wave reflective objects. Similarly, one or more radars may be located near the front of the vehicle 200 (e.g., sensor systems 212, 214) to actively scan the environment near the front of the vehicle 200. The radar may be positioned in a location suitable for illuminating an area including the forward path of vehicle 200, for example, without being obstructed by other features of vehicle 200. For example, the radar may be embedded in and / or mounted on or near the front bumper, front headlights, cowl, and / or hood, etc. Additionally, one or more additional radars may be positioned to actively scan the sides and / or rear of vehicle 200 for the presence of radio wave reflective objects, such as by including such devices on or near the rear bumper, side panels, rocker panels, and / or undercarriage, etc.

[0060] Vehicle 200 may include one or more cameras. For example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more cameras. The cameras may be light-sensitive instruments, such as still cameras, video cameras, thermal imaging cameras, stereo cameras, night vision cameras, etc., configured to capture multiple images of the environment surrounding vehicle 200. To this end, the cameras may be configured to detect visible light and may additionally or alternatively be configured to detect light from other parts of the spectrum, such as infrared or ultraviolet light. The cameras may be two-dimensional detectors and may optionally have a sensitivity range in three-dimensional space. In some embodiments, the cameras may include range detectors configured to generate two-dimensional images, for example, indicative of distances from the camera to points in the surrounding environment. To this end, the cameras may use one or more range detection techniques. For example, the camera can provide range information by using structured light techniques, in which the vehicle 200 illuminates objects in the surrounding environment with a predetermined light pattern, such as a grid or checkerboard pattern, and uses the camera to detect reflections of the predetermined light pattern from the surrounding environment. Based on distortions in the reflected light pattern, the vehicle 200 can determine the distance to a point on the object. The predetermined light pattern can include infrared light, or other wavelengths of radiation suitable for such measurements. In some examples, the camera can be mounted on the inside of the windshield of the vehicle 200. Specifically, the camera can be positioned to capture images from a forward view relative to the orientation of the vehicle 200. Other mounting locations and viewing angles of the camera, whether internal or external to the vehicle 200, can be used. The camera can also have associated optics operable to provide an adjustable field of view. Furthermore, the camera can be mounted on the vehicle 200 with a movable mount to change the pointing angle of the camera, such as via a pan / tilt mechanism.

[0061] The vehicle 200 may also include one or more acoustic sensors used to sense the vehicle's 200 surrounding environment (e.g., one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). The acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, and micro-electromechanical system (MEMS) microphones) used to sense acoustic waves (i.e., pressure differentials) in the fluid (e.g., air) of the environment surrounding the vehicle 200. Such acoustic sensors may be used to identify sounds (e.g., sirens, human speech, animal sounds, and alarms) in the surrounding environment upon which a control strategy of the vehicle 200 may be based. For example, if the acoustic sensor detects a siren (e.g., a mobile siren, and / or a fire engine siren), the vehicle 200 may slow down and / or navigate to the edge of the road.

[0062] Although not shown in FIGS. 2A-2E, vehicle 200 may include a wireless communication system (e.g., similar to and / or in addition to wireless communication system 146 of FIG. 1). The wireless communication system may include a wireless transmitter and a wireless receiver that may be configured to communicate with devices external or internal to vehicle 200. Specifically, the wireless communication system may include a transceiver configured to communicate with other vehicles and / or computing devices, for example, in a vehicle communication system or a roadside station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transport systems.

[0063] The vehicle 200 may include one or more other components in addition to or instead of those shown. The additional components may include electrical or mechanical functions.

[0064] A control system of the vehicle 200 may be configured to control the vehicle 200 according to a control strategy from among a plurality of possible control strategies. The control system may be configured to receive information from sensors (on or off the vehicle 200) coupled to the vehicle 200, modify the control strategy (and associated driving behavior) based on the information, and control the vehicle 200 according to the modified control strategy. The control system may be further configured to monitor the information received from the sensors and continuously evaluate the driving conditions, and may be configured to modify the control strategy and driving behavior based on changes in the driving conditions. For example, the path taken by the vehicle from one destination to another may be modified based on the driving conditions. Additionally or alternatively, speed, acceleration, turn angle, following distance (i.e., distance to the vehicle ahead of the current vehicle), lane selection, etc. may all be modified in response to changes in driving conditions.

[0065] As noted above, in some embodiments, the vehicle 200 may take the form of a van, although alternative forms are also possible and contemplated herein. Accordingly, Figures 2F-2I illustrate an embodiment in which the vehicle 250 takes the form of a semi-truck. For example, Figure 2F illustrates a front view of the vehicle 250, and Figure 2G illustrates an isometric view of the vehicle 250. In an embodiment in which the vehicle 250 is a semi-truck, the vehicle 250 may include a tractor portion 260 and a trailer portion 270 (illustrated in Figure 2G). Figures 2H and 2I provide side and top views, respectively, of the tractor portion 260. Similar to the vehicle 200 illustrated above, the vehicle 250 illustrated in Figures 2F-2I may also include various sensor systems (e.g., similar to the sensor systems 202, 206, 208, 210, 212, 214 shown and described with reference to Figures 2A-2E). In some embodiments, the vehicle 200 of FIGS. 2A-2E may include only a single copy of some sensor systems (e.g., sensor system 204), while the vehicle 250 illustrated in FIGS. 2F-2I may include multiple copies of its sensor systems (e.g., as illustrated, sensor systems 204A and 204B).

[0066] While the figures and general description may refer to a given vehicle form (e.g., semi-truck vehicle 250 or van vehicle 200), it is understood that the embodiments described herein may be equally applied in various vehicle contexts (e.g., with modifications employed to account for the vehicle form factor). For example, sensors and / or other components described or illustrated as being part of van vehicle 200 may also be used in semi-truck vehicle 250 (e.g., for navigation and / or obstacle detection and avoidance).

[0067] FIG. 2J illustrates various sensor fields of view (e.g., associated with vehicle 250, described above). As described above, vehicle 250 may contain multiple sensors / sensor units. The various sensor locations may correspond to the sensor locations disclosed in FIGS. 2F-2I, for example. However, in some cases, the sensors may have other locations. For simplicity of the drawings, sensor location reference numbers are omitted from FIG. 2J. For each sensor unit of vehicle 250, FIG. 2J illustrates a representative field of view (e.g., fields of view labeled as 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The field of view of the sensor may include an angular region (e.g., azimuth region and / or elevation region) in which the sensor may detect objects.

[0068] FIG. 2K illustrates beam steering for a sensor of a vehicle (e.g., vehicle 250 shown and described with reference to FIGS. 2F-2J) according to an exemplary embodiment. In various embodiments, the sensor unit of vehicle 250 may be radar, lidar, sonar, etc. Additionally, in some embodiments, during operation of the sensor, the sensor may be scanned within the field of view of the sensor. Various different scan angles for the exemplary sensor are shown as regions 272, each indicating an angular region in which the sensor is operating. The sensor may periodically or repeatedly change the region in which it is operating. In some embodiments, multiple sensors may be used by vehicle 250 to measure region 272. Additionally, other regions may be included in other examples. For example, one or more sensors may measure aspects of trailer 270 of vehicle 250 and / or the region ahead of vehicle 250.

[0069] At some angles, the sensor's operating area 275 may include the rear wheels 276A, 276B of the trailer 270. Thus, the sensor may measure the rear wheels 276A and / or the rear wheels 276B during operation. For example, the rear wheels 276A, 276B may reflect lidar or radar signals transmitted by the sensor. The sensor may receive signals reflected from the rear wheels 276A, 276B. Thus, the data collected by the sensor may include data from reflections from the wheels.

[0070] In some cases, such as when the sensor is a radar, reflections from the rear wheels 276A, 276B may appear as noise in the received radar signal. As a result, the radar may operate with an enhanced signal-to-noise ratio in cases where the rear wheels 276A, 276B direct the radar signal away from the sensor.

[0071] 3 is a conceptual illustration of wireless communication between various computing systems associated with an autonomous or semi-autonomous vehicle, according to an example embodiment. In particular, wireless communication may occur between a remote computing system 302 and the vehicle 200 over a network 304. Wireless communication may also occur between a server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0072] Vehicle 200 may correspond to various types of vehicles capable of transporting passengers or objects between locations and may take the form of any one or more of the vehicles discussed above. In some cases, vehicle 200 may operate in an autonomous or semi-autonomous mode that enables a control system to safely navigate vehicle 200 between destinations using sensor measurements. When operating in an autonomous or semi-autonomous mode, vehicle 200 may navigate with or without a passenger. As a result, vehicle 200 may pick up and drop off passengers between desired destinations.

[0073] Remote computing system 302 may represent any type of device associated with remote assistance technologies, including but not limited to those described herein. In examples, remote computing system 302 may represent any type of device configured to (i) receive information related to vehicle 200, (ii) provide an interface through which a human operator can then observe the information and enter a response related to the information, and (iii) transmit the response to vehicle 200 or to another device. Remote computing system 302 may take a variety of forms, such as a workstation, a desktop computer, a laptop, a tablet, a mobile phone (e.g., a smartphone), and / or a server. In some examples, remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0074] The remote computing system 302 may include one or more subsystems and components similar to or identical to those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface including input / output devices such as a touch screen and a speaker. Other examples are possible as well.

[0075] The network 304 represents an infrastructure that enables wireless communication between the remote computing system 302 and the vehicle 200. The network 304 also enables wireless communication between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0076] The location of the remote computing system 302 can vary within the examples. For example, the remote computing system 302 can be at a location remote from the vehicle 200 having wireless communication over the network 304. In another example, the remote computing system 302 can correspond to a computing device within the vehicle 200 that is separate from the vehicle 200 but that allows a human operator to interact with a passenger or driver of the vehicle 200. In some examples, the remote computing system 302 can be a computing device with a touch screen that can be operated by a passenger of the vehicle 200.

[0077] In some embodiments, operations described herein performed by remote computing system 302 may additionally or alternatively be performed by vehicle 200 (i.e., by any system or subsystem of vehicle 200). In other words, vehicle 200 may be configured to provide remote assistance mechanisms with which a driver or passengers of the vehicle can interact.

[0078] The server computing system 306 may be configured to wirelessly communicate with the remote computing system 302 and the vehicle 200 over the network 304 (or, in some cases, directly with the remote computing system 302 and / or the vehicle 200). The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information regarding the vehicle 200 and its remote assistance. As such, the server computing system 306 may be configured to perform any operation or portion of such operation described herein as being performed by the remote computing system 302 and / or the vehicle 200. Some embodiments of wireless communication related to remote assistance may utilize the server computing system 306, while other embodiments may not.

[0079] The server computing system 306 may include one or more subsystems and components similar to or identical to the subsystems and components of the remote computing system 302 and / or the vehicle 200, such as a processor configured to perform various operations described herein, as well as a wireless communication interface for receiving information from and providing information to the remote computing system 302 and the vehicle 200.

[0080] The various systems described above may perform various operations, and these operations and associated features will now be described.

[0081] In keeping with the above discussion, computing systems (e.g., remote computing system 302, server computing system 306, and computing systems local to vehicle 200) may operate to capture images of the surrounding environment of the autonomous or semi-autonomous vehicle using cameras. Typically, at least one of the computing systems may analyze the images and possibly control the autonomous or semi-autonomous vehicle.

[0082] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data representative of objects in the environment surrounding the vehicle (also referred to herein as "environmental data") in various manners. The vehicle's sensor system may provide the environmental data representative of objects in the surrounding environment. For example, the vehicle may have a variety of sensors including cameras, radar, lidar, microphones, radio units, and other sensors. Each of these sensors may communicate environmental data to a processor within the vehicle regarding the information each respective sensor receives.

[0083] In one example, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have two or more cameras positioned at different orientations. Also, in some embodiments, the camera may be capable of moving to capture images and / or video in different directions. The camera may be configured to store the captured images and video in a memory for later processing by a processing system of the vehicle. The captured images and / or video may be environmental data. Additionally, the camera may include an image sensor as described herein.

[0084] In another example, a radar may be configured to transmit electromagnetic signals that are reflected by various objects near the vehicle and then capture the electromagnetic signals that reflect from the objects. The captured reflected electromagnetic signals may enable the radar (or a processing system) to make various determinations about the objects that reflected the electromagnetic signals. For example, distances and locations to various reflecting objects may be determined. In some embodiments, a vehicle may have two or more radars at different orientations. The radar may be configured to store the captured information in a memory for later processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0085] In another example, a LIDAR may be configured to transmit an electromagnetic signal (e.g., infrared light such as from a gas or diode laser, or other possible light source) that is reflected by a target object near the vehicle. The LIDAR may be capable of acquiring the reflected electromagnetic (e.g., infrared light) signal. The captured reflected electromagnetic signal may enable a ranging system (or processing system) to determine distances to various objects. The LIDAR may also determine the velocity or speed of the target object, which may be stored as environmental data.

[0086] Additionally, in one example, a microphone may be configured to capture audio of the vehicle's surrounding environment. Sounds captured by the microphone may include emergency vehicle sirens and other vehicle sounds. For example, the microphone may capture the sounds of ambulance, fire engine, and police sirens. The processing system may be capable of identifying that the captured audio signal is indicative of an emergency vehicle. In another example, the microphone may capture the sounds of an exhaust from another vehicle, such as an exhaust from a motorcycle. The processing system may be capable of identifying that the captured audio signal is indicative of a motorcycle. Data captured by the microphone may form a portion of the environmental data.

[0087] In yet another example, the radio unit may be configured to transmit an electromagnetic signal that may take the form of a Bluetooth signal, an 802.11 signal, and / or other wireless technology signal. The first electromagnetic radiation signal may be transmitted via one or more antennas located on the radio unit. Furthermore, the first electromagnetic radiation signal may be transmitted in one of many different wireless signal modes. However, in some embodiments, it may be desirable to transmit the first electromagnetic radiation signal in a signal mode that requests a response from devices located near the autonomous or semi-autonomous vehicle. The processing system may be capable of detecting nearby devices based on responses transmitted back to the radio unit and using this communicated information as part of the environmental data.

[0088] In some embodiments, the processing system may be able to combine information from various sensors to further determine the vehicle's surroundings. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, other combinations of sensor data may be used by the processing system to make decisions about the surrounding environment.

[0089] While operating in an autonomous mode (or semi-autonomous mode), the vehicle may control its operation with little or no human input. For example, if a human operator inputs an address into the vehicle, the vehicle may be able to drive to the specified destination without further input from the human (e.g., without the human having to steer or touch the brake / accelerator pedals). Additionally, while the vehicle is operating autonomously or semi-autonomously, the sensor system may receive environmental data. The vehicle's processing system may alter control of the vehicle based on the environmental data received from the various sensors. In some embodiments, the vehicle may alter the speed of the vehicle in response to the environmental data from the various sensors. The vehicle may alter its speed to avoid obstacles, obey traffic laws, etc. When the processing system at the vehicle identifies an object near the vehicle, the vehicle may be able to alter its speed or otherwise change its movement.

[0090] If the vehicle detects an object but is not fully confident in the detection of the object, the vehicle can request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object is actually present in the surrounding environment (e.g., is there actually a stop sign or is there actually no stop sign), (ii) verifying whether the vehicle's identification of the object is correct, (iii) correcting the identification if it is incorrect, and / or (iv) providing supplemental instructions (or modifying current instructions) to the autonomous or semi-autonomous vehicle. Remote assistance tasks can also include the human operator providing instructions to control the vehicle's behavior (e.g., if the human operator determines that the object is a stop sign, commanding the vehicle to stop at the stop sign), although in some scenarios the vehicle itself may control its own behavior based on the human operator's feedback related to the object's identification.

[0091] To facilitate this, the vehicle may analyze environmental data representative of objects in the surrounding environment to determine at least one object having a detection confidence below a threshold. A processor of the vehicle may be configured to detect various objects in the surrounding environment based on the environmental data from the various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, bicyclists, street signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0092] The detection confidence may indicate the likelihood that a determined object is correctly identified or present in the surrounding environment. For example, the processor may perform object detection of an object in image data in the received environmental data and determine that an object has a detection confidence below a threshold based on at least one object not being able to be identified with a detection confidence above a threshold. If the object detection or object recognition results for an object are inconclusive, the detection confidence may be low or below a set threshold.

[0093] The vehicle may detect objects in the surrounding environment in a variety of ways depending on the source of the environmental data. In some embodiments, the environmental data may be image or video data coming from a camera. In other embodiments, the environmental data may come from a lidar. The vehicle may analyze the captured image or video data to identify objects in the image or video data. The methods and apparatus may be configured to monitor the image and / or video data for objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in the surrounding environment based on radar, audio, or other data.

[0094] In some embodiments, the technique used by the vehicle to detect the object may be based on a set of known data. For example, data related to environmental objects may be stored in a memory located in the vehicle. The vehicle may compare the received data to the stored data to determine the object. In other embodiments, the vehicle may be configured to determine the object based on the context of the data. For example, construction-related street signs may generally have an orange color. Thus, the vehicle may be configured to detect an orange object located near the side of the road as a construction-related street sign. Additionally, as the vehicle's processing system detects objects in the captured data, it may also calculate a confidence level for each object.

[0095] Additionally, the vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object detected. For example, the confidence threshold may be lower for objects that may require a quick responsive action from the vehicle, such as the brake lights of another vehicle. However, in other embodiments, the confidence threshold may be the same for all detected objects. If the confidence associated with a detected object is higher than the confidence threshold, the vehicle may assume that the object was correctly recognized and responsively adjust the vehicle's control based on that assumption.

[0096] If the confidence associated with the detected object is below a confidence threshold, the action taken by the vehicle may vary. In some embodiments, the vehicle may react as if the detected object is present despite the low confidence level. In other embodiments, the vehicle may react as if the detected object is not present.

[0097] Upon detecting an object in the surrounding environment, the vehicle may also calculate a confidence level associated with a particular detected object. The confidence level may be calculated in a variety of ways depending on the embodiment. In one example, upon detecting an object in the surrounding environment, the vehicle may compare the environmental data to predefined data associated with known objects. The closer the match between the environmental data and the predefined data, the higher the confidence level. In other embodiments, the vehicle may use a mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0098] In response to determining that the object has a detection confidence below the threshold, the vehicle may transmit a request for remote assistance along with an identification of the object to a remote computing system. As discussed above, the remote computing system may take a variety of forms. For example, the remote computing system may be an in-vehicle computing device separate from the vehicle, but with which a human operator may interact with a passenger or driver of the vehicle, a touch screen interface for displaying remote assistance information, and the like. Additionally or alternatively, as another example, the remote computing system may be a remote computer terminal or other device located at a location not near the vehicle.

[0099] A request for remote assistance may include environmental data, including objects, image data, audio data, etc. The vehicle may transmit the environmental data over a network (e.g., network 304) to a remote computing system, in some embodiments, via a server (e.g., server computing system 306). A human operator of the remote computing system may then use the environmental data as a basis for responding to the request.

[0100] In some embodiments, if an object is detected as having a confidence below a confidence threshold, the object may be given a preliminary identification and the vehicle may be configured to adjust the operation of the vehicle in response to the preliminary identification. Such adjustments in operation may take the form of stopping the vehicle, switching the vehicle to a human-controlled mode, changing the vehicle's velocity (e.g., speed and / or direction), among other possible adjustments.

[0101] In other embodiments, if the vehicle detects an object with a confidence level that meets or exceeds a threshold, the vehicle may still act on the detected object (e.g., stopping if the object is identified with high confidence as a stop sign), but may be configured to request remote assistance at the same time (or after) the vehicle acts on the detected object.

[0102] 4, a particular illustrative example of a system 400 operable to reduce auto-exposure latency is shown. The system 400 includes a processor 402, a memory 404, and an image sensor 406. According to one implementation, the processor 402, the memory 404, and the image sensor 406 may be integrated into a circuit or may be integrated into a single chip, such that the processor 402 provides "on-chip" processing functionality for the image sensor 406. The processor 402 may be coupled to the memory 404 to enable exchange of data and commands between the processor 402 and the memory 404, the memory 404 may be coupled to the image sensor 406 to enable exchange of data between the memory 404 and the image sensor 406, and the image sensor 406 may be coupled to the processor 402 to enable exchange of data and commands between the image sensor 406 and the processor 402.

[0103] According to one implementation, the system 400 may be integrated into one or more sensor systems. As a non-limiting example, the system 400 may be integrated into one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, 218 of the vehicle 200. As another non-limiting example, the system 400 may be integrated into the sensor system 104 of the vehicle 100. In a particular implementation, the system 400 may be integrated into the camera 130. The above embodiments are not intended to be limiting, and it should be understood that the system 400 may be integrated into other sensor systems and cameras.

[0104] The processor 402 includes an exposure operation unit 410, a pre-exposure operation unit 412, a frame analysis unit 414, and an exposure parameter determination unit 416. According to one implementation, one or more of the units 410, 412, 414, 416 may be implemented using dedicated hardware, such as one or more ASICs or one or more FPGAs. According to one implementation, one or more of the units 410, 412, 414, 416 may be implemented using instructions 490 stored in the memory 404 that are executed by the processor 402. For example, the processor 402 may execute the instructions 490 stored in the memory 404 (e.g., a non-transitory computer-readable medium) to perform the operations described herein.

[0105] It should be noted that in some implementations, operations or functions associated with one or more of units 410, 412, 414, 416 may be integrated into a single unit. As a non-limiting example, the functions of the exposure operation unit 410 and the pre-exposure operation unit 412 may be integrated into a single unit. Also, it should be noted that in some implementations, one or more of the units 410, 412, 414, 416 may not be present in the processor 402. As a non-limiting example, in some implementations, such as the implementation described with respect to FIG. 5, the pre-exposure operation unit 412 may not be present in the processor 402 if no pre-exposure frames are generated.

[0106] The exposure operation unit 410 may be configured to initiate frame exposure operations that enable the image sensor 406 to capture different frames 450. According to one implementation, the exposure operation unit 410 may send a trigger signal 415 to the image sensor 406 to enable the image sensor 406 to capture the frame 450. For example, during a frame period for capturing frame 450A, the exposure operation unit 410 may send a trigger signal 415 to the image sensor 406 to enable the image sensor 406 to capture frame 450A. Similarly, during a frame period for capturing other frames 450B, 450C, the exposure operation unit 410 may send a trigger signal 415 to the image sensor 406 to enable the image sensor 406 to capture frames 450B, 450C, respectively. In some implementations, the trigger signal 415 may be generated from a host external to the system 400 and sent to the image sensor 406.

[0107] The frame analysis unit 414 may be configured to determine frame characteristics 460 (e.g., image characteristics) of the frame 450 captured by the image sensor 406. For example, the frame analysis unit 414 may determine frame characteristics 460A of the frame 450A in response to the image sensor 406 capturing frame 450A, the frame analysis unit 414 may determine frame characteristics 460B of the frame 450B in response to the image sensor 406 capturing frame 450B, and the frame analysis unit 414 may determine frame characteristics 460C of the frame 450C in response to the image sensor 406 capturing frame 450C. According to one implementation, the frame characteristics 460 may indicate lighting characteristics associated with the frame 450. However, in other implementations, the frame characteristics 460 may indicate other characteristics, such as noise level, contrast, etc.

[0108] In some implementations, the pre-exposure operation unit 412 may be configured to initiate a frame exposure operation that enables the image sensor 406 to capture a pre-exposure frame 452. For example, the pre-exposure operation unit 412 may generate and transmit a trigger signal 413 that enables the image sensor 406 to perform a pre-exposure operation to capture the pre-exposure frame 452. The pre-exposure frame 452 is a "partial" frame that is captured at the beginning of a frame period and can be analyzed to determine one or more exposure parameters 430, 434 for a frame (e.g., a "full" frame) captured later in the frame period. The pre-exposure frame 452 may be generated in response to sampling a subset of image pixels on the image sensor 406. According to one implementation, the subset of image pixels is associated with a particular region of interest on the image sensor 406. As a non-limiting example, the particular region of interest on the image sensor 406 may correspond to a rectangular region proximate a central section on the image sensor 406. As another non-limiting example, a particular region of interest on the image sensor 406 may correspond to one or more rows (e.g., pixel rows), one or more columns (e.g., pixel columns), or both. According to another implementation, a subset of image pixels is distributed across the image sensor 406.

[0109] In a similar manner as frame analysis unit 414 determines frame characteristics 460 of frame 450, frame analysis unit 414 may be configured to determine pre-exposure frame characteristics 462 (e.g., image characteristics) of pre-exposure frame 452 captured by image sensor 406. According to one implementation, pre-exposure frame characteristics 462 may be indicative of lighting characteristics associated with pre-exposure frame 452. However, in other implementations, pre-exposure frame characteristics 462 may be indicative of other characteristics, such as noise level, contrast, etc.

[0110] The exposure parameter determination unit 416 may be configured to determine target settings 432, 436 for each of the different exposure parameters 430, 434. The exposure parameters 430, 434 may correspond to an exposure time (e.g., shutter speed) of the image sensor 406, an aperture size of the image sensor 406, an analog gain, a digital gain, etc. For ease of explanation, unless otherwise stated, the exposure parameter 430 corresponds to an exposure time. However, it should be understood that in other implementations, one or more of the exposure parameters 430, 434 may correspond to different parameters (e.g., aperture size, analog gain, and / or digital gain). It should also be noted that in other implementations, the exposure parameter determination unit 416 may be configured to determine target settings for three or more exposure parameters, or may be configured to determine a target setting for a single exposure parameter.

[0111] The exposure parameter determination unit 416 can determine settings 432, 436 of the exposure parameters 430, 434, respectively, for a frame captured by the image sensor 406 based at least in part on the frame characteristics 460, 462. For example, according to one implementation, as described in more detail with respect to FIG. 5, the exposure parameter determination unit 416 can analyze frame characteristics 460A of a previous frame (e.g., frame 450A) and determine, based on the analysis, how to adjust settings 432, 436 of the exposure parameters 430, 434, respectively, of subsequent frames 450B, 450C captured by the image sensor 406. According to another implementation, as described in more detail with respect to FIG. 6, the exposure parameter determination unit 416 can analyze pre-exposure frame characteristics 462 of a pre-exposure frame 452 and determine, based on the analysis, how to adjust settings 432, 436 of the exposure parameters 430, 434, respectively, of a captured frame 450B. According to yet another implementation, as described in more detail with respect to FIG. 7 , the exposure parameter determination unit 416 can analyze data from an external source (e.g., data collected from another sensor) and, based on the analysis, determine how to adjust the settings 432, 436 of the exposure parameters 430, 434, respectively, of the frame 450 being captured.

[0112] As described above, image sensor 406 may be configured to capture images of the surrounding environment. By way of non-limiting example, image sensor 406 may be configured to capture images (e.g., frames) of the surrounding environment of vehicle 100, vehicle 200, or both. Although generally described herein as an active pixel sensor (e.g., a complementary metal oxide semiconductor (CMOS) sensor), in some implementations image sensor 406 may be a charge-coupled device (CCD).

[0113] In some scenarios, when characteristics of the surrounding environment change (e.g., when the vehicle 100 enters or exits a dark tunnel), the exposure parameters 430, 434 associated with the image sensor 406 may need to be adjusted to capture a high quality image. For example, when the vehicle 100 enters a tunnel and the surrounding environment becomes relatively dark, the exposure time used to capture the image with the image sensor 406 may be increased to increase the illumination characteristics of the image (e.g., brighten the image). However, when the vehicle 100 exits the tunnel and the surrounding environment becomes relatively bright, the exposure time used to capture the image with the image sensor 406 may be decreased to decrease the illumination characteristics of the image (e.g., darken the image). As described below with respect to FIGS. 5-7, the on-chip processor 402 may be configured to reduce the latency associated with adjusting the exposure parameters 430, 434 of the image sensor 406 such that the exposure parameters 430, 434 may be adjusted and a corresponding frame having the adjusted exposure parameters generated within a single frame period.

[0114] 5, a particular illustrative example of a process 500 for reducing auto-exposure latency is shown. Process 500 may be performed by various components of system 400 of FIG. 4. In particular, one or more operations of process 500 may be performed by processor 402 of system 400.

[0115] The process 500 represents three frame periods 550 associated with the capture of different frames 450. For example, the process 500 represents a frame period 550A associated with the capture of frame 450A, a frame period 550B associated with the capture of frame 450B, and a frame period 550C associated with the capture of frame 450C. During each frame period 550, a trigger signal 415 is fired to initiate the capture of the corresponding frame 450. For example, when the trigger signal 415 is fired during the frame period 550A, a frame exposure operation for capturing frame 450A is initiated. During the frame exposure operation for capturing frame 450A, a shutter release 580A occurs to capture frame 450A, and after an exposure time 584A has elapsed, frame 450A is read 582A. Similarly, when the trigger signal 415 is fired during the frame period 550B, a frame exposure operation for capturing frame 450B is initiated. During a frame exposure operation to capture frame 450B, a shutter release 580B occurs to capture frame 450B and after an exposure time 584B has elapsed, frame 450B is read 582B.

[0116] According to the process 500, the processor 402 may be configured to determine a setting 432 of the exposure parameters 430 (e.g., exposure time 584B) for a frame 450B captured by the image sensor 406. For example, during an exposure parameter determination period 570, the processor 402 may determine an exposure time 584B for the frame 450B to be captured. The exposure parameter determination period 570 is a subset of the frame period 550B for capturing the frame 450B. Thus, the processor 402 may determine a setting 432 of the exposure parameters 430 for the frame 450B (e.g., may set the exposure time 584B) during the same frame period 550B in which the frame 450B is generated.

[0117] The processor 402 can determine settings 432 of exposure parameters 430 for frame 450B based at least in part on frame characteristics 460A of frame 450A (e.g., the previous frame). As a non-limiting example, if the frame characteristics 460 of the previous frame 450A indicate that the illumination metric of the previous frame 450A did not meet (e.g., did not exceed) a lower illumination threshold (e.g., the previous frame 450A was too dark), then during the frame period 550B of the current frame 450B (e.g., during the exposure parameter determination period 570), the processor 402 can determine that the exposure time 584B of the current frame 450B should be extended (relative to the exposure time 584A of the previous frame 450A) and perform the extended exposure time 584B on the image sensor 406. Conversely, if the frame characteristics 460 of the previous frame 450A indicate that the illumination metric of the previous frame 450A did not meet (e.g., did not fall below) the upper illumination threshold (e.g., the previous frame 450A was too bright), then during the frame period 550B of the current frame 450B (e.g., during the exposure parameter determination period 570), the processor 402 can determine that the exposure time 584B of the current frame 450B should be shortened (compared to the exposure time 584A of the previous frame 450A) and perform the shortened exposure time 584B on the image sensor 406.

[0118] While the above example describes adjusting exposure time 584A based on frame characteristics 460A of frame 450A (the previous frame), it should be understood that processor 402 can adjust settings of other exposure parameters during exposure parameter determination period 570 and apply the adjusted settings to frame 450B. As a non-limiting example, based on frame characteristics 460A of frame 450A, processor 402 can adjust analog gain settings, digital gain settings, etc. Additionally, as described in more detail with respect to FIG. 7, the exposure parameters can be further adjusted based on external inputs (e.g., data from other sensors).

[0119] Thus, the techniques described in connection with Figures 4 and 5 enable the processor 402 to reduce latency associated with adjusting the exposure time 584B. For example, instead of having a delay (single frame delay or multi-frame delay) between when the target exposure time 584B (or other target exposure parameter) is determined and when the frame 450B having the target exposure time 584B is read from the image sensor 406, the techniques described in connection with Figures 4 and 5 can reduce (or eliminate) the delay such that the frame 450B having the target exposure time 584B is read during the same frame period 550B that the target exposure time 584B is determined. By reducing latency as described above, the techniques described in Figures 4 and 5 can reduce the number of overexposed frames that are captured and can reduce the number of underexposed frames that are captured.

[0120] 6, a particular illustrative example of a process 600 for reducing auto-exposure latency is shown. Process 600 may be performed by various components of system 400 of FIG. 4. In particular, one or more operations of process 600 may be performed by processor 402 of system 400.

[0121] The process 600 represents two frame periods 650 associated with the capture of different frames 450. For example, the process 600 represents a frame period 650A associated with the capture of frame 450A, and a frame period 650B associated with the capture of frame 450B. During each frame period 650, a trigger signal 415 is released to initiate the capture of the corresponding frame 450. For example, when the trigger signal 415 is released during the frame period 650A, a frame exposure operation for capturing frame 450A is initiated. During the frame exposure operation for capturing frame 450A, a shutter release 680A occurs to capture frame 450A, and after an exposure time 684A has elapsed, frame 450A is read from a read step 682A. Similarly, when the trigger signal 415 is released during the frame period 650B, a frame exposure operation for capturing frame 450B is initiated. During a frame exposure operation to capture frame 450B, a shutter release 680B occurs to capture frame 450B and after an exposure time 684B has elapsed, frame 450B is read 682B.

[0122] Prior to capturing frame 450B, a pre-exposure frame 452 is captured during frame period 650B. For example, at the beginning of frame period 650B, a trigger signal 413 is emitted to enable capture of pre-exposure frame 452. The pre-exposure frame 452 may be generated in response to sampling a subset of image pixels on the image sensor 406. The subset of image pixels may be associated with a particular region of interest (e.g., an area proximate a central section of the image sensor, a particular row, and / or a particular column) or may be distributed across the image sensor 406.

[0123] In response to capturing the pre-exposure frame 452 at the beginning of frame period 650B, during an exposure parameter determination period 670 that falls within (e.g., is a subset of) frame period 650B, the processor 402 may determine an exposure time 684B for the captured frame 450B. Thus, the processor 402 may determine the setting 432 of the exposure parameters 430 for frame 450B (e.g., set exposure time 684B) during the same frame period 650B in which frame 450B is generated.

[0124] According to process 600, the processor 402 may be configured to determine an exposure time 684B for the captured frame 450B based at least in part on the pre-exposure frame characteristics 462 of the pre-exposure frame 452. As a non-limiting example, if the pre-exposure frame characteristics 462 of the pre-exposure frame 452 indicate that the illumination metric of the pre-exposure frame 452 did not meet a lower illumination threshold (e.g., the pre-exposure frame 452 was too dark), then during the exposure parameter determination period 670, the processor 402 may determine that the exposure time 684B of the current frame 450B should be extended and execute the extended exposure time 684B at the image sensor 406. Conversely, if the pre-exposure frame characteristics 462 of the pre-exposure frame 452 indicate that the illumination metric of the pre-exposure frame 452 did not meet the upper illumination threshold (e.g., the pre-exposure frame 452 was too bright), then during the exposure parameter determination period 670, the processor 402 may determine that the exposure time 684B of the current frame 450B should be shortened, and the shortened exposure time 684B may be executed on the image sensor 406.

[0125] While the above example describes adjusting exposure time 684A based on pre-exposure frame characteristics 462 of pre-exposure frame 452, it should be understood that processor 402 can adjust settings of other exposure parameters during exposure parameter determination period 670 and apply the adjusted settings to frame 450B. As a non-limiting example, based on pre-exposure frame characteristics 462 of pre-exposure frame 452, processor 402 can adjust analog gain settings, digital gain settings, etc. Additionally, as described in more detail with respect to FIG. 7, the exposure parameters can be further adjusted based on external inputs (e.g., data from other sensors).

[0126] Thus, the techniques described in connection with Figures 4 and 6 enable the processor 402 to reduce latency associated with adjusting the exposure time 684B. For example, instead of having a delay (single frame delay or multi-frame delay) between when the target exposure time 684B (or other target exposure parameter) is determined and when the frame 450B having the target exposure time 684B is read from the image sensor 406, the techniques described in connection with Figures 4 and 6 can reduce (or eliminate) the delay such that the frame 450B having the target exposure time 684B is read during the same frame period 650B that the target exposure time 684B is determined. By reducing the above-mentioned latency, the techniques described in Figures 4 and 6 can reduce the number of overexposed frames that are captured and can reduce the number of underexposed frames that are captured.

[0127] 7, a particular illustrative example of a process 700 for reducing auto-exposure latency is shown. Process 700 may be performed by various components of system 400 of FIG. 4. In particular, one or more operations of process 700 may be performed by processor 402 of system 400.

[0128] The process 700 shows two frame periods 750A and 750B associated with the capture of different frames 450. For example, the process 700 shows a frame period 750A associated with the capture of frame 450A, and a frame period 750B associated with the capture of frame 450B. During each frame period, a trigger signal 415 is fired to initiate the capture of the corresponding frame 450. For example, when the trigger signal 415 is fired during the frame period 750A, a frame exposure operation for capturing frame 450A is initiated. During the frame exposure operation for capturing frame 450A, a shutter release 780A occurs to capture frame 450A, and after an exposure time 784A has elapsed, frame 450A is read in a read step 782A. Similarly, when the trigger signal 415 is fired during the frame period 750B, a frame exposure operation for capturing frame 450B is initiated. During a frame exposure operation to capture frame 450B, a shutter release 780B occurs to capture frame 450B, and after an exposure time 784B has elapsed, frame 450B is read in a read step 782B.

[0129] Prior to capturing frame 450B, a pre-exposure frame 452 is captured during frame period 750B. For example, in a manner similar to that described with respect to FIG. 6, at the beginning of frame period 750B, a trigger signal 413 is emitted to enable capture of pre-exposure frame 452. In addition, an external input 790 may also be received by processor 402 during frame period 750B. External input 790 may include data captured from one or more other sensors (e.g., other cameras, one or more lidars, one or more radars, and / or one or more accelerometers). As a non-limiting example, external input 790 may include data indicative of environmental conditions based on image frames captured by one or more other sensors. According to some implementations, the image frame on which external input 790 is based may be captured immediately prior to image frame 450B being captured during frame period 750B.

[0130] During an exposure parameter determination period 770 that falls within (e.g., is a subset of) the frame period 750B, the processor 402 may determine settings 432, 436 of the exposure parameters 430, 434 for the captured frame 450B based on an external input 790, the frame characteristics 460A of the previous frame 450A, the pre-exposure frame characteristics 462 of the pre-exposure frame 452, or a combination thereof. For example, based on at least one of the external input 790, the frame characteristics 460A of the previous frame 450A, or the pre-exposure frame characteristics 462 of the pre-exposure frame 452, the processor 402 may determine an exposure time 784B for frame 450B, an analog gain for frame 450B, a digital gain for frame 450B, etc., to adjust the exposure parameters 430, 434 for frame 450B.

[0131] Thus, the techniques described in connection with Figures 4 and 7 enable the processor 402 to reduce latency associated with adjusting the exposure time 784B. For example, instead of having a delay (single frame delay or multiple frame delay) between when the target exposure time 784B is determined and when the frame 450B having the target exposure time 684B is read from the image sensor 406, the techniques described in connection with Figures 4 and 7 can reduce (or eliminate) the delay such that the frame 450B having the target exposure time 784B is read during the same frame period 750B that the target exposure time 784B is determined. By reducing the above-mentioned latency, the techniques described in Figures 4 and 7 can reduce the number of overexposed frames that are captured and can reduce the number of underexposed frames that are captured.

[0132] 8, a particular illustrative example of a method 800 for reducing auto-exposure latency is shown. Method 800 may be performed by various components of system 400 of FIG. 4. In particular, one or more operations associated with method 800 may be performed by processor 402 of system 400.

[0133] The method 800 includes, at block 802, determining, by a processor, a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of a previous frame captured by the image sensor, the first setting of exposure parameters being determined during a first frame period associated with the capture of the first frame. For example, with reference to FIGS. 4 and 5, the processor 402 can determine a setting 432 of exposure parameters 430 for a frame 450B captured by the image sensor 406. The setting 432 of exposure parameters 430 can be determined based at least in part on frame characteristics 460A of a frame 450A captured by the image sensor 406, the setting 432 being determined during a frame period 550B associated with the capture of the frame 450B. According to one implementation of the method 800, the previous frame 450A is captured during a frame period 550A that precedes and is adjacent to the first frame period 550B.

[0134] The method 800 also includes initiating, by the processor, a first frame exposure operation during the first frame period, at block 804. The first frame exposure operation is based on a first setting of the exposure parameters. For example, with reference to FIGS. 4 and 5, the processor 402 may initiate the frame exposure operation during frame period 550B. The frame exposure operation may be based on setting 432 of the exposure parameters 430.

[0135] According to one implementation of method 800, the exposure parameter corresponds to an exposure time 584 B. According to another implementation of method 800, the exposure parameter corresponds to an analog gain or a digital gain.

[0136] According to one implementation, the method 800 also includes determining a first setting of the exposure parameters based on an external input. For example, with reference to FIGS. 4 and 7, the processor 402 may determine a setting 432 of the exposure parameters 430 for the frame 450B based on the frame characteristics 460A of the frame 450A as well as the external input 790. The external input 790 may be based on external sensor data from one or more other sensors. For example, according to some implementations, the external sensor data corresponds to image sensor data or environmental sensor data.

[0137] According to one implementation of method 800, the first frame exposure operation is initiated in response to receiving a trigger signal, such as trigger signal 415. Method 800 may also include scheduling transmission of trigger signal 415 during first frame period 550B and transmitting the trigger signal 415 to image sensor 406 during first frame period 550B in accordance with the scheduled transmission of trigger signal 415.

[0138] According to one implementation, the method 800 also includes determining a second setting of the exposure parameters for a second frame captured by the image sensor. For example, with reference to FIG. 4 and FIG. 5, the processor 402 can determine another setting of the exposure parameters 430 for the frame 450C captured by the image sensor 406. The second setting of the exposure parameters 430 can be determined based at least in part on the characteristics 460B of the first frame 450B. The second setting can be determined during a second frame period 550C associated with the capture of the second frame 450C, which can occur subsequent to and adjacent to the first frame period 550B. The method 800 can also include performing a second frame exposure operation during the second frame period 550C. The second frame exposure operation is based on the second setting of the exposure parameters 430. According to one implementation of the method 800, the second setting of the exposure parameters 430 is further determined based on the characteristics 460A of the previous frame 450A.

[0139] 8 allows the processor 402 to reduce latency associated with adjusting the exposure time 584B. For example, instead of having a delay between when the target exposure time 584B (or other target exposure parameter) is determined and when a frame 450B having a target exposure time 584B is read from the image sensor 406, the method 800 can reduce (or eliminate) the delay such that the frame 450B having the target exposure time 584B is read during the same frame period 550B that the target exposure time 584B is determined. By reducing latency as described above, the method 800 can reduce the number of overexposed frames captured and can reduce the number of underexposed frames captured, which in turn can improve storage capacity.

[0140] 9, a particular illustrative example of a method 900 for reducing auto-exposure latency is shown. Method 900 may be performed by various components of system 400 of FIG. 4. In particular, one or more operations associated with method 900 may be performed by processor 402 of system 400.

[0141] The method 900 includes receiving, by a processor, a first pre-exposure frame from the image sensor during a first frame period associated with the capture of the first frame, at block 902. For example, with reference to Figures 4 and 6, the processor 402 may receive a pre-exposure frame 452 from the image sensor 406 during a frame period 650B associated with the capture of frame 450B.

[0142] The method 900 also includes, at block 904, determining, by the processor, a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined based at least in part on characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during a first frame period. For example, with reference to FIGS. 4 and 6, the processor may determine a setting 432 of exposure parameters 430 for a frame 450B captured by the image sensor 406. The setting 432 of exposure parameters 430 is determined based at least in part on pre-exposure frame characteristics 462 of the pre-exposure frame 452, and the setting 432 of exposure parameters 430 is determined during a frame period 650B.

[0143] The method 900 also includes initiating, by the processor, a first frame exposure operation based on the first setting of the exposure parameters, at block 906. During the first frame exposure operation, the image sensor captures a first frame during a first frame period. For example, the processor 402 may initiate the frame exposure operation based on the setting 432 of the exposure parameters 430. During the frame exposure operation, the image sensor 406 captures frame 450B during frame period 650B.

[0144] According to one implementation of method 900, a first pre-exposure frame 452 is captured by the image sensor 406 during the first frame period 650B. The first pre-exposure frame 452 may be generated in response to sampling a subset of image pixels on the image sensor 406. The subset of image pixels may be associated with a particular region of interest on the image sensor 406 or may be distributed across the image sensor 406. The particular region of interest may correspond to a rectangular region proximate a central section of the image sensor 406, one or more rows of pixels, one or more columns of pixels, etc.

[0145] According to one implementation, the method 900 also includes determining a first setting of the exposure parameters based on an external input. For example, with reference to Figures 4 and 7, the processor 402 may determine the setting 432 of the exposure parameters 430 for frame 450B based on the pre-exposure frame characteristics 462 of the pre-exposure frame 452 as well as the external input 790. The external input 790 may be based on external sensor data from one or more other sensors. For example, according to some implementations, the external sensor data corresponds to image sensor data or environmental sensor data.

[0146] 9 allows the processor 402 to reduce latency associated with adjusting the exposure time 684B. For example, instead of having a delay between when the target exposure time 684B (or other target exposure parameter) is determined and when a frame 450B having a target exposure time 684B is read from the image sensor 406, the method 900 can reduce (or eliminate) the delay such that the frame 450B having the target exposure time 684B is read during the same frame period 650B that the target exposure time 684B is determined. By reducing the above-mentioned latency, the method 900 can reduce the number of overexposed frames captured and can reduce the number of underexposed frames captured, which in turn can improve storage capacity.

[0147] The present disclosure is not limited with respect to the specific embodiments described in this application, which are intended as illustrations of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope of the present disclosure. In addition to the methods and devices recited herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0148] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the figures, similar symbols typically refer to similar components identically unless the context dictates otherwise. The exemplary embodiments described herein and in the figures are not intended to be limiting. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated.

[0149] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and discussed herein, each step, block, operation, and / or communication may represent processing of information and / or transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in a different order than shown or discussed, such as substantially simultaneously or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be used in any of the message flow diagrams, scenarios, and flowcharts discussed herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other in part or in whole.

[0150] A step, block, or operation corresponding to the processing of information may correspond to circuitry that can be configured to perform a particular logical function of a method or technique described herein. Alternatively or additionally, a step or block corresponding to the processing of information may correspond to a module, segment, or portion of program code (including associated data). Program code may include one or more instructions executable by a processor to perform a particular logical operation or operation in a method or technique. The program code and / or associated data may be stored in any type of computer-readable medium, such as a storage device including a RAM, a disk drive, a solid-state drive, or another storage medium.

[0151] Additionally, steps, blocks, or operations corresponding to one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device, whereas other information transmissions may be information transmissions between software and / or hardware modules in different physical devices.

[0152] The particular arrangement shown in the figures should not be considered as limiting. It should be understood that other embodiments may include more or less of each element shown in a given figure. Further, some of the illustrated elements may be combined or omitted. Additionally, example embodiments may include elements not illustrated in the figures.

[0153] Various aspects and embodiments are disclosed herein, while other aspects and embodiments will be apparent to those of ordinary skill in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

1. 1. A system comprising: An image sensor; a processor coupled to the image sensor; the processor is configured to determine a first setting of exposure parameters for a first frame captured by the image sensor; the first setting of exposure parameters is determined based at least in part on characteristics of a previous frame captured by the image sensor; the first setting of the exposure parameters is determined during a first frame period associated with capturing the first frame; the image sensor is configured to perform a first frame exposure operation during the first frame period; The first frame exposure operation is based on the first setting of the exposure parameters.

2. The system of claim 1 , wherein the exposure parameter corresponds to an aperture size, an exposure time, an analog gain, or a digital gain.

3. 2. The system of claim 1, wherein the previous frame is captured during a frame period preceding and adjacent to the first frame period.

4. The system of claim 1 , wherein the processor is further configured to determine the first setting of the exposure parameters based on an external input.

5. The system of claim 4 , wherein the external input is based on external sensor data from one or more other sensors.

6. The system of claim 5 , wherein the external sensor data corresponds to image sensor data or environmental sensor data.

7. The system of claim 1 , wherein the first frame exposure operation is initiated in response to receiving a trigger signal.

8. The processor, scheduling transmission of the trigger signal during the first frame period; and The system of claim 7 , further configured to transmit the trigger signal to the image sensor during the first frame period in accordance with a scheduled transmission of the trigger signal.

9. the processor is further configured to determine a second setting of the exposure parameters for a second frame captured by the image sensor; the second set of exposure parameters is determined based at least in part on characteristics of the first frame; the second setting of the exposure parameters is determined during a second frame period associated with capturing the second frame; the second frame period follows and is adjacent to the first frame period; the image sensor is further configured to perform a second frame exposure operation during the second frame period; The system of claim 1 , wherein the second frame exposure operation is based on the second setting of the exposure parameters.

10. The system of claim 9 , wherein the processor is further configured to determine the second setting of the exposure parameters based on the characteristics of the previous frame.

11. 1. A system comprising: An image sensor; a processor coupled to the image sensor, the processor comprising: receiving a first pre-exposure frame from the image sensor during a first frame period associated with capturing a first frame; and configured to determine a first setting of exposure parameters for the first frame captured by the image sensor, the first setting of exposure parameters being determined based at least in part on characteristics of the first pre-exposure frame, the first setting of exposure parameters being determined during the first frame period; the image sensor is configured to perform a first frame exposure operation based on the first setting of the exposure parameters; During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

12. The system of claim 11 , wherein the first pre-exposure frame is captured by the image sensor during the first frame period.

13. The system of claim 11 , wherein the first pre-exposure frame is generated in response to sampling a subset of image pixels on the image sensor.

14. The system of claim 13 , wherein the subset of image pixels is associated with a region of interest on the image sensor.

15. The system of claim 14 , wherein the region of interest corresponds to a rectangular region proximate a central section of the image sensor.

16. The system of claim 14 , wherein the region of interest corresponds to one or more rows, one or more columns, or both.

17. The system of claim 13 , wherein the subset of image pixels is distributed across the image sensor.

18. The system of claim 11 , wherein the exposure parameter corresponds to an aperture size, an exposure time, an analog gain, or a digital gain.

19. The system of claim 11 , wherein the processor is further configured to determine the first setting of the exposure parameters based on an external input.

20. 1. A method comprising: receiving, by the processor, a first pre-exposure frame from the image sensor during a first frame period associated with capturing the first frame; determining, by the processor, a first setting of exposure parameters for the first frame captured by the image sensor, the first setting of exposure parameters being determined based at least in part on characteristics of the first pre-exposure frame, and the first setting of exposure parameters being determined during the first frame period; initiating, by the processor, a first frame exposure operation based on the first setting of the exposure parameters, wherein during the first frame exposure operation, the image sensor captures the first frame during the first frame period.

Images