Reduced automatic exposure latency

Abstract

To reduce auto-exposure latency.SOLUTION: An example embodiment includes a method of reducing auto-exposure latency. The method includes determining, by a processor, a first setting of an exposure parameter for a first frame to be captured by an image sensor. The first setting of the exposure parameter is determined based at least in part on characteristics of a previous frame captured by the image sensor. The first setting of the exposure parameter is determined during a first frame period associated with capturing the first frame. The method also includes initiating, by the processor, a first frame exposure operation based on the first setting of the exposure parameter. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

Description

Background Art

[0003]

[0001] Unless otherwise stated in this specification, the description in this chapter is not prior art for the claims of this application and should not be considered prior art by inclusion in this chapter.

[0002] An autonomous vehicle can include one or more image sensors that capture images of the surrounding environment. Image sensors typically have different exposure settings that are selectively used to improve the quality of the images. As a non-limiting example, when the surrounding environment is relatively dark, the exposure time used to capture an image with the image sensor can be adjusted to have a relatively long duration, which can then increase the illumination characteristics of the image (e.g., brighten the image). However, when the surrounding environment is relatively bright, the exposure time used to capture an image with the image sensor can be adjusted to have a relatively short duration, which can then decrease the illumination characteristics of the image (e.g., darken the image). Typically, there is a delay between when the exposure setting is adjusted and when the image sensor captures an image with the adjusted exposure setting. For example, the image sensor can have a single-frame delay or a multi-frame delay between when the exposure setting is changed and when a frame acquired with the new setting is read. As a result, during the frame delay, the image sensor can capture one or more overexposed frames and / or underexposed frames, which occupy memory space and provide little (or no) benefit to the operation of the autonomous vehicle.

Summary of the Invention

[0003] The techniques described herein reduce latency (e.g., frame delay) associated with automatic exposure operations. In particular, the techniques described herein enable the determination of target exposure parameters, such as exposure time, analog gain, or digital gain, and their execution within a single frame period. For example, according to one embodiment, during a particular frame period, an on-chip processor can analyze previous frames captured by the image sensor to determine target exposure parameters for the “current” frame captured by the image sensor during that particular frame period. As another example, according to one embodiment, during a particular frame period, an on-chip processor can sample a subset of pixels on the image sensor to generate a “previous exposure” frame, analyze the previous exposure frame to determine target exposure parameters. In response to determining the target exposure parameters (e.g., based on previous frames or based on the previous exposure frame), the image sensor can capture the current frame using the target exposure parameters such that there is no frame delay between determining the target exposure parameters and reading the image frame captured using the target exposure parameters.

[0004] The system includes an image sensor and a processor coupled to the image sensor. The processor is configured to determine a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined at least in part based on the characteristics of a previous frame captured by the image sensor. The first setting of exposure parameters is determined during a first frame period associated with the capture of the first frame. The image sensor is configured to perform a first frame exposure operation during the first frame period. The first frame exposure operation is based on the first setting of exposure parameters.

[0005] The method includes the processor determining a first setting of exposure parameters for a first frame to be captured by the image sensor. The first setting of exposure parameters is determined at least in part based on the characteristics of a previous frame captured by the image sensor. The first setting of exposure parameters is determined during a first frame period associated with the capture of the first frame. The method also includes the processor initiating a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures the first frame during the first frame period.

[0006] Non-temporary computer-readable media, when executed by a processor, include instructions that cause the processor to perform an action. The action includes determining a first setting of exposure parameters for a first frame to be captured by an image sensor. The first setting of exposure parameters is determined at least in part on the characteristics of a previous frame captured by the image sensor. The first setting of exposure parameters is determined during a first frame period associated with the capture of the first frame. The action also includes initiating a first frame exposure action based on the first setting of exposure parameters. During the first frame exposure action, the image sensor captures the first frame during the first frame period.

[0007] The system includes an image sensor and a processor coupled to the image sensor. The processor is configured to receive a first pre-exposure frame from the image sensor during a first frame period associated with the capture of a first frame. The processor is also configured to determine a first setting of exposure parameters for the first frame captured by the image sensor. The first setting of exposure parameters is determined at least in part based on the characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. The image sensor is configured to perform a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures a first frame during the first frame period.

[0008] The method includes the processor receiving a first pre-exposure frame from the image sensor during a first frame period associated with the capture of a first frame. The method also includes the processor determining a first setting of exposure parameters for the first frame to be captured by the image sensor. The first setting of exposure parameters is determined at least in part based on the characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. The method further includes the processor initiating a first frame exposure operation based on the first setting of exposure parameters. During the first frame exposure operation, the image sensor captures a first frame during the first frame period.

[0009] A non-temporary computer-readable medium, when executed by a processor, contains instructions that cause the processor to perform an action. The action includes receiving a first pre-exposure frame from the image sensor during a first frame period associated with the capture of a first frame. The action also includes determining a first setting of exposure parameters for the first frame to be captured by the image sensor. The first setting of exposure parameters is determined at least in part on the characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. The action further includes initiating a first frame exposure action based on the first setting of exposure parameters. During the first frame exposure action, the image sensor captures a first frame during the first frame period.

[0010] These and other embodiments, advantages, and alternatives will become apparent to those skilled in the art by reading the following detailed description with appropriate reference to the accompanying drawings. [Brief explanation of the drawing]

[0011] [Figure 1] Figure 1 is a functional block diagram illustrating a vehicle according to an exemplary embodiment. [Figure 2A] Figure 2A is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2B] Figure 2B is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2C] Figure 2C is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2D] Figure 2D is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2E] Figure 2E is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2F] Figure 2F is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2G] Figure 2G is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2H] Figure 2H is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2I] Figure 2I is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2J] Figure 2J is an illustrative diagram of the field of view of various sensors according to an exemplary embodiment. [Figure 2K] Figure 2K is an illustrative diagram of beam steering for a sensor according to an exemplary embodiment. [Figure 3] Figure 3 is a conceptual illustration of wireless communication between various computing systems related to autonomous or semi-autonomous vehicles, according to an exemplary embodiment. [Figure 4] Figure 4 shows a system, according to an exemplary embodiment, that can operate to reduce automatic exposure latency. [Figure 5] Figure 5 shows a process for reducing automatic exposure latency based on the frame characteristics of a previously captured frame, according to an exemplary embodiment. [Figure 6] Figure 6 shows a process for reducing automatic exposure latency based on the frame characteristics of a pre-exposure frame, according to an exemplary embodiment. [Figure 7] Figure 7 shows another process for reducing automatic exposure latency according to an exemplary embodiment. [Figure 8] Figure 8 shows a method for reducing automatic exposure latency according to an exemplary embodiment. [Figure 9] Figure 9 shows another method for reducing automatic exposure latency according to an exemplary embodiment. [Modes for carrying out the invention]

[0012] Exemplary methods and systems are contemplated herein. Any example of an exemplary embodiment or feature described herein should not necessarily be construed as being preferable or advantageous to other embodiments or features. Furthermore, the exemplary embodiments described herein are not intended to be limiting. It will be readily apparent that particular aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein. In addition, specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each of the elements shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Moreover, exemplary embodiments may include elements not illustrated in the figures.

[0013] The techniques described herein reduce latency associated with automatic exposure operations. In particular, the techniques described herein enable the determination and execution (e.g., adjustment) of exposure parameters for a frame, such as exposure time, during the frame duration associated with frame capture. As a result, instead of having a delay (e.g., single-frame or multi-frame delay) between when the target exposure parameters are determined and when the captured frame is read from the image sensor using the target exposure parameters, the techniques described herein can reduce (or eliminate) the delay so that the frame with the target exposure parameters is read during the same frame duration in which the target exposure parameters are determined. By reducing the latency described herein, the techniques described herein can reduce the number of overexposed frames captured and reduce the number of underexposed frames captured.

[0014] As used herein, the "frame period" of the current frame corresponds to the period that starts immediately after the capture of the previous frame and ends immediately after the capture of the current frame. Thus, when the shutter closes and the capture of the previous frame is completed, the frame period of the previous frame ends and the frame period of the current frame starts. Additionally, when the shutter closes and the capture of the current frame is completed, the frame period of the current frame ends. The length of time defining the frame period can be defined as the reciprocal of the frame rate, which is specific to the associated camera system.

[0015] As described above, to reduce latency, some implementations allow the processor to dynamically change the exposure parameters used to capture the current frame based on the characteristics of previously captured frames during the frame duration of the "current frame" (e.g., the frame duration associated with the capture of the current frame). For illustrative purposes, during the frame duration of the current frame, the processor may analyze previously captured frames and determine the exposure parameter settings based on the analysis. As a non-limiting example, if the illumination metric of a previous frame did not meet the lower illumination threshold during the frame duration of the current frame (e.g., the previous frame was too dark), the processor may decide that the exposure time for the current frame should be extended, and the image sensor may perform the extended exposure time. Conversely, if the illumination metric of a previous frame did not meet the upper illumination threshold during the frame duration of the current frame (e.g., the previous frame was too bright), the processor may decide that the exposure time for the current frame should be shortened, and the image sensor may perform the shortened exposure time. Naturally, in addition to analyzing previous frames to determine exposure parameters for the current frame, the processor can also analyze other data such as one or more previously captured additional frames, a histogram of data associated with previously captured frames, and external inputs showing data collected from other sensors. After the exposure parameters are determined, the image sensor can use the exposure parameters to perform a frame exposure operation to capture the current frame during its frame duration.

[0016] According to other implementations, in order to reduce latency as described above, during the frame period of the current frame, the processor can dynamically change the exposure parameters used to capture the current frame based on the characteristics of the "pre-exposure" frame. In this implementation, at the start of the frame period of the current frame, the processor can sample a subset of the image pixels on the image sensor (for a short period of time) to generate a pre-exposure frame. The subset of image pixels can correspond to a specific region of interest or to a distributed region across the image sensor. The pre-exposure frame can be used by the processor to determine how, or whether, to adjust the exposure parameters when capturing the current frame. After determining how the exposure parameters should be adjusted, during the frame period of the current frame, the image sensor can perform a frame exposure operation using the adjusted exposure parameters to capture the current frame.

[0017] The exposure parameters can be adjusted based on the pre-exposure frame, but in some implementations, the exposure parameters can additionally be adjusted based on data from one or more other sensors.As a non-limiting example, at the start of the frame period of the current frame, the processor can also receive data (e.g., an image) from one or more other sensors. In these implementations, the processor can adjust the exposure parameters of the current frame based on the pre-exposure frame and based on data (e.g., an image) from one or more other sensors.

[0018] The following description and the accompanying drawings disclose the features of various exemplary embodiments. The embodiments provided are by way of example and are not intended to be limiting. Therefore, the dimensions in the drawings are not necessarily to scale.

[0019] Specific embodiments are described herein with reference to the drawings. In the description, common features are indicated by common reference numbers throughout the drawings. In some figures, multiple examples of a particular type of feature are used. These features are physically and / or logically different, but the same reference number is used for each, and different examples are distinguished by the addition of a letter to the reference number. When a feature is referred herein as a group or type (for example, when no particular feature is referred), the reference number is used without an identifying letter. However, when a particular feature of one of several features of the same type is referred herein, the reference number is used with an identifying letter. For example, referring to Figure 4, several frames are illustrated and associated with reference numbers 450A, 450B, etc. When referring to a particular one of these frames, such as frame 450A, the identifying letter "A" is used. However, when referring to any one of these frames or these frames as a group, the reference number 450 is used without an identifying letter.

[0020] Herein, exemplary systems within the scope of this disclosure will be described in more detail. Exemplary systems may be implemented in or take the form of automobiles. Additionally, exemplary systems may also be implemented in or take the form of various vehicles, such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor-trailers), motorcycles, buses, airplanes, helicopters, drones, lawnmowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, agricultural machinery or agricultural vehicles, construction machinery or construction vehicles, warehouse equipment or warehouse vehicles, factory equipment or factory vehicles, trams, golf carts, electric trains, trolleys, pedestrian transport vehicles, and robotic devices. Other vehicles are also possible. Furthermore, in some embodiments, exemplary systems may not include vehicles.

[0021] Referring here to the figure, Figure 1 is a functional block diagram illustrating an exemplary vehicle 100 that may be configured to operate fully or partially in autonomous mode. More specifically, the vehicle 100 may operate in autonomous mode without human interaction by receiving control commands from a computing system. As part of its operation in autonomous mode, the vehicle 100 may use sensors to detect and, in some cases, identify objects in its surrounding environment to enable safe navigation. Additionally, the exemplary vehicle 100 may operate in a partially autonomous (i.e., semi-autonomous) mode in which some functions of the vehicle 100 are controlled by a human driver of the vehicle 100, and some functions of the vehicle 100 are controlled by a computing system. For example, the vehicle 100 may also include subsystems that allow the driver to control the vehicle's actions such as steering, acceleration, and braking, while the computing system implements assistance functions such as lane departure warning / lane keeping assist or adaptive cruise control based on other objects (e.g., other vehicles) in the surrounding environment.

[0022] As described herein, in a partially autonomous driving mode, the vehicle assists with one or more driving actions (e.g., lane centering, adaptive cruise control, advanced driver-assistance systems (ADAS), and steering, braking, and / or acceleration for emergency braking), but the human driver is expected to situationally perceive the surroundings of the vehicle 100 and supervise the assisted driving actions. Here, the vehicle 100 may perform all driving tasks in certain situations, but the human driver is expected to be responsible for taking control as needed.

[0023] For the sake of simplification and brevity, various systems and methods are described below in conjunction with autonomous vehicles, but these or similar systems and methods may be used in various driver assistance systems that do not reach the level of fully autonomous driving systems (i.e., partially autonomous driving systems). In the United States, the Society of Automotive Engineers (SAE) defines different levels of automated driving behavior to indicate how much or how little a vehicle controls the driving, but different organizations in the United States or other countries may classify the levels differently. More specifically, the systems and methods of this disclosure may be used in SAE Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver assistance. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems that enable autonomous driving under limited conditions (e.g., highways). Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 automated driving systems that operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such systems, accurate lane estimation is performed automatically without driver input or control (e.g., while the vehicle is moving), resulting in improved reliability of vehicle positioning and navigation, as well as overall safety for autonomous, semi-autonomous, and other driver assistance systems. As described above, in addition to the way the SAE classifies levels of autonomous driving operations, other organizations in the United States or other countries may classify levels of autonomous driving operations differently. The systems and methods disclosed herein may be used in driver assistance systems defined by the levels of autonomous driving operations of these other organizations, but are not limited to those disclosed herein.

[0024] As shown in Figure 1, the vehicle 100 may include various subsystems such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112 (which may also be referred to as a computing system) having data storage 114, and a user interface 116. In other examples, the vehicle 100 may include more or fewer subsystems, each of which may include multiple elements. The subsystems and components of the vehicle 100 may be interconnected in various ways. In addition, the functions of the vehicle 100 described herein may be divided into additional functional or physical components, or combined into fewer functional or physical components within an embodiment. For example, the control system 106 and the computer system 112 may be combined into a single system that operates the vehicle 100 according to various operations.

[0025] The propulsion system 102 may include one or more components that are operable to provide powered motion to the vehicle 100, among other possible components, an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121. For example, the engine / motor 118 may be configured to convert the energy source 119 into mechanical energy, and among other possible options, it may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine. For example, in some embodiments, the propulsion system 102 may include a number of types of engines and / or motors, such as a gasoline engine and an electric motor.

[0026] The energy source 119 represents an energy source that can power one or more systems of the vehicle 100 (e.g., engine / motor 118), all or in part. For example, the energy source 119 can be gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, the energy source 119 may include a combination of a fuel tank, batteries, a condenser, and / or a flywheel.

[0027] The transmission 120 can transmit mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. Thus, the transmission 120 may include, among other possible components, a gearbox, a clutch, a differential, and a drive shaft. The drive shaft may include an axle connected to one or more of the wheels / tires 121.

[0028] The wheels / tires 121 of the vehicle 100 can have various configurations within the exemplary embodiment. For example, the vehicle 100 may exist in the form of a unicycle, bicycle / motorcycle, tricycle, or four-wheeled automobile / truck, among other possible configurations. Thus, the wheels / tires 121 can be attached to the vehicle 100 in various ways and may exist in different materials such as metal and rubber.

[0029] The sensor system 104 may include various types of sensors, among other possible sensors, particularly a Global Positioning System (GPS) 122, an Inertial Measurement Unit (IMU) 124, radar 126, lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125. In some embodiments, the sensor system 104 may also include sensors configured to monitor the internal systems of the vehicle 100 (e.g., an O2 monitor, a fuel gauge, engine oil temperature, and brake wear).

[0030] The GPS 122 may include a transceiver capable of operating to provide information regarding the position of the vehicle 100 relative to the Earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes, and may sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 while the vehicle 100 is stationary or in motion.

[0031] The radar 126 can represent one or more systems configured to sense objects in the surrounding environment of the vehicle 100 using radio signals, including the speed and direction of the objects. Accordingly, the radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, the radar 126 may correspond to a mountable radar configured to obtain measurements of the surrounding environment of the vehicle 100.

[0032] Lidar 128 may include, among other system components, one or more laser sources, a laser scanner, and one or more detectors, and may operate in coherent mode (e.g., using heterodyne detection, etc.) or incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of Lidar 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes). In some examples, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Furthermore, such photodetectors may be arranged in an array (e.g., like silicon photomultiplier tubes (SiPMs)) (e.g., through series electrical connections). In some examples, one or more photodetectors are devices operating in Geiger mode, and Lidar includes subcomponents designed for such Geiger mode operation.

[0033] The camera 130 may include one or more devices (e.g., still cameras, video cameras, thermal imaging cameras, stereo cameras, and night vision cameras) configured to capture images of the surrounding environment of the vehicle 100.

[0034] The steering sensor 123 may sense the steering angle of the vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, the steering sensor 123 may measure the angles of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the front axle of the vehicle 100. The steering sensor 123 may also be configured to measure a combination (or subset) of the steering wheel angle, an electrical signal representing the angle of the steering wheel, and the angles of the wheels of the vehicle 100.

[0035] The throttle / brake sensor 125 may detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angles of both the accelerator pedal (throttle) and the brake pedal, or it may measure an electrical signal representing the angles of the accelerator pedal (throttle) and / or the brake pedal. The throttle / brake sensor 125 may also measure the angle of the throttle body of the vehicle 100, which may include part of a physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., butterfly valve and carburetor). In addition, the throttle / brake sensor 125 may measure the pressure of one or more brake pads on the rotor of the vehicle 100, or a combination (or subset) of the angles of the accelerator pedal (throttle) and the brake pedal, an electrical signal representing the angles of the accelerator pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied by at least one brake pad to the rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure pressure applied to a vehicle pedal, such as the throttle or brake pedal.

[0036] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a brake unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / route finding system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the direction of the vehicle 100, and the throttle 134 may control the acceleration of the vehicle 100 by controlling the operating speed of the engine / motor 118. The brake unit 136 may decelerate the vehicle 100, which may involve using friction to decelerate the wheels / tires 121. In some embodiments, the brake unit 136 may convert the kinetic energy of the wheels / tires 121 into an electric current for subsequent use by one or more systems of the vehicle 100.

[0037] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithms capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide assessments based on incoming sensor data, such as evaluation of individual objects and / or features, evaluation of specific situations, and / or evaluation of possible effects within a given situation.

[0038] The computer vision system 140 may include hardware and software capable of processing and analyzing images to determine objects that are in motion (e.g., other vehicles, pedestrians, cyclists, or animals) and objects that are not in motion (e.g., traffic lights, road boundaries, speed bumps, or potholes). These may include a general-purpose processor such as a central processing unit (CPU), a dedicated processor such as a graphics processing unit (GPU) or tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models. Thus, the computer vision system 140 may use object recognition, structure from motion (SFM), video tracking, and other algorithms used in computer vision, such as recognizing objects, mapping the environment, tracking objects, and estimating the speed of objects.

[0039] The navigation / route search system 142 can determine the driving path of the vehicle 100, which may involve dynamically adjusting the navigation during operation. Thus, the navigation / route search system 142 can navigate the vehicle 100 using, among many other information sources, the sensor fusion algorithm 138, GPS 122, and map data. The obstacle avoidance system 144 can evaluate potential obstacles based on sensor data and cause the vehicle 100's system to avoid or otherwise navigate around potential obstacles.

[0040] As shown in Figure 1, the vehicle 100 may also include peripherals 108 such as a wireless communication system 146, a touchscreen 148, an internal microphone 150, and / or a speaker 152. The peripherals 108 may provide controls or other elements for the user to interact with the user interface 116. For example, the touchscreen 148 may provide information to the user of the vehicle 100. The user interface 116 may also accept input from the user via the touchscreen 148. The peripherals 108 may also enable the vehicle 100 to communicate with devices such as devices in other vehicles.

[0041] The wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication such as Code Division Multiple Access (CDMA), Evolution Data Optimization (EVDO), Global System for Mobile Communications (GSM) / General-Purpose Packet Radio Service (GPRS), or cellular communication such as 4G Worldwide Interoperability for Microwave Access (WiMAX) or Long-Term Evolution (LTE), or 5G. Alternatively, the wireless communication system 146 may communicate with a wireless local area network (WLAN) using Wi-Fi® or other possible connections. The wireless communication system 146 may also communicate directly with devices using, for example, an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are possible within the context of this disclosure. For example, the wireless communication system 146 may include one or more dedicated narrow-area communication (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0042] The vehicle 100 may include a power supply 110 for supplying power to its components. In some embodiments, the power supply 110 may include a rechargeable lithium-ion or lead-acid battery. For example, the power supply 110 may include one or more batteries configured to provide power. The vehicle 100 may also use other types of power supplies. In exemplary embodiments, the power supply 110 and the energy source 119 may be integrated to form a single energy source.

[0043] The vehicle 100 may also include a computer system 112 for performing operations such as those described therein. Thus, the computer system 112 may include at least one processor 113 (which may include at least one microprocessor) capable of executing instructions 115 stored in a non-temporary computer-readable medium such as data storage 114. In some embodiments, the computer system 112 may represent a plurality of computing devices capable of functioning to control individual components or subsystems of the vehicle 100 in a distributed manner.

[0044] In some embodiments, the data storage 114 may include instructions 115 (e.g., program logic) executable by the processor 113 for performing various functions of the vehicle 100, including those described above in relation to Figure 1. The data storage 114 may also include additional instructions, including instructions for transmitting, receiving, interacting with, and / or controlling one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0045] In addition to the instruction 115, the data storage 114 may store data such as road maps and route information, among other things. Such information may be used by the vehicle 100 and the computer system 112 during the operation of the vehicle 100 in autonomous mode, semi-autonomous mode, and / or manual mode.

[0046] Vehicle 100 may include a user interface 116 for providing information to the user of vehicle 100 or for receiving input from the user of vehicle 100. The user interface 116 may control or enable control of the layout of content and / or interactive images that may be displayed on the touchscreen 148. Furthermore, the user interface 116 may include one or more input / output devices in a set of peripherals 108, such as a wireless communication system 146, a touchscreen 148, a microphone 150, and a speaker 152.

[0047] The computer system 112 can control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, or the control system 106) and from the user interface 116. For example, the computer system 112 may utilize input from the sensor system 104 to estimate the outputs generated by the propulsion system 102 and the control system 106. Depending on the embodiment, the computer system 112 may be able to operate to monitor many aspects of the vehicle 100 and its subsystems. In some embodiments, the computer system 112 may disable some or all of the functions of the vehicle 100 based on signals received from the sensor system 104.

[0048] The components of the vehicle 100 may be configured to function in a manner that interconnects with other components within or outside their respective systems. For example, in an exemplary embodiment, the camera 130 may capture multiple images that can represent information about the state of the surrounding environment of the vehicle 100 operating in autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, the computer vision system 140 may be able to recognize inclines (gradients) or other features based on multiple images of the road. Additionally, a combination of the GPS 122 and the features recognized by the computer vision system 140 may be used together with map data stored in the data storage 114 to determine specific road parameters. Furthermore, radar 126 and / or lidar 128, as well as / or several other environmental mapping, range, and / or positioning sensor systems may also provide information about the vehicle's surroundings.

[0049] In other words, a combination of various sensors (which can be called input indicator sensors and output indicator sensors) and the computer system 112 can interact to provide an indicator of input or an indicator of the vehicle's surroundings, which is provided for controlling the vehicle.

[0050] In some embodiments, the computer system 112 may make decisions about various objects based on data provided by systems other than the wireless system. For example, the vehicle 100 may have lasers or other optical sensors configured to sense objects within the vehicle's field of view. The computer system 112 may use the outputs from the various sensors to determine information about objects within the vehicle's field of view and to determine distance and directional information to various objects. The computer system 112 may also determine whether an object is desirable or undesirable based on the outputs from the various sensors.

[0051] Figure 1 shows various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) integrated into vehicle 100, although one or more of these components may be mounted separately from or associated with vehicle 100. For example, data storage 114 may exist partially or completely separately from vehicle 100. Thus, vehicle 100 may be provided in the form of device elements that can be located separately or together. The device elements constituting vehicle 100 may be coupled together so as to be able to communicate together in a wired and / or wireless manner.

[0052] Figures 2A–2E show an exemplary vehicle 200 (e.g., a fully autonomous or semi-autonomous vehicle) which may include some or all of the functions described in relation to vehicle 100 with reference to Figure 1. For illustrative purposes, vehicle 200 is illustrated in Figures 2A–2E as a van with side mirrors, but the disclosure is not limited thereto. For example, vehicle 200 may represent a truck, automobile, semi-trailer truck, motorcycle, golf cart, off-road vehicle, agricultural vehicle, or any other vehicle described elsewhere in this specification (e.g., bus, boat, airplane, helicopter, drone, lawnmower, bulldozer, submarine, all-terrain vehicle, snowmobile, aircraft, recreation vehicle, amusement park vehicle, farm equipment, construction machinery or construction vehicle, warehouse equipment or warehouse vehicle, factory equipment or factory vehicle, tram, train, trolley, pedestrian transport vehicle, and robotic device).

[0053] An exemplary vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidar, one or more radar, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding the vehicle 200. In other words, any sensor system currently known or to be created may be coupled to the vehicle 200 and / or used in conjunction with various operations of the vehicle 200. For example, lidar may be used for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of the vehicle 200. In addition, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent combinations of sensors described herein (e.g., one or more lidar and radar, one or more lidar and cameras, one or more cameras and radar, one or more lidar, cameras, and radar).

[0054] It should be noted that the number, location, and type of sensor systems (e.g., 202 and 204) depicted in Figures 2A-E are intended as non-limiting embodiments of the location, number, and type of such sensor systems in autonomous or semi-autonomous vehicles. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to reduce vehicle size, shape, aerodynamics, fuel economy, aesthetics, or cost, or to adapt to other conditions for specific environments or application situations). For example, sensor systems (e.g., 202 and 204) may be installed in various other locations on the vehicle (e.g., in location 216) and may have a field of view corresponding to the interior and / or surrounding environment of the vehicle 200.

[0055] The sensor system 202 may include one or more sensors mounted on top of the vehicle 200 and configured to detect information about the environment surrounding the vehicle 200 and output an indicator of that information. For example, the sensor system 202 may include any combination of cameras, radar, lidar, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). The sensor system 202 may include one or more movable mounts that can be operated to adjust the orientation of one or more sensors within the sensor system 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to obtain information from each direction around the vehicle 200. In another embodiment, the movable mount of the sensor system 202 may be movable in a scanning manner within a range of a specific angle and / or azimuth and / or elevation. The sensor system 202 may be mounted on the roof of the vehicle, although other mounting locations are also possible.

[0056] Additionally, the sensors of sensor system 202 may be distributed to various locations and do not need to be placed together in a single location. Furthermore, each sensor of sensor system 202 may be configured to move or scan independently of the other sensors of sensor system 202. Additionally or alternatively, multiple sensors may be mounted on one or more of sensor locations 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two lidar devices mounted on sensor locations, and / or one lidar device and one radar mounted on sensor locations.

[0057] One or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more lidar devices. For example, a lidar device may include a plurality of light-emitting devices arranged over a range of angles with respect to a given plane (e.g., the xy-plane). For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot around an axis perpendicular to a given plane (e.g., the z-axis) to illuminate the environment surrounding the vehicle 200 with light pulses. Information about the surrounding environment can be determined based on the detection of various aspects of the reflected light pulses (e.g., elapsed time of flight, polarization, and intensity).

[0058] In exemplary embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide point cloud information that may relate to physical objects in the surrounding environment of the vehicle 200. While the vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are illustrated to include specific features, it will be understood that other types of sensor systems are contemplated within the scope of this disclosure. Furthermore, the exemplary vehicle 200 may include any of the components described in relation to the vehicle 100 in Figure 1.

[0059] In an exemplary configuration, one or more radars may be located on the vehicle 200. Similar to radar 126 described above, one or more radars may include antennas configured to transmit and receive radio waves (e.g., electromagnetic waves having frequencies between 30 Hz and 300 GHz). Such radio waves may be used to determine the distance and / or speed of one or more objects in the environment surrounding the vehicle 200. For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some examples, one or more radars may be located near the rear of the vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of the vehicle 200 for the presence of radio wave reflecting objects. Similarly, one or more radars may be located near the front of the vehicle 200 (e.g., sensor systems 212 and 214) to actively scan the environment near the front of the vehicle 200. The radar may be positioned in a location suitable for illuminating an area including the forward path of the vehicle 200 without being obstructed by other features of the vehicle 200. For example, the radar may be embedded in the front bumper, front headlights, cowl, and / or hood, and / or mounted on or near them. Furthermore, one or more additional radars may be positioned to actively scan the sides and / or rear of the vehicle 200 for the presence of radar-reflective objects, for example by including such devices in or near the rear bumper, side panels, rocker panels, and / or lower running gear.

[0060] The vehicle 200 may include one or more cameras. For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more cameras. The cameras may be photosensitive devices such as still cameras, video cameras, thermal imaging cameras, stereo cameras, and night vision cameras, configured to capture multiple images of the surrounding environment of the vehicle 200. For this purpose, the cameras may be configured to detect visible light and, additionally or alternatively, to detect light from other parts of the spectrum, such as infrared or ultraviolet light. The cameras may be two-dimensional detectors and, optionally, may have a sensitivity range in three-dimensional space. In some embodiments, the cameras may include range detectors configured to produce a two-dimensional image showing the distance from the camera to several points in the surrounding environment. For this purpose, the cameras may use one or more range detection techniques. For example, a camera can provide range information by using structured light techniques, in which the vehicle 200 illuminates objects in the surrounding environment with a predetermined light pattern, such as a grid or checkerboard pattern, and the camera is used to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, the vehicle 200 can determine the distance to a point on the object. The predetermined light pattern may include infrared light or radiation of other wavelengths suitable for such measurements. In some examples, the camera may be mounted inside the windshield of the vehicle 200. Specifically, the camera may be positioned to capture an image from a forward view relative to the orientation of the vehicle 200. Other mounting locations and field of view of the camera may be used, either inside or outside the vehicle 200. The camera may also have associated optical elements that are operable to provide an adjustable field of view. Furthermore, the camera may be mounted on the vehicle 200 using a movable mount to change the camera's directional angle via a pan / tilt mechanism, etc.

[0061] Vehicle 200 may also include one or more acoustic sensors used to sense the surrounding environment of Vehicle 200 (for example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). Acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, and microelectromechanical system (MEMS) microphones) used to sense acoustic waves (i.e., pressure differences) in the fluid (e.g., air) of the environment surrounding Vehicle 200. Such acoustic sensors may be used to identify sounds in the surrounding environment (e.g., sirens, human speech, animal sounds, and alarms) on which the control strategy of Vehicle 200 may be based. For example, if an acoustic sensor detects a siren (e.g., a mobile siren, and / or a fire truck siren), Vehicle 200 may slow down and / or navigate to the edge of the road.

[0062] Although not shown in Figures 2A-2E, the vehicle 200 may include a wireless communication system (e.g., similar to and / or in addition to the wireless communication system 146 in Figure 1). The wireless communication system may include a radio transmitter and a radio receiver that can be configured to communicate with devices outside or inside the vehicle 200. Specifically, the wireless communication system may include, for example, a transceiver configured to communicate with other vehicles and / or computing devices in a vehicle communication system or roadside station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transport systems.

[0063] Vehicle 200 may include, in addition to or instead of, these indicated components. These additional components may include electrical or mechanical functions.

[0064] The control system of the vehicle 200 may be configured to control the vehicle 200 according to a control strategy selected from among several possible control strategies. The control system may be configured to receive information from sensors coupled to the vehicle 200 (on or outside the vehicle 200), modify the control strategy (and associated driving behavior) based on that information, and control the vehicle 200 according to the modified control strategy. The control system may be further configured to monitor the information received from the sensors and continuously evaluate the driving conditions, and may also be configured to modify the control strategy and driving behavior based on changes in the driving conditions. For example, the path taken by the vehicle from one destination to another may be modified based on the driving conditions. Additionally or alternatively, speed, acceleration, turning angle, inter-vehicle distance (i.e., distance to the vehicle in front of the current vehicle), lane selection, etc., may all be modified in response to changes in the driving conditions.

[0065] As described above, in some embodiments, the vehicle 200 may take the form of a van, but alternative forms are also possible and intended herein. Accordingly, Figures 2F to 2I illustrate embodiments in which the vehicle 250 takes the form of a semi-track. For example, Figure 2F illustrates a front view of the vehicle 250, and Figure 2G illustrates an isometric view of the vehicle 250. In embodiments in which the vehicle 250 is a semi-track, the vehicle 250 may include a tractor portion 260 and a trailer portion 270 (illustrated in Figure 2G). Figures 2H and 2I provide a side view and a top view of the tractor portion 260, respectively. Similar to the vehicle 200 illustrated above, the vehicle 250 illustrated in Figures 2F to 2I may also include various sensor systems (for example, sensor systems 202, 206, 208, 210, 212, and 214 shown and described with reference to Figures 2A to 2E). In some embodiments, the vehicle 200 in Figures 2A-2E may include only a single copy of several sensor systems (e.g., sensor system 204), while the vehicle 250 illustrated in Figures 2F-2I may include multiple copies of its sensor system (e.g., sensor systems 204A and 204B as illustrated).

[0066] While the drawings and overall description may refer to a given vehicle configuration (e.g., a semi-truck vehicle 250 or a van vehicle 200), it should be understood that the embodiments described herein are equally applicable in the context of various vehicles (e.g., with modifications adopted to take into account the vehicle's form factor). For example, sensors and / or other components described or illustrated as part of a van vehicle 200 may also be used in a semi-truck vehicle 250 (e.g., for navigation and / or obstacle detection and avoidance).

[0067] Figure 2J illustrates various sensor fields of view (e.g., associated with the vehicle 250 described above). As described above, the vehicle 250 may contain multiple sensors / sensor units. The locations of various sensors may correspond to the sensor locations disclosed in Figures 2F-2I, for example. However, in some cases, sensors may have other locations. For the sake of simplicity in the drawings, sensor location reference numbers are omitted from Figure 2J. For each sensor unit of the vehicle 250, Figure 2J illustrates typical fields of view (e.g., fields of view labeled as 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The sensor field of view may include angular regions (e.g., azimuth and / or elevation regions) in which the sensor can detect objects.

[0068] Figure 2K illustrates beam steering with respect to sensors in a vehicle (e.g., vehicle 250 shown and described with reference to Figures 2F-2J) according to an exemplary embodiment. In various embodiments, the sensor unit of vehicle 250 may be radar, lidar, sonar, etc. Furthermore, in some embodiments, while the sensor is operating, the sensor may scan within its field of view. Various different scanning angles for the exemplary sensor are shown as regions 272, each representing the angular region in which the sensor is operating. The sensor may periodically or iteratively change the region in which it is operating. In some embodiments, multiple sensors may be used by vehicle 250 to measure the regions 272. In addition, other regions may be included in other examples. For example, one or more sensors may measure the aspect of the trailer 270 of vehicle 250 and / or the region in front of vehicle 250.

[0069] At certain angles, the sensor's operating area 275 may include the rear wheels 276A and 276B of the trailer 270. Therefore, the sensor may measure the rear wheels 276A and / or 276B during operation. For example, the rear wheels 276A and 276B may reflect lidar or radar signals transmitted by the sensor. The sensor may receive signals reflected from the rear wheels 276A and 276. Thus, the data collected by the sensor may include data from reflections from the wheels.

[0070] In some cases, such as when the sensor is radar, reflections from rear wheels 276A and 276B may appear as noise in the received radar signal. As a result, the radar may operate with an enhanced signal-to-noise ratio in cases where rear wheels 276A and 276B direct the radar signal away from the sensor.

[0071] Figure 3 is a conceptual illustration of wireless communication between various computing systems related to an autonomous or semi-autonomous vehicle, according to an exemplary embodiment. In particular, wireless communication may occur between a remote computing system 302 and a vehicle 200 via a network 304. Wireless communication may also occur between a server computing system 306 and a remote computing system 302, and between the server computing system 306 and a vehicle 200.

[0072] Vehicle 200 can be of various types capable of transporting passengers or objects between locations and may take any one or more forms of the vehicles considered above. In some cases, vehicle 200 may operate in autonomous or semi-autonomous mode, allowing a control system to safely navigate vehicle 200 between destinations using sensor measurements. When operating in autonomous or semi-autonomous mode, vehicle 200 may navigate with or without passengers. As a result, vehicle 200 may pick up and drop off passengers between desired destinations.

[0073] The remote computing system 302 may represent any type of device relating to remote assistance technology, including but not limited to those described herein. In examples, the remote computing system 302 may represent any type of device configured to (i) receive information relating to the vehicle 200, (ii) provide an interface through which a human operator can then become aware of the information and input a response relating to the information, and (iii) transmit the response to the vehicle 200 or to another device. The remote computing system 302 may take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some examples, the remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0074] The remote computing system 302 may include one or more subsystems and components similar to or identical to those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface, including input / output devices such as a touchscreen and speakers. Other examples are similarly possible.

[0075] Network 304 represents the infrastructure that enables wireless communication between the remote computing system 302 and the vehicle 200. Network 304 also enables wireless communication between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0076] The location of the remote computing system 302 can vary within the scope of the example. For example, the remote computing system 302 may be located remotely from the vehicle 200, which has wireless communication via the network 304. In another embodiment, the remote computing system 302 may correspond to a computing device within the vehicle 200, separate from the vehicle 200, but in which a human operator can interact with the passengers or driver of the vehicle 200. In some examples, the remote computing system 302 may be a computing device with a touchscreen that can be operated by the passengers of the vehicle 200.

[0077] In some embodiments, the operations described herein, performed by the remote computing system 302, may be additionally or alternatively performed by the vehicle 200 (i.e., by any system or subsystem of the vehicle 200). In other words, the vehicle 200 may be configured to provide a remote assistance mechanism that can be interacted with by the vehicle's driver or passengers.

[0078] The server computing system 306 may be configured to communicate wirelessly with the remote computing system 302 and the vehicle 200 (or, optionally, directly with the remote computing system 302 and / or the vehicle 200) via the network 304. The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information about the vehicle 200 and its remote assistance. Thus, the server computing system 306 may be configured to perform any operation or part of such operation described herein as being performed by the remote computing system 302 and / or the vehicle 200. The server computing system 306 may be available in some embodiments of the wireless communication related to remote assistance, but not in other embodiments.

[0079] The server computing system 306 may include one or more subsystems and components that are similar to or identical to those of the remote computing system 302 and / or vehicle 200, such as a processor configured to perform the various operations described herein, and a wireless communication interface for receiving and providing information to the remote computing system 302 and the vehicle 200.

[0080] The various systems described above can perform a variety of operations. These operations and their associated characteristics are described below.

[0081] In line with the above considerations, computing systems (e.g., remote computing system 302, server computing system 306, and computing system local to vehicle 200) may operate to capture images of the environment surrounding the autonomous or semi-autonomous vehicle using cameras. Generally, at least one computing system can analyze the images and, if possible, control the autonomous or semi-autonomous vehicle.

[0082] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data (also referred to herein as “environmental data”) representing objects in the environment surrounding the vehicle in various ways. The vehicle’s sensor system may provide environmental data representing objects in the surrounding environment. For example, a vehicle may have a variety of sensors, including cameras, radar, lidar, microphones, radio units, and other sensors. Each of these sensors may communicate the environmental data to a processor in the vehicle about the information it receives.

[0083] In one example, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have two or more cameras positioned in different orientations. Also in some embodiments, the cameras may be able to move to capture images and / or video in different directions. The camera may be configured to store the captured images and video in memory for subsequent processing by the vehicle's processing system. The captured images and / or video may be environmental data. Furthermore, the camera may include an image sensor as described herein.

[0084] In another embodiment, the radar may be configured to transmit electromagnetic signals reflected by various objects near the vehicle and then capture the electromagnetic signals reflected from those objects. The captured reflected electromagnetic signals may enable the radar (or processing system) to make various decisions about the objects that reflected the electromagnetic signals. For example, the distance and position to various reflective objects may be determined. In some embodiments, the vehicle may have two or more radars in different orientations. The radar may be configured to store the captured information in memory for subsequent processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0085] In another embodiment, the lidar may be configured to transmit electromagnetic signals (e.g., infrared light, such as from a gas or diode laser, or other possible light source) reflected by a target object near the vehicle. The lidar may be capable of acquiring the reflected electromagnetic (e.g., infrared) signals. The acquired reflected electromagnetic signals may allow a ranging system (or processing system) to determine the distance to various objects. The lidar can also determine the velocity or speed of the target object, which can be stored as environmental data.

[0086] Additionally, in one example, the microphone may be configured to capture audio from the vehicle's surrounding environment. The sounds captured by the microphone may include the sirens of emergency vehicles and the sounds of other vehicles. For example, the microphone may capture the sounds of sirens from ambulances, fire engines, and police vehicles. The processing system may be able to identify that the captured audio signal indicates an emergency vehicle. In another embodiment, the microphone may capture the sound of exhaust from another vehicle, such as the exhaust from a motorcycle. The processing system may be able to identify that the captured audio signal indicates a motorcycle. The data captured by the microphone may form part of the environmental data.

[0087] In yet another embodiment, the radio unit may be configured to transmit an electromagnetic signal that may take the form of a Bluetooth signal, an 802.11 signal, and / or other radio technology signal. The first electromagnetic radiation signal may be transmitted via one or more antennas located on the radio unit. Furthermore, the first electromagnetic radiation signal may be transmitted in one of many different radio signal modes. However, in some embodiments, it is desirable to transmit the first electromagnetic radiation signal in a signal mode that requests a response from a device located near an autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on the response returned to the radio unit and use this communicated information as part of the environmental data.

[0088] In some embodiments, the processing system may be able to combine information from various sensors to further determine the vehicle's surrounding environment. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, other combinations of sensor data may be used by the processing system to make decisions about the surrounding environment.

[0089] While operating in autonomous (or semi-autonomous) mode, a vehicle may be able to control its movements with little or no human input. For example, if a human operator enters an address into the vehicle, the vehicle may be able to drive to the designated destination without further human input (e.g., without the human needing to operate or touch the brake / accelerator pedals). Furthermore, while the vehicle is operating autonomously or semi-autonomously, sensor systems may receive environmental data. The vehicle's processing system may modify the vehicle's control based on the environmental data received from various sensors. In some embodiments, the vehicle may change its speed in response to environmental data from various sensors. The vehicle may change its speed to avoid obstacles, comply with traffic laws, etc. If the processing system in the vehicle identifies an object near the vehicle, the vehicle may change its speed or otherwise alter its movement.

[0090] If a vehicle detects an object but is not confident in its detection, it may request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object is actually present in the surrounding environment (e.g., whether there is actually a stop sign or not), (ii) verifying whether the vehicle's identification of the object is correct, (iii) correcting the identification if it was incorrect, and / or (iv) providing supplementary instructions to the autonomous or semi-autonomous vehicle (or modifying the current instructions). Remote assistance tasks may also include providing instructions for the human operator to control the vehicle's actions (e.g., if the human operator determines that the object is a stop sign, instructing the vehicle to stop at the stop sign), although in some scenarios the vehicle itself may control its actions based on human operator feedback related to object identification.

[0091] To facilitate this, the vehicle may analyze environmental data representing objects in the surrounding environment to determine at least one object with a detection confidence level below a threshold. The vehicle's processor may be configured to detect various objects in the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, cyclists, street signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0092] Detection confidence can indicate the likelihood that a determined object is correctly identified or present in its surrounding environment. For example, a processor may perform object detection on objects in image data within the received environmental data and determine that an object has a detection confidence below a threshold if it cannot identify at least one object with a detection confidence above a threshold. If the result of object detection or recognition of an object is inconclusive, the detection confidence may be low or below a set threshold.

[0093] Depending on the source of the environmental data, a vehicle may detect objects in its surroundings in various ways. In some embodiments, the environmental data may be image or video data coming from a camera. In other embodiments, the environmental data may come from a LiDAR. The vehicle may analyze the captured image or video data to identify objects within the image or video data. Methods and apparatus may be configured to monitor image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in the surrounding environment based on radar, audio, or other data.

[0094] In some embodiments, the technique used by a vehicle to detect objects may be based on a set of known data. For example, data related to environmental objects may be stored in memory located in the vehicle. The vehicle may determine an object by comparing the received data with the stored data. In other embodiments, the vehicle may be configured to determine an object based on the context of the data. For example, road signs related to construction may generally be orange. Therefore, the vehicle may be configured to detect an orange object located near the side of the road as a road sign related to construction. Additionally, when the vehicle's processing system detects an object in the captured data, it may also calculate the confidence level of each object.

[0095] Furthermore, a vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object being detected. For example, an object that may require a quick response action from the vehicle, such as the brake lights of another vehicle, may have a lower confidence threshold. However, in other embodiments, the confidence threshold may be the same for all detected objects. If the confidence associated with a detected object is higher than the confidence threshold, the vehicle may assume that the object has been correctly recognized and responsively adjust its control based on that assumption.

[0096] If the confidence level associated with a detected object is lower than the confidence threshold, the action taken by the vehicle may change. In some embodiments, the vehicle may react as if the detected object exists, despite the low confidence level. In other embodiments, the vehicle may react as if the detected object does not exist.

[0097] When a vehicle detects an object in its surrounding environment, it can also calculate a confidence level associated with that specific detected object. The confidence level can be calculated in various ways depending on the embodiment. In one example, upon detecting an object in the surrounding environment, the vehicle may compare environmental data with predetermined data associated with known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle may use a mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0098] In response to a determination that an object has a detection confidence level below a threshold, the vehicle may send a request for remote assistance, along with the identification of the object, to a remote computing system. As considered above, the remote computing system can take various forms. For example, the remote computing system may be a computing device located within the vehicle, separate from the vehicle itself, but which may be a touchscreen interface for displaying remote assistance information, allowing a human operator to interact with the vehicle's passengers or driver. Additionally or alternatively, in another embodiment, the remote computing system may be a remote computer terminal or other device located not near the vehicle.

[0099] Remote assistance requests may include environmental data, such as image data and audio data, including objects. The vehicle may transmit the environmental data to the remote computing system via a network (e.g., network 304) and, in some embodiments, a server (e.g., server computing system 306). A human operator at the remote computing system may then use the environmental data as a basis for responding to the request.

[0100] In some embodiments, if an object is detected as having a confidence level below a confidence threshold, the object may be given a preliminary identification, and the vehicle may be configured to adjust its operation in response to the preliminary identification. Such adjustments to operation may take the form of stopping the vehicle, switching the vehicle to a human-controlled mode, or changing the vehicle's speed (e.g., speed and / or direction), among other possible adjustments.

[0101] In other embodiments, even if the vehicle detects an object with a confidence level that meets or exceeds a threshold, the vehicle may act according to the detected object (for example, stop if the object is identified with high confidence as a stop sign), but the vehicle may be configured to request remote assistance at the same time as (or after) acting according to the detected object.

[0102] Referring to Figure 4, a specific exemplary example of a system 400 capable of operating to reduce automatic exposure latency is shown. The system 400 includes a processor 402, a memory 404, and an image sensor 406. According to one implementation, the processor 402, the memory 404, and the image sensor 406 may be integrated into the circuit or into a single chip so that the processor 402 provides “on-chip” processing capabilities to the image sensor 406. The processor 402 may be coupled to the memory 404 to enable the exchange of data and commands between the processor 402 and the memory 404, the memory 404 may be coupled to the image sensor 406 to enable the exchange of data between the memory 404 and the image sensor 406, and the image sensor 406 may be coupled to the processor 402 to enable the exchange of data and commands between the image sensor 406 and the processor 402.

[0103] In one implementation, system 400 may be integrated with one or more sensor systems. As a non-limiting example, system 400 may be integrated with one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 of vehicle 200. As another non-limiting example, system 400 may be integrated with sensor system 104 of vehicle 100. In a particular implementation, system 400 may be integrated with camera 130. The above embodiments are not intended to be limiting, and it should be understood that system 400 may be integrated with other sensor systems and cameras.

[0104] The processor 402 includes an exposure operation unit 410, a pre-exposure operation unit 412, a frame analysis unit 414, and an exposure parameter determination unit 416. According to one implementation, one or more of the units 410, 412, 414, and 416 may be implemented using dedicated hardware such as one or more ASICs or one or more FPGAs. According to one implementation, one or more of the units 410, 412, 414, and 416 may be implemented using instructions 490 stored in memory 404, which are executed by the processor 402. For example, the processor 402 can perform the operations described herein by executing instructions 490 stored in memory 404 (e.g., a non-temporary computer-readable medium).

[0105] Note that in some implementations, operations or functions associated with one or more of units 410, 412, 414, and 416 may be integrated into a single unit. As a non-limiting example, the functions of exposure operation unit 410 and pre-exposure operation unit 412 may be integrated into a single unit. Also note that in some implementations, one or more of units 410, 412, 414, and 416 may not be present in processor 402. As a non-limiting example, in some implementations, such as the one described with respect to Figure 5, pre-exposure operation unit 412 may not be present in processor 402 if no pre-exposure frame is generated.

[0106] The exposure operation unit 410 may be configured to initiate a frame exposure operation that allows the image sensor 406 to capture different frames 450. According to one implementation, the exposure operation unit 410 may send a trigger signal 415 to the image sensor 406 to enable the image sensor 406 to capture a frame 450. For example, during a frame period to capture frame 450A, the exposure operation unit 410 may send a trigger signal 415 to the image sensor 406 to enable the image sensor 406 to capture frame 450A. Similarly, during frame periods to capture other frames 450B, 450C, the exposure operation unit 410 may send a trigger signal 415 to the image sensor 406 to enable the image sensor 406 to capture frames 450B, 450C, respectively. In some implementations, the trigger signal 415 may be generated from a host outside the system 400 and sent to the image sensor 406.

[0107] The frame analysis unit 414 may be configured to determine the frame characteristics 460 (e.g., image characteristics) of a frame 450 captured by the image sensor 406. For example, the frame analysis unit 414 can determine the frame characteristics 460A of frame 450A in response to the image sensor 406 capturing frame 450B, the frame analysis unit 414 can determine the frame characteristics 460B of frame 450B in response to the image sensor 406 capturing frame 450C, and the frame analysis unit 414 can determine the frame characteristics 460C of frame 450C in response to the image sensor 406 capturing frame 450C. According to one implementation, the frame characteristics 460 may represent illumination characteristics associated with frame 450. However, in other implementations, the frame characteristics 460 may represent other characteristics such as noise level or contrast.

[0108] In some implementations, the pre-exposure operation unit 412 may be configured to initiate a frame exposure operation that enables the image sensor 406 to capture a pre-exposure frame 452. For example, the pre-exposure operation unit 412 may generate and transmit a trigger signal 413 that enables the image sensor 406 to perform a pre-exposure operation and capture a pre-exposure frame 452. The pre-exposure frame 452 is a "partial" frame that is captured at the beginning of the frame period and analyzed to determine one or more exposure parameters 430, 434 for frames (e.g., "full" frames) captured later in the frame period. The pre-exposure frame 452 may be generated in response to sampling a subset of image pixels on the image sensor 406. According to one implementation, the subset of image pixels is associated with a particular region of interest on the image sensor 406. As a non-limiting example, the particular region of interest on the image sensor 406 may correspond to a rectangular region adjacent to the central section on the image sensor 406. As another non-restrictive example, a particular region of interest on the image sensor 406 may correspond to one or more rows (e.g., pixel rows), one or more columns (e.g., pixel columns), or both. According to another implementation, a subset of image pixels is distributed across the image sensor 406.

[0109] In a manner similar to how the frame analysis unit 414 determines the frame characteristics 460 of frame 450, the frame analysis unit 414 may be configured to determine the pre-exposure frame characteristics 462 (e.g., image characteristics) of the pre-exposure frame 452 captured by the image sensor 406. According to one implementation, the pre-exposure frame characteristics 462 may represent illumination characteristics associated with the pre-exposure frame 452. However, in other implementations, the pre-exposure frame characteristics 462 may represent other characteristics such as noise level or contrast.

[0110] The exposure parameter determination unit 416 may be configured to determine target settings 432, 436 for each of the different exposure parameters 430, 434. The exposure parameters 430, 434 may correspond to the exposure time of the image sensor 406 (e.g., shutter speed), the aperture size of the image sensor 406, analog gain, digital gain, etc. For ease of explanation, unless otherwise stated, exposure parameter 430 corresponds to the exposure time. However, it should be understood that in other implementations, one or more of the exposure parameters 430, 434 may correspond to different parameters (e.g., aperture size, analog gain, and / or digital gain). Also note that in other implementations, the exposure parameter determination unit 416 may be configured to determine target settings for three or more exposure parameters, or to determine a target setting for a single exposure parameter.

[0111] The exposure parameter determination unit 416 can determine the settings 432 and 436, respectively, of the exposure parameters 430 and 434 for the frame captured by the image sensor 406, at least in part, based on the frame characteristics 460 and 462. For example, according to one implementation, as described in more detail with respect to Figure 5, the exposure parameter determination unit 416 can analyze the frame characteristics 460A of a previous frame (e.g., frame 450A) and, based on the analysis, determine how to adjust the settings 432 and 436, respectively, of the exposure parameters 430 and 434 for the subsequent frames 450B and 450C captured by the image sensor 406. According to another implementation, as described in more detail with respect to Figure 6, the exposure parameter determination unit 416 can analyze the pre-exposure frame characteristics 462 of the previous-exposure frame 452 and, based on the analysis, determine how to adjust the settings 432 and 436, respectively, of the exposure parameters 430 and 434 for the captured frame 450B. In another implementation, as described in more detail with respect to Figure 7, the exposure parameter determination unit 416 can analyze data from an external source (for example, data collected from another sensor) and, based on the analysis, determine how to adjust the settings 432, 436 of the exposure parameters 430, 434, respectively, for the captured frame 450.

[0112] As described above, the image sensor 406 may be configured to capture an image of the surrounding environment. In a non-limiting example, the image sensor 406 may be configured to capture an image (e.g., a frame) of the surrounding environment of the vehicle 100, the vehicle 200, or both. Although generally described herein as an active pixel sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) sensor), in some implementations, the image sensor 406 may be a charge-coupled device (CCD).

[0113] In some scenarios, when the characteristics of the surrounding environment change (for example, when vehicle 100 enters or exits a dark tunnel), the exposure parameters 430, 434 associated with the image sensor 406 need to be adjusted to capture a high-quality image. For example, when vehicle 100 enters a tunnel and the surrounding environment becomes relatively dark, the exposure time used to capture an image with image sensor 406 can be increased to increase the illumination characteristics of the image (e.g., brighten the image). However, when vehicle 100 exits a tunnel and the surrounding environment becomes relatively bright, the exposure time used to capture an image with image sensor 406 can be decreased to reduce the illumination characteristics of the image (e.g., darken the image). As described below with respect to Figures 5-7, the on-chip processor 402 may be configured to reduce the latency associated with adjusting the exposure parameters 430, 434 of image sensor 406 so that the exposure parameters 430, 434 can be adjusted within a single frame period and a corresponding frame with the adjusted exposure parameters can be generated.

[0114] Referring to Figure 5, a specific exemplary example of process 500 for reducing automatic exposure latency is shown. Process 500 can be carried out by various components of system 400 in Figure 4. In particular, one or more operations of process 500 can be carried out by processor 402 of system 400.

[0115] Process 500 represents three frame periods 550 associated with the capture of different frames 450. For example, process 500 represents frame period 550A associated with the capture of frame 450A, frame period 550B associated with the capture of frame 450B, and frame period 550C associated with the capture of frame 450C. During each frame period 550, a trigger signal 415 is emitted to initiate the capture of the corresponding frame 450. For example, if the trigger signal 415 is emitted during frame period 550A, a frame exposure operation to capture frame 450A is initiated. During the frame exposure operation to capture frame 450A, a shutter release 580A occurs to capture frame 450A, and after an exposure time 584A has elapsed, frame 450A is read 582A. Similarly, if the trigger signal 415 is emitted during frame period 550B, a frame exposure operation to capture frame 450B is initiated. During the frame exposure operation to capture frame 450B, a shutter release 580B occurs to capture frame 450B, and after an exposure time of 584B has elapsed, frame 450B is read at 582B.

[0116] According to process 500, the processor 402 may be configured to determine the setting 432 of exposure parameters 430 (e.g., exposure time 584B) for a frame 450B captured by the image sensor 406. For example, during the exposure parameter determination period 570, the processor 402 can determine the exposure time 584B for the captured frame 450B. The exposure parameter determination period 570 is a subset of the frame period 550B for capturing the frame 450B. Thus, the processor 402 can determine the setting 432 of exposure parameters 430 (e.g., set the exposure time 584B) for the frame 450B during the same frame period 550B in which the frame 450B is generated.

[0117] The processor 402 can determine the setting of exposure parameters 430 for frame 450B based at least in part on the frame characteristics 460A of frame 450A (e.g., the previous frame). As a non-limiting example, if the frame characteristics 460 of the previous frame 450A indicate that the illumination metric of the previous frame 450A did not meet (e.g., did not exceed) the lower illumination threshold (e.g., the previous frame 450A was too dark), then during the frame duration 550B of the current frame 450B (e.g., during the exposure parameter determination period 570), the processor 402 may determine that the exposure time 584B of the current frame 450B should be extended (compared to the exposure time 584A of the previous frame 450A), and the extended exposure time 584B can be performed on the image sensor 406. Conversely, if the frame characteristics 460 of the previous frame 450A indicate that the illumination metric of the previous frame 450A did not meet (e.g., did not fall below) the upper illumination threshold (e.g., the previous frame 450A was too bright), then during the frame duration 550B of the current frame 450B (e.g., during the exposure parameter determination period 570), the processor 402 may decide that the exposure time 584B of the current frame 450B should be shortened (compared to the exposure time 584A of the previous frame 450A), and the shortened exposure time 584B can be performed on the image sensor 406.

[0118] The above example describes adjusting the exposure time 584A based on the frame characteristics 460A of frame 450A (the previous frame), but it should be understood that the processor 402 can adjust the settings of other exposure parameters during the exposure parameter determination period 570 and apply the adjusted settings to frame 450B. As a non-limiting example, based on the frame characteristics 460A of frame 450A, the processor 402 can adjust analog gain settings, digital gain settings, etc. In addition, as will be described in more detail with respect to Figure 7, exposure parameters may be further adjusted based on external inputs (e.g., data from other sensors).

[0119] Therefore, the techniques described in relation to Figures 4 and 5 enable the processor 402 to reduce the latency associated with adjusting the exposure time 584B. For example, instead of having a delay (single-frame delay or multiple-frame delay) between when the target exposure time 584B (or other target exposure parameter) is determined and when a frame 450B with the target exposure time 584B is read from the image sensor 406, the techniques described in relation to Figures 4 and 5 can reduce (or eliminate) the delay so that the frame 450B with the target exposure time 584B is read during the same frame period 550B in which the target exposure time 584B is determined. By reducing latency as described above, the techniques described in Figures 4 and 5 can reduce the number of overexposed frames captured and reduce the number of underexposed frames captured.

[0120] Referring to Figure 6, a specific exemplary example of process 600 for reducing automatic exposure latency is shown. Process 600 can be carried out by various components of system 400 in Figure 4. In particular, one or more operations of process 600 can be carried out by processor 402 of system 400.

[0121] Process 600 represents two frame periods 650 associated with the capture of different frames 450. For example, process 600 represents frame period 650A associated with the capture of frame 450A, and frame period 650B associated with the capture of frame 450B. During each frame period 650, a trigger signal 415 is emitted to initiate the capture of the corresponding frame 450. For example, if the trigger signal 415 is emitted during frame period 650A, a frame exposure operation to capture frame 450A is initiated. During the frame exposure operation to capture frame 450A, a shutter release 680A occurs to capture frame 450A, and after an exposure time 684A has elapsed, frame 450A is read from the read process 682A. Similarly, if the trigger signal 415 is emitted during frame period 650B, a frame exposure operation to capture frame 450B is initiated. During the frame exposure operation to capture frame 450B, a shutter release 680B occurs to capture frame 450B, and after an exposure time of 684B has elapsed, frame 450B is read at 682B.

[0122] Prior to capturing frame 450B, the pre-exposure frame 452 is captured during frame period 650B. For example, at the start of frame period 650B, a trigger signal 413 is emitted to enable the capture of the pre-exposure frame 452. The pre-exposure frame 452 may be generated in response to sampling a subset of image pixels on the image sensor 406. The subset of image pixels may be associated with a particular region of interest (e.g., a region adjacent to the central section of the image sensor, a particular row, and / or a particular column), or it may be distributed across the image sensor 406.

[0123] In response to capturing the previous exposure frame 452 at the start of frame period 650B, during the exposure parameter determination period 670 which falls within frame period 650B (e.g., a subset thereof), the processor 402 can determine the exposure time 684B for the captured frame 450B. Thus, during the same frame period 650B in which frame 450B is generated, the processor 402 can determine the setting 432 for the exposure parameter 430 for frame 450B (e.g., the setting of the exposure time 684B).

[0124] According to process 600, the processor 402 may be configured to determine the exposure time 684B of the frame to be captured 450B based at least in part on the pre-exposure frame characteristics 462 of the pre-exposure frame 452. As a non-limiting example, if the pre-exposure frame characteristics 462 of the pre-exposure frame 452 indicate that the illumination metric of the pre-exposure frame 452 did not meet the lower illumination threshold (e.g., the pre-exposure frame 452 was too dark), during the exposure parameter determination period 670, the processor 402 may determine that the exposure time 684B of the current frame 450B should be extended, and the extended exposure time 684B can be performed on the image sensor 406. Conversely, if the pre-exposure frame characteristics 462 of the pre-exposure frame 452 indicate that the illumination metric of the pre-exposure frame 452 did not meet the upper illumination threshold (for example, the pre-exposure frame 452 was too bright), then during the exposure parameter determination period 670, the processor 402 may determine that the exposure time 684B of the current frame 450B should be shortened, and the shortened exposure time 684B can be performed on the image sensor 406.

[0125] The above example describes adjusting the exposure time 684A based on the pre-exposure frame characteristics 462 of the pre-exposure frame 452, but it should be understood that the processor 402 can adjust the settings of other exposure parameters during the exposure parameter determination period 670 and apply the adjusted settings to frame 450B. As a non-limiting example, based on the pre-exposure frame characteristics 462 of the pre-exposure frame 452, the processor 402 can adjust analog gain settings, digital gain settings, etc. In addition, as will be described in more detail with respect to Figure 7, exposure parameters may be further adjusted based on external inputs (e.g., data from other sensors).

[0126] Therefore, the techniques described in relation to Figures 4 and 6 enable the processor 402 to reduce the latency associated with adjusting the exposure time 684B. For example, instead of having a delay (single-frame delay or multiple-frame delay) between when the target exposure time 684B (or other target exposure parameter) is determined and when a frame 450B with the target exposure time 684B is read from the image sensor 406, the techniques described in relation to Figures 4 and 6 can reduce (or eliminate) the delay so that the frame 450B with the target exposure time 684B is read during the same frame period 650B in which the target exposure time 684B is determined. By reducing the latency as described above, the techniques described in Figures 4 and 6 can reduce the number of overexposed frames captured and reduce the number of underexposed frames captured.

[0127] Referring to Figure 7, a specific exemplary example of process 700 for reducing automatic exposure latency is shown. Process 700 can be implemented by various components of system 400 in Figure 4. In particular, one or more operations of process 700 can be implemented by processor 402 of system 400.

[0128] Process 700 represents two frame periods 750A and 750B associated with the capture of different frames 450. For example, process 700 represents frame period 750A associated with the capture of frame 450A, and frame period 750B associated with the capture of frame 450B. During each frame period, a trigger signal 415 is emitted to initiate the capture of the corresponding frame 450. For example, if the trigger signal 415 is emitted during frame period 750A, a frame exposure operation to capture frame 450A is initiated. During the frame exposure operation to capture frame 450A, a shutter release 780A occurs to capture frame 450A, and after an exposure time 784A has elapsed, frame 450A is read in read process 782A. Similarly, if the trigger signal 415 is emitted during frame period 750B, a frame exposure operation to capture frame 450B is initiated. During the frame exposure operation to capture frame 450B, a shutter release 780B occurs to capture frame 450B, and after an exposure time of 784B has elapsed, frame 450B is read in the reading process 782B.

[0129] Prior to capturing frame 450B, the previous exposure frame 452 is captured during frame period 750B. For example, in a manner similar to that described with respect to Figure 6, a trigger signal 413 is emitted at the start of frame period 750B to enable the capture of the previous exposure frame 452. In addition, an external input 790 may also be received by the processor 402 during frame period 750B. The external input 790 may include data captured from one or more other sensors (e.g., other cameras, one or more lidars, one or more radars, and / or one or more accelerometers). As a non-limiting example, the external input 790 may include data indicating environmental conditions based on image frames captured by one or more other sensors. According to some implementations, the image frame underlying the external input 790 may be captured immediately before image frame 450B, which is captured during frame period 750B.

[0130] During the exposure parameter determination period 770, which falls within the frame period 750B (for example, a subset thereof), the processor 402 can determine the settings 432, 436 for the exposure parameters 430, 434 for the captured frame 450B based on the external input 790, the frame characteristics 460A of the previous frame 450A, the pre-exposure frame characteristics 462 of the previous exposure frame 452, or a combination thereof. For example, based on at least one of the external input 790, the frame characteristics 460A of the previous frame 450A, or the pre-exposure frame characteristics 462 of the previous exposure frame 452, the processor 402 can determine the exposure time 784B of frame 450B, the analog gain of frame 450B, the digital gain of frame 450B, etc., and adjust the exposure parameters 430, 434 for frame 450B.

[0131] Therefore, the techniques described in relation to Figures 4 and 7 enable the processor 402 to reduce the latency associated with adjusting the exposure time 784B. For example, instead of having a delay (single-frame delay or multiple-frame delay) between when the target exposure time 784B is determined and when a frame 450B with a target exposure time 684B is read from the image sensor 406, the techniques described in relation to Figures 4 and 7 can reduce (or eliminate) the delay so that the frame 450B with a target exposure time 784B is read during the same frame period 750B in which the target exposure time 784B is determined. By reducing the latency as described above, the techniques described in Figures 4 and 7 can reduce the number of overexposed frames captured and reduce the number of underexposed frames captured.

[0132] Referring to Figure 8, a specific exemplary example of method 800 for reducing automatic exposure latency is shown. Method 800 can be implemented by various components of system 400 in Figure 4. In particular, one or more operations associated with method 800 can be implemented by the processor 402 of system 400.

[0133] Method 800 includes, in block 802, having the processor determine a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined at least in part on the characteristics of a previous frame captured by the image sensor, and the first setting of exposure parameters is determined during a first frame period associated with the capture of the first frame. For example, referring to Figures 4 and 5, the processor 402 may determine a setting 432 of exposure parameters 430 for a frame 450B captured by the image sensor 406. The setting 432 of exposure parameters 430 can be determined at least in part on the frame characteristics 460A of a frame 450A captured by the image sensor 406, and the setting 432 can be determined during a frame period 550B associated with the capture of frame 450B. According to one implementation of Method 800, the preceding frame 450A precedes the first frame period 550B and is captured during an adjacent frame period 550A.

[0134] Method 800 also includes, in block 804, having the processor initiate a first frame exposure operation during a first frame period. The first frame exposure operation is based on a first setting of exposure parameters. For example, referring to Figures 4 and 5, the processor 402 may initiate a frame exposure operation during a frame period 550B. The frame exposure operation may be based on a setting 432 of exposure parameters 430.

[0135] According to one implementation of Method 800, the exposure parameter corresponds to an exposure time of 584B. According to another implementation of Method 800, the exposure parameter corresponds to an analog gain or a digital gain.

[0136] According to one implementation, method 800 also includes determining a first setting of exposure parameters based on an external input. For example, referring to Figures 4 and 7, the processor 402 can determine a setting 432 of exposure parameters 430 for frame 450B based on the frame characteristics 460A of frame 450A, as well as the external input 790. The external input 790 may be based on external sensor data from one or more other sensors. For example, according to some implementations, the external sensor data may correspond to image sensor data or environmental sensor data.

[0137] According to one implementation of Method 800, the first frame exposure operation is initiated in response to the reception of a trigger signal, such as a trigger signal 415. Method 800 may also include scheduling the transmission of the trigger signal 415 during the first frame period 550B, and transmitting the trigger signal 415 to the image sensor 406 during the first frame period 550B in accordance with the scheduled transmission of the trigger signal 415.

[0138] According to one implementation, method 800 also includes determining a second setting of exposure parameters for a second frame captured by the image sensor. For example, referring to Figures 4 and 5, the processor 402 may determine another setting of exposure parameter 430 for frame 450C captured by the image sensor 406. The second setting of exposure parameter 430 may be determined at least in part on the characteristics 460B of the first frame 450B. The second setting may be determined during a second frame period 550C associated with the capture of the second frame 450C, the second frame period 550C may occur following and adjacent to the first frame period 550B. Method 800 may also include performing a second frame exposure operation during the second frame period 550C. The second frame exposure operation is based on the second setting of exposure parameter 430. According to one implementation of method 800, the second setting of exposure parameter 430 is further determined on the characteristics 460A of the previous frame 450A.

[0139] Method 800 in Figure 8 allows the processor 402 to reduce the latency associated with adjusting the exposure time 584B. For example, instead of having a delay between when the target exposure time 584B (or other target exposure parameter) is determined and when a frame 450B with the target exposure time 584B is read from the image sensor 406, Method 800 can reduce (or eliminate) the delay so that the frame 450B with the target exposure time 584B is read during the same frame period 550B in which the target exposure time 584B is determined. By reducing latency as described above, Method 800 can reduce the number of overexposed frames captured and the number of underexposed frames captured, which in turn can improve memory capacity.

[0140] Referring to Figure 9, a specific exemplary example of method 900 for reducing automatic exposure latency is shown. Method 900 can be implemented by various components of system 400 in Figure 4. In particular, one or more operations associated with method 900 can be implemented by the processor 402 of system 400.

[0141] Method 900 includes, in block 902, having the processor receive a first pre-exposure frame from the image sensor during a first frame period associated with the capture of the first frame. Referring, for example, to Figures 4 and 6, the processor 402 may receive a pre-exposure frame 452 from the image sensor 406 during a frame period 650B associated with the capture of frame 450B.

[0142] Method 900 also includes, in block 904, the processor determining a first setting of exposure parameters for a first frame captured by the image sensor. The first setting of exposure parameters is determined at least in part on the characteristics of a first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. Referring to Figures 4 and 6, for example, the processor may determine a setting 432 of exposure parameters 430 for a frame 450B captured by the image sensor 406. The setting 432 of exposure parameters 430 is determined at least in part on the pre-exposure frame characteristics 462 of a pre-exposure frame 452, and the setting 432 of exposure parameters 430 is determined during the frame period 650B.

[0143] Method 900 also includes, in block 906, having the processor initiate a first frame exposure operation based on a first setting of exposure parameters. During the first frame exposure operation, the image sensor captures a first frame during a first frame period. For example, processor 402 may initiate a frame exposure operation based on a setting 432 of exposure parameter 430. During the frame exposure operation, image sensor 406 captures a frame 450B during a frame period 650B.

[0144] According to one implementation of Method 900, a first pre-exposure frame 452 is captured by the image sensor 406 during a first frame period 650B. The first pre-exposure frame 452 may be generated in response to sampling a subset of image pixels on the image sensor 406. The subset of image pixels may be associated with a particular region of interest on the image sensor 406, or it may be distributed across the image sensor 406. The particular region of interest may correspond to a rectangular area adjacent to the central section of the image sensor 406, one or more rows of pixels, one or more columns of pixels, and so on.

[0145] According to one implementation, method 900 also includes determining a first setting of exposure parameters based on an external input. For example, referring to Figures 4 and 7, the processor 402 can determine the setting 432 of exposure parameters 430 for frame 450B based on the external input 790 in addition to the pre-exposure frame characteristics 462 of the pre-exposure frame 452. The external input 790 may be based on external sensor data from one or more other sensors. For example, according to some implementations, the external sensor data may correspond to image sensor data or environmental sensor data.

[0146] Method 900 in Figure 9 allows the processor 402 to reduce the latency associated with adjusting the exposure time 684B. For example, instead of having a delay between when the target exposure time 684B (or other target exposure parameter) is determined and when a frame 450B with the target exposure time 684B is read from the image sensor 406, Method 900 can reduce (or eliminate) the delay so that the frame 450B with the target exposure time 684B is read during the same frame period 650B in which the target exposure time 684B is determined. By reducing the latency as described above, Method 900 can reduce the number of overexposed frames captured and the number of underexposed frames captured, which in turn can improve memory capacity.

[0147] This disclosure is not limited to the specific embodiments described herein, and the specific embodiments are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope of this disclosure. In addition to the methods and apparatus enumerated herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0148] The above detailed description, with reference to the accompanying drawings, illustrates the various features and functions of the disclosed systems, devices, and methods. In the drawings, unless the context otherwise indicates, similar symbols typically represent similar components identically. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized and other modifications may be made without departing from the scope of the subject matter presented herein. It will be readily apparent that the aspects of this disclosure generally described herein and illustrated in the drawings can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly intended.

[0149] In the figures and in any or all of the message flow diagrams, scenarios, and flowcharts considered herein, each step, block, action, and / or communication may represent the processing and / or transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, actions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in an order different from those shown or discussed, such as substantially simultaneously or in reverse order, depending on the relevant functions. Furthermore, more or fewer blocks and / or actions may be used in any of the message flow diagrams, scenarios, and flowcharts considered herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other in part or as a whole.

[0150] Steps, blocks, or operations corresponding to the processing of information may correspond to a network of circuits that can be configured to perform a specific logical function of the method or technique described herein. Alternatively or additionally, steps or blocks corresponding to the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions executable by the processor for performing a specific logical operation or operation in the method or technique. The program code and / or associated data may be stored in any type of computer-readable medium, such as a memory device including RAM, a disk drive, a solid-state drive, or another storage medium.

[0151] Furthermore, one or more steps, blocks, or operations corresponding to information transmission may correspond to information transmission between software modules and / or hardware modules within the same physical device. However, other information transmissions may be between software modules and / or hardware modules in various physical devices.

[0152] The specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each of the elements shown in the given figures. Furthermore, some of the illustrated elements may be combined or omitted. In addition, exemplary embodiments may include elements not illustrated in the figures.

[0153] Various aspects and embodiments are disclosed herein, but other aspects and embodiments will be obvious to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to limit, and the true scope is indicated by the following claims.

Claims

1. It is a system, Image sensor and, The image sensor is coupled to a processor, and the processor is The system is configured to receive a first pre-exposure frame from the image sensor during a first frame period associated with capturing a first frame, wherein the first pre-exposure frame is a frame prior to the first frame, generated in response to sampling a subset of image pixels on the image sensor during the first frame period, and the subset of image pixels is distributed across the image sensor, and The system is configured to determine a first setting of exposure parameters for the first frame captured by the image sensor, wherein the first setting of exposure parameters is determined at least in part based on the characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. The image sensor is configured to perform a first frame exposure operation based on the first setting of the exposure parameters, A system in which, during the first frame exposure operation, the image sensor captures the first frame during the first frame period.

2. The system according to claim 1, wherein the subset of image pixels is associated with a region of interest on the image sensor.

3. The system according to claim 1, wherein the exposure parameters correspond to aperture size, exposure time, analog gain, or digital gain.

4. The system according to claim 1, wherein the processor is further configured to determine the first setting of the exposure parameters based on an external input.

5. It is a method, The processor receives a first pre-exposure frame from the image sensor during a first frame period associated with capturing a first frame, wherein the first pre-exposure frame is a frame prior to the first frame, generated in response to sampling a subset of image pixels on the image sensor during the first frame period, and the subset of image pixels is distributed across the image sensor. The processor determines a first setting of exposure parameters for the first frame captured by the image sensor, wherein the first setting of exposure parameters is determined at least in part based on the characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. A method comprising: the processor initiating a first frame exposure operation based on the first setting of the exposure parameters, wherein during the first frame exposure operation, the image sensor captures the first frame during the first frame period.

6. The method according to claim 5, wherein the subset of image pixels is associated with a region of interest on the image sensor.

7. The method according to claim 5, wherein the exposure parameters correspond to aperture size, exposure time, analog gain, or digital gain.

8. The method according to claim 5, wherein the processor is further configured to determine the first setting of the exposure parameter based on an external input.

9. When executed by the processor, the processor will Receiving a first pre-exposure frame from an image sensor during a first frame period associated with capturing a first frame, wherein the first pre-exposure frame is a frame prior to the first frame, generated in response to sampling a subset of image pixels on the image sensor during the first frame period, and the subset of image pixels is distributed across the image sensor. Determining a first setting of exposure parameters for the first frame captured by the image sensor, wherein the first setting of exposure parameters is determined at least in part based on the characteristics of the first pre-exposure frame, and the first setting of exposure parameters is determined during the first frame period. The first frame exposure operation is initiated based on the first setting of the exposure parameters, and during the first frame exposure operation, the image sensor captures the first frame during the first frame period. A non-temporary computer-readable medium containing instructions that cause an action to be performed.

10. The non-temporary computer-readable medium according to claim 9, wherein the subset of image pixels is associated with a region of interest on the image sensor.

11. The non-temporary computer-readable medium according to claim 9, wherein the exposure parameters correspond to aperture size, exposure time, analog gain, or digital gain.

12. The non-temporary computer-readable medium according to claim 9, wherein the processor is further configured to determine the first setting of the exposure parameters based on an external input.

Images