Method and system for camera direction control

Abstract

Methods and systems for camera direction control are provided. Example embodiments relate to taking images at certain predetermined angles so as to have consistent exposure between images. An example embodiment includes a method. The method includes determining, using a lidar device, light intensity information of a surrounding environment of the lidar device. The light intensity information includes a plurality of angles within an exposure threshold range. The method also includes determining a rotation time associated with each angle within the exposure threshold range. Further, the method includes determining a plurality of target image times based on the rotation time associated with each angle within the exposure threshold range. Moreover, the method includes capturing, by a camera system, a plurality of images at the plurality of target image times.

Description

Technical Field

[0001] The embodiments described herein may relate to methods for controlling camera orientation by adjusting frame time to avoid inconsistent exposure. Example embodiments relate to camera systems programmed to capture images at certain predetermined yaw and / or elevation angles to achieve consistent exposure between images. Background Technology

[0002] Unless otherwise stated herein, the materials described in this section are not prior art to the claims of this application and are not acknowledged as prior art simply because they are included in this section.

[0003] Cameras and image sensors are devices used to capture images of a scene. Some cameras (e.g., film cameras, etc.) capture images on film using chemical methods. Other cameras (e.g., digital cameras, etc.) capture image data using electrical methods (e.g., using charge-coupled devices (CCDs), complementary metal-oxide-semiconductor (CMOS) sensors, etc.). Images captured by cameras can be analyzed to determine their content. For example, a processor can execute machine learning algorithms to identify objects in a scene based on a library of previously categorized objects, including their shape, color, size, etc. (For example, such machine learning algorithms can be applied to computer vision in robotics or other applications).

[0004] Cameras can possess a variety of features that distinguish them from one another. For example, a camera and / or the images captured by it can be identified by values ​​such as aperture size, f-number, exposure time, shutter speed, depth of field, focal length, ISO sensitivity (or gain), pixel size, sensor resolution, and exposure distance. These features can be based on lenses, image sensors, and / or additional aspects of the camera. Furthermore, these features can also be adjustable within a single camera (e.g., the aperture of a lens on the camera can be adjusted between frames, etc.).

[0005] In addition, cameras can be used in LiDAR equipment applications to detect any dust, water, or other debris on the window surrounding the LiDAR. Dust, water, and other debris can also be referred to as "dirt" and can negatively affect LiDAR readings. Among other artifacts, dirt on the LiDAR aperture can cause range degradation, inaccurate ranging, and blurring artifacts in the point cloud due to stray light. Some systems involve cameras mounted on the LiDAR equipment. A camera that images the surrounding window of the LiDAR can be mounted inside the LiDAR equipment. In this way, the camera can be used to detect any obstructions (water, dust, debris, etc.) on the LiDAR equipment itself. Summary of the Invention

[0006] In one aspect, a method is provided. The method includes using a lidar device to determine light intensity information of the environment surrounding the lidar device. The light intensity information includes multiple angles within an exposure threshold range. The method also includes determining a rotation time associated with each angle within the exposure threshold range. Furthermore, the method includes determining multiple target image times based on the rotation time associated with each angle within the exposure threshold range. Additionally, the method includes capturing multiple images at the multiple target image times using a camera system.

[0007] On the other hand, a non-transitory computer-readable medium is provided having instructions stored thereon. When executed by a processor, the instructions cause the processor to perform a method. The method includes using a lidar device to determine light intensity information of the environment surrounding the lidar device. The light intensity information includes multiple angles within an exposure threshold range. The method also includes determining a rotation time associated with each angle within the exposure threshold range. Furthermore, the method includes determining multiple target image times based on the rotation time associated with each angle within the exposure threshold range. Additionally, the method includes capturing multiple images at the multiple target image times using a camera system.

[0008] In an additional aspect, an optical system is provided. The optical detector system includes optical components. The optical detector system also includes an image sensor configured to receive light from a scene via imaging optics. Furthermore, the optical detector system includes a controller configured to perform an imaging routine. The imaging routine includes determining light intensity information of the environment surrounding the lidar device using a lidar device. The light intensity information includes multiple angles within an exposure threshold range. The imaging routine also includes determining a rotation time associated with each angle within the exposure threshold range. Additionally, the imaging routine includes determining multiple target image times based on the rotation time associated with each angle within the exposure threshold range. Furthermore, the imaging routine includes capturing multiple images at the multiple target image times using a camera system.

[0009] These, and other aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description with appropriate reference to the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a functional block diagram illustrating a vehicle according to an example embodiment.

[0011] Figure 2A This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0012] Figure 2B This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0013] Figure 2CThis is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0014] Figure 2D This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0015] Figure 2E This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0016] Figure 3 This is a conceptual illustration of wireless communication between various computing systems associated with an autonomous vehicle, according to an example embodiment.

[0017] Figure 4A This is a block diagram of a system including a lidar device according to an example embodiment.

[0018] Figure 4B This is a block diagram of a lidar device according to an example embodiment.

[0019] Figure 5 This is an illustration of an optical system that interacts with the environment according to an example embodiment.

[0020] Figure 6A This is an illustration of background light at different angles relative to a lidar device, according to an example embodiment.

[0021] Figure 6B This is an illustration of background light at different angles relative to a lidar device, according to an example embodiment.

[0022] Figure 7 This is an illustration of a method according to an example embodiment.

[0023] Figure 8 This is an illustration of converting background light into exposure time according to an example embodiment. Detailed Implementation

[0024] This document considers exemplary methods and systems. Any exemplary embodiments or features described herein are not necessarily to be construed as preferred or advantageous over other embodiments or features. Furthermore, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a variety of different configurations, all of which are considered herein. Additionally, the specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in the given figures. Additionally, some of the elements shown may be combined or omitted. Furthermore, exemplary embodiments may include elements not shown in the figures.

[0025] The lidar device described herein may include one or more light emitters and one or more detectors for detecting light emitted by the one or more light emitters and reflected by one or more objects in the environment surrounding the lidar device. For example, the surrounding environment may include an internal or external environment, such as the interior or exterior of a building. Additionally or alternatively, the surrounding environment may include the interior of a vehicle. Furthermore, the surrounding environment may include the area around a road and / or adjacent areas on a road. Examples of objects in the surrounding environment include, but are not limited to, other vehicles, traffic signs, pedestrians, cyclists, road surfaces, buildings, terrain, etc. Additionally, the one or more light emitters may emit light into the local environment of the lidar system itself. For example, light emitted from the one or more light emitters may interact with the housing of the lidar system and / or surfaces or structures coupled to the lidar system. And in some cases, the lidar system may be mounted on a vehicle, in which case the one or more light emitters may be configured to emit light that interacts with objects in the vicinity of the vehicle. Furthermore, the light emitters may include fiber optic amplifiers, laser diodes, light-emitting diodes (LEDs), and other possibilities.

[0026] A lidar system comprises multiple components for capturing images. For example, some lidar systems may include a dome to surround and protect other lidar components, and may also include a window for the lidar to operate outwards. In some cases, the window may be obscured, damaged, misaligned, etc., during manufacturing, assembly, or normal use. Cameras can be used in lidar applications to detect any dirt, water, or other debris (e.g., dust, dirt, mud, insects, or other types of organic or inorganic matter) on the lidar dome, or to detect damage to the lidar dome (e.g., breakage, fracture, etc.). Dust, water, and other debris can also be referred to as "dirt" and can negatively affect lidar readings. Some systems involve cameras mounted inside the lidar device that image the dome window surrounding the lidar. In this configuration, the camera can see any obstructions (water, dust, other debris, etc.) on the lidar itself. The camera can rotate with the lidar to detect when dirt obscures any part of the lidar window. These cameras may be referred to as cameras, dirt cameras, or obstruction detection cameras.

[0027] One technique currently used during operation is for the camera to capture as many images as possible as the system rotates. Therefore, images are taken from different angles. However, depending on the environment around the vehicle, images taken at certain yaw and elevation angles may be overexposed or underexposed. For example, consider a car leaving a tunnel on a sunny day. When the camera is pointing backward into the tunnel, the ambient lighting is dim, so a longer exposure time should be used to avoid underexposed images. When the camera is pointing forward into the sun, a shorter exposure time should be used to avoid overexposed images. Overexposed and underexposed images cannot be used for any further calculations and are therefore unusable. While automatic exposure can be used to address exposure in images, it should be noted that there may still be areas in the field of view that result in saturation.

[0028] The example embodiments presented herein provide a camera system programmed to capture images at certain predetermined yaw and / or elevation angles to ensure consistent exposure between images. The camera system can be any type of light-sensitive instrument configured to capture images, such as a still camera, video camera, thermal imaging camera, stereo camera, night vision camera, etc. The method of this technique can determine which portions of the field of view should not be imaged using a lidar device. For example, when a vehicle is in operation, the associated lidar device (which may have a higher dynamic range than an occlusion detection camera) can collect information from the surrounding scene. The surrounding scene can include an internal or external environment, such as inside a building or tunnel, or outside a building. Additionally or alternatively, the surrounding scene can include the area around a road and / or adjacent areas on the road. Examples of objects in the surrounding environment include, but are not limited to, other vehicles, traffic signs, pedestrians, cyclists, road surfaces, buildings, terrain, etc. The associated lidar device can collect light intensity information from the surrounding scene within a fixed-size yaw sector. This intensity information can indicate the brightness of the surrounding scene within the fixed-size yaw sector (e.g., the brightness of the solar background in the surrounding scene, etc.). Brightness measurements can be used to determine the appropriate exposure time for an occlusion detection camera, which varies with the camera's yaw angle. For example, an average brightness can be calculated for each yaw sector. The "average brightness" can then be converted to "exposure time" using a pre-computed lookup table. In additional embodiments, any mapping (such as polynomial fitting) can be used to convert the average brightness to exposure time.

[0029] In this embodiment, capturing an image at a certain angle can be achieved by actively changing the camera's image capture time based on the current LiDAR yaw angle and / or elevation angle assigned to the system. The LiDAR software can track the LiDAR angle, and the camera system can take the angle into account and actively adjust the image timing so that the next image can be captured at the specified angle. Since the camera in this application is typically a streaming camera (i.e., a camera that can feed or stream images or videos to or via a computer network (such as the Internet) in real time), angle locking can be achieved by increasing or decreasing the blank lines associated with image readout. The blank lines do not contain any image data but can be parameters typically used by camera modules to change the frame rate per second. Specifically, adjusting the blank lines causes the image readout to take a longer or shorter amount of time, which then affects when the next image is captured. Increasing the number of lines will increase the camera readout time, thereby delaying the start time of the next camera frame. Decreasing the number of lines will decrease the camera readout time, thereby speeding up the start time of the next camera frame. In the example sensor model, an additional blank line can delay the next frame by 54.792 microseconds. In this way, by assuming a constant rate of lidar rotation, the angle at which an image is captured can be selected by manipulating the timing of the image capture. The image can then be used to determine whether the lidar window is obscured by dirt. In an alternative embodiment, a non-streaming camera can also be used to capture images at a predetermined angle by increasing or decreasing blank lines associated with image readout. The non-streaming camera may include at least one camera that stores multiple images that will be processed at a later time.

[0030] As described herein, some embodiments may include using the lidar device to determine the light intensity information of the surrounding environment when the lidar device rotates. The lidar device may include at least one highly sensitive photodiode to determine the average background brightness of the environment. The average background brightness may include the average exposure amount sensed by the highly sensitive photodiode. The light intensity information may also include multiple angles within an exposure threshold range. The exposure threshold range may be a range of desired exposure times.

[0031] Some embodiments may then include determining a rotation time associated with each angle within an exposure threshold range. The rotation time can be determined using a constant rotation rate of the lidar and a known angle currently facing the lidar device. Components of the lidar device can track the angle the lidar device is facing. Based on the rotation time associated with each angle within the exposure threshold range, some embodiments include determining multiple target image times. Target imaging times are determined to attempt to capture an image during the most desired exposure period. Embodiments then include capturing multiple images at multiple target image times using a camera system. The multiple images can then be used to determine whether the lidar device's window is obstructed.

[0032] The following description and accompanying drawings will illustrate the features of various exemplary embodiments. The embodiments provided are exemplary and are not intended to be limiting. Therefore, the dimensions of the drawings are not necessarily drawn to scale.

[0033] The example systems within the scope of this disclosure will now be described in more detail. The example systems can be implemented in or take the form of automobiles. Additionally, the example systems can also be implemented in or take the form of various vehicles, such as automobiles, trucks, motorcycles, buses, airplanes, helicopters, drones, lawnmowers, dump trucks, boats, submarines, all-terrain vehicles, snowmobiles, recreational vehicles, amusement park vehicles, farm equipment or vehicles, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, trams, golf carts, trains, handcarts, sidewalk transport vehicles, robotic equipment, etc. Other vehicles are also possible. Furthermore, in some embodiments, the example systems may not include a vehicle.

[0034] Now refer to the attached diagram, Figure 1 This is a functional block diagram illustrating an example vehicle 100, which can be configured to operate fully or partially in autonomous mode. More specifically, vehicle 100 can operate in autonomous mode without human interaction by receiving control commands from a computing system. As part of operating in autonomous mode, vehicle 100 can use sensors to detect and possibly identify objects in the surrounding environment to enable safe navigation. Additionally, the example vehicle 100 can operate in a partially autonomous (i.e., semi-autonomous) mode, wherein some functions of vehicle 100 are controlled by a human driver and some functions are controlled by the computing system. For example, vehicle 100 may also include a subsystem that enables the driver to control the operation of vehicle 100, such as steering, acceleration, and braking, while the computing system performs assistance functions such as lane departure warning / lane keeping assist or adaptive cruise control based on other objects in the surrounding environment (e.g., other vehicles).

[0035] As described herein, in partially autonomous driving modes, even when the vehicle assists with one or more driving operations (e.g., steering, braking, and / or acceleration to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), emergency braking, etc.), the human driver is expected to be aware of the situation around the vehicle and supervise the assisted driving operations. Here, even if the vehicle can perform all driving tasks in certain situations, the human driver is expected to be responsible for taking control as needed.

[0036] Although various systems and methods are described below in conjunction with autonomous vehicles for the sake of brevity and simplicity, these or similar systems and methods can be used in various driver assistance systems that do not reach the level of fully autonomous driving systems (i.e., partially autonomous driving systems). In the United States, the Society of Automotive Engineers (SAE) has defined different levels of autonomous driving operation to indicate how much or how little vehicle control is involved in driving, although different organizations in the United States or other countries may classify levels differently. More specifically, the disclosed systems and methods can be used in SAE Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver support. The disclosed systems and methods can be used in SAE Level 3 driver assistance systems that are capable of autonomous driving under restricted conditions (e.g., highways). Similarly, the disclosed systems and methods can be used in vehicles using SAE Level 4 autonomous driving systems that operate autonomously in most normal driving situations and require only occasional human intervention. In all such systems, accurate lane estimation can be performed autonomously (e.g., when the vehicle is in motion), without driver input or control, leading to improved reliability of vehicle positioning and navigation, as well as overall safety for autonomous, semi-autonomous, and other driver assistance systems. As previously mentioned, other organizations in the United States or other countries may classify levels of automated driving operations differently than the SAE does. The systems and methods disclosed herein can be used, but are not limited to, driver assistance systems defined by the automated driving operation levels of these other organizations.

[0037] like Figure 1 As shown, vehicle 100 may include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112 (also referred to as a computing system), a data storage device 114, and a user interface 116. In other examples, vehicle 100 may include more or fewer subsystems, each subsystem including multiple components. The subsystems and components of vehicle 100 may be interconnected in various ways. Additionally, in embodiments, the functionality of vehicle 100 described herein may be divided into additional functions or physical components, or combined into fewer functions or physical components. For example, control system 106 and computer system 112 may be combined into a single system to operate vehicle 100 according to various operations.

[0038] The propulsion system 102 may include one or more components operable to provide powered motion to the vehicle 100, and may include an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121, as well as other possible components. For example, the engine / motor 118 may be configured to convert the energy source 119 into mechanical energy, and may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine, as well as other possible options. For example, in some embodiments, the propulsion system 102 may include multiple types of engines and / or motors, such as gasoline engines and electric motors.

[0039] Energy source 119 represents a source of energy that can provide power, in whole or in part, to one or more systems of vehicle 100 (e.g., engine / motor 118). For example, energy source 119 may correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other electrical sources. In some embodiments, energy source 119 may include a combination of a fuel tank, battery, capacitor, and / or flywheel.

[0040] The transmission 120 can transmit mechanical power from the engine / motor 118 to the wheels / tires 121 of the vehicle 100 and / or other possible systems. Thus, the transmission 120 may include a gearbox, clutch, differential, and drive shaft, as well as other possible components. The drive shaft may include an axle connected to one or more wheels / tires 121.

[0041] In the example embodiment, the wheels / tires 121 of the vehicle 100 can have various configurations. For example, the vehicle 100 can exist as a unicycle, bicycle / motorcycle, tricycle, or four-wheeled car / truck, as well as other possible configurations. Thus, the wheels / tires 121 can be attached to the vehicle 100 in various ways and can be made of different materials (such as metal and rubber).

[0042] Sensor system 104 may include various types of sensors, such as a Global Positioning System (GPS) 122, an Inertial Measurement Unit (IMU) 124, a radar 126, a laser rangefinder / LIDAR 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, as well as other possible sensors. In some embodiments, sensor system 104 may also include sensors configured to monitor the internal systems of vehicle 100 (e.g., O2 monitor, fuel gauge, engine oil temperature, brake wear).

[0043] GPS 122 may include a transceiver operable to provide information about the position of vehicle 100 relative to the Earth. IMU 124 may be configured to use one or more accelerometers and / or gyroscopes and can sense changes in the position and orientation of vehicle 100 based on inertial acceleration. For example, when vehicle 100 is stationary or moving, IMU 124 can detect the pitch and yaw of vehicle 100.

[0044] Radar 126 may represent one or more systems configured to use radio signals to sense objects in the environment surrounding vehicle 100, including the speed and heading of the objects. Thus, radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, radar 126 may correspond to an installable radar system configured to acquire measurements of the environment surrounding vehicle 100.

[0045] The laser rangefinder / LIDAR 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components, and may operate in a coherent mode (e.g., using heterodyne detection) or an incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of the laser rangefinder / LIDAR 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes, etc.). In some examples, such photodetectors may be able to detect single photons (e.g., single-photon avalanche diodes (SPADs), etc.). Furthermore, such photodetectors may be arranged (e.g., via series electrical connections) in an array (e.g., as in a silicon photomultiplier (SiPM)). In some examples, one or more photodetectors are Geiger-mode operating devices, and the lidar includes sub-components designed for such Geiger-mode operation.

[0046] Camera 130 may include one or more devices (e.g., still camera or video camera, thermal imaging camera, stereo camera, night vision camera, etc.) configured to capture images of the environment surrounding vehicle 100.

[0047] Steering sensor 123 can sense the steering angle of vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, steering sensor 123 can measure the angle of the wheels of vehicle 100, such as detecting the angle of the wheels relative to the forward axis of vehicle 100. Steering sensor 123 may also be configured to measure a combination (or subset) of the angle of the steering wheel of vehicle 100, an electrical signal representing the angle of the steering wheel, and the angle of the wheels.

[0048] Throttle / brake sensor 125 can detect the position of the throttle or brake of vehicle 100. For example, throttle / brake sensor 125 can measure the angle of both the accelerator pedal (throttle) and the brake pedal, or it can measure an electrical signal that represents, for example, the angle of the accelerator pedal (throttle) and / or the angle of the brake pedal. Throttle / brake sensor 125 can also measure the angle of the throttle body of vehicle 100, which may include part of a modulated physical mechanism that supplies energy source 119 to engine / motor 118 (e.g., butterfly valve, carburetor). Furthermore, throttle / brake sensor 125 can measure the pressure of one or more brake pads on the rotor of vehicle 100, or a combination (or subset) of the angle of the accelerator pedal (throttle) and the brake pedal, an electrical signal representing the angle of the accelerator pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied by at least one brake pad to the rotor of vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure the pressure applied to a vehicle pedal (such as a throttle or brake pedal).

[0049] The control system 106 may include components configured to assist navigation of the vehicle 100, such as a steering unit 132, a throttle valve 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / path system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 is operable to adjust the direction of the vehicle 100, and the throttle valve 134 can control the operating rate of the engine / motor 118 to control the acceleration of the vehicle 100. The braking unit 136 can decelerate the vehicle 100, which may involve using friction to slow down the wheels / tires 121. In some embodiments, the braking unit 136 can convert the kinetic energy of the wheels / tires 121 into electrical current for subsequent use by one or more systems of the vehicle 100.

[0050] Sensor fusion algorithm 138 may include Kalman filters, Bayesian networks, or other algorithms capable of processing data from sensor system 104. In some embodiments, sensor fusion algorithm 138 may provide evaluations based on incoming sensor data, such as evaluations of individual objects and / or features, evaluations of specific situations, and / or evaluations of potential impacts within a given situation.

[0051] Computer vision system 140 may include hardware and software (e.g., a general-purpose processor, an application-specific integrated circuit (ASIC), volatile memory, non-volatile memory, one or more machine learning models, etc.) operable to process and analyze images in an effort to determine moving objects (e.g., other vehicles, pedestrians, cyclists, animals, etc.) and stationary objects (e.g., traffic lights, road boundaries, speed bumps, potholes, etc.). Thus, computer vision system 140 may use object recognition, structure-of-motion (SFM), video tracking, and other algorithms used in computer vision, such as object recognition, environment mapping, object tracking, and object rate estimation.

[0052] The navigation / path system 142 can determine the driving path of the vehicle 100, which may involve dynamically adjusting the navigation during operation. Thus, the navigation / path system 142 can use data from sensor fusion algorithm 138, GPS 122 and maps, as well as other sources, to navigate the vehicle 100. The obstacle avoidance system 144 can assess potential obstacles based on sensor data and enable the vehicle 100's systems to avoid or otherwise negotiate potential obstacles.

[0053] like Figure 1 As shown, vehicle 100 may also include peripheral devices 108, such as wireless communication system 146, touchscreen 148, microphone 150, and / or speaker 152. Peripheral devices 108 can provide users with controls or other elements to interact with user interface 116. For example, touchscreen 148 can provide information to the user of vehicle 100. User interface 116 can also accept input from the user via touchscreen 148. Peripheral devices 108 can also enable vehicle 100 to communicate with devices such as other vehicle equipment.

[0054] The wireless communication system 146 can wirelessly communicate with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication, such as Code Division Multiple Access (CDMA), Evolved Data Optimization (EVDO), Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or cellular communication (such as 4G Global Microwave Access Interoperability (WiMAX) or Long Term Evolution (LTE)). Alternatively, the wireless communication system 146 can use... Or other possible connections for communication with a wireless local area network (WLAN). For example, the wireless communication system 146 may also communicate directly with the device using an infrared link, BLUETOOTH, or ZIGBEE. In the context of this disclosure, other wireless protocols, such as those used in various vehicle communication systems, are also possible. For example, the wireless communication system 146 may include one or more dedicated short-range communication (DSRC) devices, which may include public and / or private data communication between the vehicle and / or roadside station.

[0055] The carrier 100 may include a power source 110 for powering components. In some embodiments, the power source 110 may include a rechargeable lithium-ion battery or a lead-acid battery. For example, the power source 110 may include one or more batteries configured to provide electrical energy. The carrier 100 may also use other types of power sources. In an example embodiment, the power source 110 and the energy source 119 may be integrated into a single energy source.

[0056] The vehicle 100 may also include a computer system 112 to perform operations such as those described herein. Thus, the computer system 112 may include at least one processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored in a non-transitory computer-readable medium (such as data storage device 114). In some embodiments, the computer system 112 may represent a plurality of computing devices that can be used to control various components or subsystems of the vehicle 100 in a distributed manner.

[0057] In some embodiments, the data storage device 114 may include instructions 115 (e.g., program logic) that can be executed by the processor 113 to perform various functions of the vehicle 100, including those described above. Figure 1 The functions described. The data storage device 114 may also contain additional instructions, including instructions to send data to, receive data from, interact with, and / or control one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0058] In addition to command 115, data storage device 114 can store data such as road maps, route information, and other information. This information can be used by vehicle 100 and computer system 112 while vehicle 100 is operating in autonomous, semi-autonomous, and / or manual modes.

[0059] Vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of vehicle 100. User interface 116 may control or enable control over the content and / or layout of interactive images that may be displayed on touchscreen 148. Furthermore, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, touchscreen 148, microphone 150, and speaker 152.

[0060] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., propulsion system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 can utilize input from sensor system 104 to estimate the outputs generated by propulsion system 102 and control system 106. Depending on the embodiment, computer system 112 can be operable to monitor many aspects of vehicle 100 and its subsystems. In some embodiments, computer system 112 can disable some or all functions of vehicle 100 based on signals received from sensor system 104.

[0061] The components of vehicle 100 can be configured to operate in a manner interconnected with other components, either within or outside their respective systems. For example, in an example embodiment, camera 130 can capture multiple images that may represent information about the state of the environment surrounding vehicle 100 operating in autonomous mode. The state of the environment may include parameters of the road on which the vehicle is operating. For example, computer vision system 140 is capable of identifying slope (grade) or other features based on multiple images of the road. Additionally, a combination of GPS 122 and features identified by computer vision system 140 can be used with map data stored in data storage device 114 to determine specific road parameters. Furthermore, radar 126 and / or laser rangefinder / LIDAR 128 and / or some other environmental mapping, ranging, and / or positioning sensor systems can also provide information about the vehicle's environment.

[0062] In other words, the combination of various sensors (which may be referred to as input indication and output indication sensors) and computer system 112 can interact to provide indications of inputs provided for controlling the vehicle, or indications of the surroundings of the vehicle.

[0063] In some embodiments, computer system 112 can make determinations about various objects based on data provided by other radio systems. For example, vehicle 100 may have lasers or other optical sensors configured to sense objects in the vehicle's field of view. Computer system 112 can use the outputs from various sensors to determine information about objects in the vehicle's field of view, and can determine distance and orientation information to various objects. Computer system 112 can also determine whether an object is desired or undesirable based on the outputs from various sensors.

[0064] although Figure 1Various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage device 114, and user interface 116) are shown as integrated into vehicle 100; however, one or more of these components may be installed or associated separately from vehicle 100. For example, data storage device 114 may exist partially or entirely separate from vehicle 100. Therefore, vehicle 100 may be provided in the form of device elements that can be positioned separately or together. The device elements constituting vehicle 100 may be communicatively coupled together in a wired and / or wireless manner.

[0065] Figures 2A to 2E An example vehicle 200 (e.g., a fully autonomous vehicle or a semi-autonomous vehicle, etc.) is shown, which may include reference Figure 1 Combining some or all of the functions described for vehicle 100. Although for illustrative purposes, vehicle 200 in… Figures 2A to 2E The vehicle is shown as a truck with side mirrors 216, but this disclosure is not limited thereto. For example, vehicle 200 may represent a truck, automobile, semi-trailer truck, motorcycle, golf cart, off-road vehicle, agricultural vehicle, or any other vehicle described elsewhere herein (e.g., bus, boat, airplane, helicopter, drone, lawnmower, dump truck, submarine, all-terrain vehicle, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment or vehicle, warehouse equipment or vehicle, factory equipment or vehicle, tram, train, handcart, sidewalk transport vehicle, and robotic equipment, etc.).

[0066] Example vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras, etc.), one or more lidar systems, one or more radar systems, one or more rangefinders, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones, sonar devices, etc.), or one or more other sensors configured to sense information about the environment surrounding vehicle 200. In other words, any sensor system now known or created hereafter can be coupled to vehicle 200 and / or can be utilized in conjunction with various operations of vehicle 200. As an example, a lidar system can be used for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of vehicle 200. Additionally, sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may represent combinations of sensors described herein (e.g., one or more lidar and radar; one or more lidar and camera; one or more cameras and radar; one or more lidar, camera and radar; etc.).

[0067] Note Figures 2A to 2E The number, location, and type of sensor systems depicted (e.g., 202, 204, etc.) are intended as non-limiting examples of the location, number, and type of such sensor systems for autonomous or semi-autonomous vehicles. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., adapted to vehicle size, shape, aerodynamics, fuel economy, aesthetics, or other conditions to reduce costs, adapt to special environments or application situations, etc.). For example, sensor systems (e.g., 202, 204, etc.) can be arranged in various other locations on the vehicle (e.g., at location 216, etc.) and can have a field of view corresponding to the interior and / or surrounding environment of the vehicle 200.

[0068] Sensor system 202 may be mounted on top of vehicle 200 and may include one or more sensors configured to detect information about the environment surrounding vehicle 200 and output indications of that information. For example, sensor system 202 may include any combination of cameras, radar, lidar, rangefinders, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones, sonar devices, etc.). Sensor system 202 may include one or more movable mounts operable to adjust the orientation of one or more sensors in sensor system 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to obtain information from every direction around vehicle 200. In another embodiment, the movable mount of sensor system 202 may be movable in a scanning manner within a specific angular and / or azimuth and / or elevation range. Sensor system 202 may be mounted on the top of the vehicle roof, although other mounting locations are also possible.

[0069] Additionally, the sensors of sensor system 202 can be distributed at different locations and do not need to be collocated at a single location. Furthermore, each sensor of sensor system 202 can be configured to be moved or scanned independently of other sensors in sensor system 202. Additionally or alternatively, multiple sensors can be installed at one or more of sensor locations 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, two lidar devices may be installed at the sensor locations, and / or one lidar device and one radar device may be installed at the sensor locations.

[0070] One or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more lidar sensors. For example, a lidar sensor may include multiple light emitter devices arranged within an angular range relative to a given plane (e.g., the xy plane, etc.). For example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot about an axis perpendicular to the given plane (e.g., the z-axis, etc.) to illuminate the environment surrounding vehicle 200 with light pulses. Information about the surrounding environment can be determined based on various aspects of the detected reflected light pulses (e.g., time of flight, polarization, intensity, etc.).

[0071] In the example embodiment, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide corresponding point cloud information that may be related to physical objects in the surrounding environment of vehicle 200. While vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are shown to include certain features, it will be understood that other types of sensor systems are contemplated within the scope of this disclosure. Furthermore, the example vehicle 200 may include combinations of... Figure 1 Any component described in vehicle 100.

[0072] In the example configuration, one or more radars may be positioned on vehicle 200. Similar to radar 126 described above, one or more radars may include antennas configured to transmit and receive radio waves (e.g., electromagnetic waves with frequencies between 30 Hz and 300 GHz). These radio waves can be used to determine the distance and / or velocity of one or more objects in the environment surrounding vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some examples, one or more radars may be positioned near the rear of vehicle 200 (e.g., sensor systems 208, 210, etc.) to actively scan the environment near the rear of vehicle 200 to detect the presence of radio-reflecting objects. Similarly, one or more radars may be positioned near the front of vehicle 200 (e.g., sensor systems 212, 214, etc.) to actively scan the environment near the front of vehicle 200. The radar can be located, for example, in a position suitable for illuminating the area including the forward movement path of the vehicle 200 without being obstructed by other features of the vehicle 200. For example, the radar can be embedded in or near the front bumper, headlights, hood, and / or engine hood. Furthermore, one or more additional radars can be positioned to actively scan the sides and / or rear of the vehicle 200 to detect the presence of radio-reflecting objects, such as by including such devices in or near the rear bumper, side panels, sill plates, and / or chassis.

[0073] Vehicle 200 may include one or more cameras. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may include one or more cameras. The camera may be a photosensitizing instrument, such as a still camera, video camera, thermal imaging camera, stereo camera, night vision camera, etc., configured to capture multiple images of the surrounding environment of vehicle 200. For this purpose, the camera may be configured to detect visible light and may additionally or alternatively be configured to detect light from other parts of the spectrum, such as infrared or ultraviolet light. The camera may be a two-dimensional detector and may optionally have sensitivity in a three-dimensional spatial range. In some embodiments, the camera may include, for example, a distance detector configured to generate a two-dimensional image indicating the distance from the camera to multiple points in the surrounding environment. For this purpose, the camera may use one or more distance detection techniques. For example, the camera may provide distance information by using structured light technology, in which vehicle 200 illuminates objects in the surrounding environment with a predetermined light pattern (such as a grid or checkerboard pattern), and the camera is used to detect reflections of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, the vehicle 200 can determine the distance to a point on an object. The predetermined light pattern may include infrared light, or radiation of other suitable wavelengths for such measurement. In some examples, the camera may be mounted inside the windshield of the vehicle 200. Specifically, the camera may be positioned to capture images from a forward line of sight relative to the orientation of the vehicle 200. Other mounting positions and viewing angles of the camera may also be used inside or outside the vehicle 200. Furthermore, the camera may have associated optics operable to provide an adjustable field of view. Further still, the camera may be mounted to the vehicle 200 using a movable bracket to vary the camera's pointing angle, such as via a rocking / tilting mechanism.

[0074] Vehicle 200 may also include one or more acoustic sensors (e.g., one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors, etc.) for sensing the surrounding environment of vehicle 200. The acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, microelectromechanical systems (MEMS) microphones, etc.) used to sense sound waves (i.e., pressure differences) in a fluid (e.g., air) in the surrounding environment of vehicle 200. Such acoustic sensors can be used to identify sounds in the surrounding environment (e.g., sirens, human speech, animal sounds, alarms, etc.), and the control strategy of vehicle 200 can be based on these sounds. For example, if the acoustic sensor detects a siren (e.g., a mobile siren, a fire truck siren, etc.), vehicle 200 may decelerate and / or navigate to the edge of a road.

[0075] Although Figures 2A to 2E Not shown, but vehicle 200 may include a wireless communication system (e.g., similar to...). Figure 1 Wireless communication systems 146 and / or other than Figure 1 In addition to the wireless communication system 146, other wireless communication systems may also be included. The wireless communication system may include a wireless transmitter and receiver, which may be configured to communicate with devices external to or internal to the vehicle 200. Specifically, the wireless communication system may include a transceiver configured to communicate with other vehicles and / or computing devices, such as in a vehicle communication system or a road station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other proposed communication standards for intelligent transportation systems.

[0076] In addition to or in place of the components shown, vehicle 200 may include one or more other components. Additional components may include electrical or mechanical functions.

[0077] The control system of vehicle 200 can be configured to control vehicle 200 according to one of a variety of possible control strategies. The control system can be configured to receive information from sensors coupled to vehicle 200 (on or off vehicle 200), modify the control strategy (and associated driving behavior) based on that information, and control vehicle 200 according to the modified control strategy. The control system can also be configured to monitor information received from sensors and continuously evaluate driving conditions; and can also be configured to modify the control strategy and driving behavior based on changes in driving conditions. For example, the route taken by the vehicle from one destination to another can be modified based on driving conditions. Additionally or alternatively, speed, acceleration, steering angle, following distance (i.e., distance to the vehicle in front of the current vehicle), lane selection, etc., can be modified according to changes in driving conditions.

[0078] Figure 3 This is a conceptual illustration of wireless communication between various computing systems associated with a vehicle, according to an example embodiment. Specifically, wireless communication can occur between remote computing system 302 and vehicle 200 via network 304. Wireless communication can also occur between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0079] Vehicle 200 can correspond to various types of vehicles capable of transporting passengers or objects between different locations, and can take any one or more forms of vehicles discussed above. In some cases, vehicle 200 can operate in autonomous or semi-autonomous mode, which allows the control system to use sensor measurements to safely navigate vehicle 200 between destinations. When operating in autonomous or semi-autonomous mode, vehicle 200 can navigate with or without passengers. As a result, vehicle 200 can pick up and drop off passengers between desired destinations.

[0080] The remote computing system 302 can represent any type of device associated with remote assistance technology, including but not limited to those described herein. In the examples, the remote computing system 302 can represent any type of device configured to: (i) receive information related to vehicle 200, (ii) provide an interface through which a human operator can sequentially perceive the information and input a response related to that information, and (iii) send the response to vehicle 200 or other devices. The remote computing system 302 can take various forms, such as a workstation, desktop computer, laptop computer, tablet computer, mobile phone (e.g., smartphone), and / or server. In some examples, the remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0081] The remote computing system 302 may include one or more subsystems and components similar to or identical to those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface including input / output devices such as a touchscreen and speakers. Other examples are also possible.

[0082] Network 304 represents the infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0083] In this example, the location of the telecomputing system 302 can vary. For instance, the telecomputing system 302 may be located remotely from the vehicle 200, with wireless communication via network 304. In another example, the telecomputing system 302 may correspond to a computing device within the vehicle 200, separate from the vehicle 200, but whose human operator can interact with the passengers or driver of the vehicle 200. In some examples, the telecomputing system 302 may be a computing device with a touchscreen operable by the passengers of the vehicle 200.

[0084] In some embodiments, the operations performed by the remote computing system 302 as described herein may be additionally or alternatively performed by the vehicle 200 (i.e., by any of the vehicle 200's systems or subsystems). In other words, the vehicle 200 may be configured to provide a remote assistance mechanism that allows the vehicle's driver or passengers to interact.

[0085] Server computing system 306 can be configured to wirelessly communicate with remote computing system 302 and vehicle 200 via network 304 (or possibly directly with remote computing system 302 and / or vehicle 200). Server computing system 306 can represent any computing device configured to receive, store, determine, and / or transmit information related to vehicle 200 and its remote assistance. Thus, server computing system 306 can be configured to perform any of the operations(s) or a portion thereof, which are described herein as being performed by remote computing system 302 and / or vehicle 200. Some embodiments of wireless communication related to remote assistance may utilize server computing system 306, while others may not.

[0086] Server computing system 306 may include one or more subsystems and components similar to or the same as those of remote computing system 302 and / or vehicle 200, such as processors configured to perform the various operations described herein, and wireless communication interfaces for receiving and providing information to remote computing system 302 and vehicle 200.

[0087] The various systems described above can perform a variety of operations. These operations and related characteristics will now be described.

[0088] Based on the above discussion, a computing system (e.g., a remote computing system 302, a server computing system 306, or a computing system local to the vehicle 200) can operate to use cameras to capture images of the autonomous vehicle's surrounding environment. Generally, at least one computing system will be able to analyze the images and potentially control the autonomous or semi-autonomous vehicle.

[0089] In some embodiments, to facilitate autonomous or semi-autonomous operation, the vehicle (e.g., vehicle 200) can receive data representing objects in its surrounding environment in various ways. Sensor systems on the vehicle can provide environmental data representing objects in the surrounding environment. For example, the vehicle can have various sensors, including cameras, radar units, laser rangefinders, microphones, radio units, and other sensors. Each of these sensors can transmit environmental data about the information received by each respective sensor to a processor within the vehicle.

[0090] In one example, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have more than one camera positioned in different orientations. Additionally, in some embodiments, the camera may be able to move to capture images and / or video in different directions. The camera may be configured to store the captured images and video in memory for later processing by the vehicle's processing system. The captured images and / or video may be environmental data. Furthermore, the camera may include an image sensor as described herein.

[0091] In another example, the radar unit can be configured to emit electromagnetic signals reflected by various objects near the vehicle and then capture the electromagnetic signals reflected from the objects. The captured reflected electromagnetic signals enable the radar system (or processing system) to make various determinations about the objects reflecting the electromagnetic signals. For example, the distance and position to the various reflecting objects can be determined. In some embodiments, the vehicle may have more than one radar in different orientations. The radar system can be configured to store the captured information in a memory for later processing by the vehicle's processing system. The information captured by the radar system may be environmental data.

[0092] In another example, a laser rangefinder can be configured to emit electromagnetic signals (e.g., infrared light, such as infrared light from a gas or diode laser, or other possible light sources) that will be reflected by a target object near the vehicle. The laser rangefinder is able to capture the reflected electromagnetic (e.g., infrared light, etc.) signals. The captured reflected electromagnetic signals allow the ranging system (or processing system) to determine the distance to various objects. The laser rangefinder can also determine the velocity or speed of the target object and store it as environmental data.

[0093] Additionally, in this example, the microphone can be configured to capture audio of the environment surrounding the vehicle. The sounds captured by the microphone may include emergency vehicle sirens and sounds from other vehicles. For example, the microphone could capture the sound of an ambulance, fire truck, or police car siren. The processing system could be able to identify the captured audio signal as indicating an emergency vehicle. In another example, the microphone could capture the exhaust sound of another vehicle, such as a motorcycle. The processing system could be able to identify the captured audio signal as indicating a motorcycle. The data captured by the microphone can form part of the environmental data.

[0094] In another example, the radio unit can be configured to transmit an electromagnetic signal, which may take the form of a Bluetooth signal, an 802.11 signal, and / or other radio technology signals. The first electromagnetic radiation signal can be transmitted via one or more antennas located in the radio unit. Furthermore, the first electromagnetic radiation signal can be transmitted using one of many different radio signal transmission modes. However, in some embodiments, it is desirable to transmit the first electromagnetic radiation signal in a signal transmission mode that requests a response from a device located near an autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on the communication response returned to the radio unit and use this communication information as part of environmental data.

[0095] In some embodiments, the processing system may be able to combine information from various sensors to further determine the vehicle's surroundings. For example, the processing system may combine data from radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, the processing system may use other combinations of sensor data to make determinations about the surroundings.

[0096] When operating in autonomous (or semi-autonomous) mode, the vehicle can control its operation with little or no human input. For example, a human operator can input an address into the vehicle, and the vehicle can then be able to travel to the designated destination without further human input (e.g., the person does not need to steer or engage the brake / accelerator pedals, etc.). Furthermore, when the vehicle operates autonomously or semi-autonomously, the sensor system can receive environmental data. The vehicle's processing system can modify the vehicle's control based on the environmental data received from various sensors. In some examples, the vehicle can change its speed in response to environmental data from various sensors. The vehicle can change its speed to avoid obstacles, comply with traffic regulations, etc. When the processing system in the vehicle identifies an object in its vicinity, the vehicle can be able to change its speed or otherwise alter its motion.

[0097] When a vehicle detects an object but lacks high confidence in the detection, it may request a human operator (or a more powerful computer) to perform one or more remotely assisted tasks, such as (i) confirming whether the object actually exists in the surrounding environment (e.g., whether a stop sign actually exists or does not actually exist), (ii) confirming whether the vehicle's identification of the object is correct, (iii) correcting the identification if incorrect, and / or (iv) providing supplementary instructions (or modifying current instructions) to the autonomous or semi-autonomous vehicle. Remotely assisted tasks may also include instructions from the human operator to control the vehicle's operation (e.g., instructing the vehicle to stop at the stop sign if the human operator determines the object is one), although in some scenarios the vehicle itself may control its own operation based on feedback from the human operator related to the object's identification.

[0098] To facilitate this, the vehicle can analyze environmental data representing objects in the surrounding environment to identify at least one object with a detection confidence level below a threshold. A processor within the vehicle can be configured to detect various objects in the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor can be configured to detect objects that may be important for the vehicle's identification. Such objects may include pedestrians, cyclists, street signs, other vehicles, indicator signals on other vehicles, and various other objects detected in the captured environmental data.

[0099] Detection confidence indicates the likelihood that a identified object is correctly identified or exists in the surrounding environment. For example, a processor may perform object detection on objects within image data in received environmental data and determine that at least one object has a detection confidence below a threshold, based on the inability to identify objects with a detection confidence above a threshold. If the result of object detection or object recognition is uncertain, the detection confidence may be low or below a set threshold.

[0100] Depending on the source of the environmental data, the vehicle can detect objects in the surrounding environment in various ways. In some embodiments, the environmental data may come from a camera and may be image or video data. In other embodiments, the environmental data may come from a lidar unit. The vehicle can analyze the captured image or video data to identify objects in the image or video data. The method and apparatus can be configured to monitor image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle can be configured to identify objects in the surrounding environment based on radar, audio, or other data.

[0101] In some embodiments, the technology used by the vehicle to detect objects can be based on a set of known data. For example, data related to environmental objects can be stored in a memory located in the vehicle. The vehicle can compare the received data with the stored data to determine objects. In other embodiments, the vehicle can be configured to determine objects based on the context of the data. For example, street signs associated with buildings are typically orange. Therefore, the vehicle can be configured to detect orange objects located near the side of a road as street signs associated with buildings. Additionally, when the vehicle's processing system detects objects in the captured data, it can also calculate a confidence level for each object.

[0102] Furthermore, the vehicle may also have a confidence threshold. The confidence threshold can vary depending on the type of object being detected. For example, a lower confidence threshold might be used for objects that may require a rapid response from the vehicle (such as brake lights on another vehicle). However, in other embodiments, the confidence threshold may be the same for all detected objects. When the confidence associated with a detected object is greater than the confidence threshold, the vehicle can assume that the object has been correctly identified and responsively adjust the vehicle's control based on that assumption.

[0103] The vehicle's actions may change when the confidence level associated with a detected object is less than a confidence threshold. In some embodiments, the vehicle may react as if the detected object were present, even with a low confidence level. In other embodiments, the vehicle may react as if the detected object were not present.

[0104] When the vehicle detects an object in the surrounding environment, it can also calculate a confidence level associated with the specifically detected object. Depending on the embodiment, the confidence level can be calculated in various ways. In one example, when detecting an object in the surrounding environment, the vehicle can compare environmental data with predetermined data associated with a known object. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle can use mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0105] In response to determining that an object has a detection confidence level below a threshold, the vehicle may transmit a request for remote assistance targeting the object's identifier to a remote computing system. As described above, the remote computing system can take various forms. For example, the remote computing system may be a computing device located within the vehicle, separate from the vehicle itself, but through which a human operator can interact with the vehicle's passengers or driver, such as through a touchscreen interface for displaying remote assistance information. Additionally or alternatively, as another example, the remote computing system may be a remote computer terminal or other device located at a location not near the vehicle.

[0106] Requests for remote assistance may include environmental data containing the object, such as image data, audio data, etc. The vehicle may transmit the environmental data to the remote computing system via a network (e.g., network 304, etc.) and, in some embodiments, via a server (e.g., server computing system 306, etc.). The human operator of the remote computing system can then use the environmental data as the basis for responding to the request.

[0107] In some embodiments, when an object is detected as having a confidence level below a confidence threshold, the object may be initially identified, and the vehicle may be configured to adjust its operation in response to the initial identification. This operational adjustment may take the form of stopping the vehicle, switching the vehicle to manual control mode, changing the vehicle speed (e.g., rate and / or direction), and other possible adjustments.

[0108] In other embodiments, although the vehicle detects an object with a confidence level that meets or exceeds a threshold, the vehicle may act based on the detected object (e.g., stop if the object is identified as a stop sign with high confidence, etc.), but may be configured to request remote assistance while the vehicle is acting based on the detected object (or at a later time).

[0109] Figure 4A This is a block diagram of a system according to an example embodiment. Specifically, Figure 4A A system 400 is shown, comprising a system controller 402, a lidar device 410, multiple sensors 412, and multiple controllable components 414. The system controller 402 includes a processor 404, a memory 406, and instructions 408 stored in the memory 406 and executable by the processor 404 to perform functions.

[0110] Processor 404 may include one or more processors, such as one or more general-purpose microprocessors (e.g., having single or multi-core cores) and / or one or more special-purpose microprocessors. One or more processors may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also considered herein.

[0111] The memory 406 may include computer-readable media, such as non-transitory computer-readable media, which may include, but are not limited to, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), solid-state drive (SSD), hard disk drive (HDD), optical disc (CD), digital video optical disc (DVD), digital magnetic tape, read / write (R / W) CD, R / W DVD, etc.

[0112] The lidar device 410, further described below, includes a plurality of light emitters configured to emit light (e.g., light pulses, etc.) and one or more photodetectors configured to detect light (e.g., reflected portions of light pulses, etc.). The lidar device 410 can generate three-dimensional (3D) point cloud data from the output of the photodetectors and provide the 3D point cloud data to a system controller 402. The system controller 402 can then perform operations on the 3D point cloud data to determine characteristics of the surrounding environment (e.g., relative positions of objects within the surrounding environment, edge detection, object detection, proximity sensing, etc.).

[0113] Similarly, system controller 402 may use outputs from multiple sensors 412 to determine characteristics of system 400 and / or the surrounding environment. For example, sensors 412 may include one or more of the following: GPS, IMU, image capture devices (e.g., cameras, etc.), light sensors, thermal sensors, and other sensors indicating parameters related to system 400 and / or the surrounding environment. For illustrative purposes, lidar device 410 is depicted as separate from sensor 412, and in some examples may be considered part of or considered as sensor 412.

[0114] Based on the characteristics of system 400 and / or its surrounding environment—which are determined by system controller 402 based on outputs from lidar device 410 and sensor 412—system controller 402 can control controllable component 414 to perform one or more actions. For example, system 400 may correspond to a vehicle, in which case controllable component 414 may include the vehicle's braking system, steering system, and / or acceleration system, and system controller 402 may modify aspects of these controllable components based on characteristics determined from lidar device 410 and / or sensor 412 (e.g., when system controller 402 controls the vehicle in autonomous or semi-autonomous mode, etc.). In this example, lidar device 410 and sensor 412 may also be controlled by system controller 402.

[0115] Figure 4B This is a block diagram of a lidar device according to an example embodiment. Specifically, Figure 4BA lidar device 410 with a controller 416 is shown, the controller 416 being configured to control a plurality of light emitters 424 and one or more photodetectors, such as a plurality of photodetectors 426, etc. The lidar device 410 also includes an ignition circuit 428 configured to select a corresponding light emitter among the plurality of light emitters 424 and supply power to it, and may include a selector circuit 430 configured to select a corresponding photodetector among the plurality of photodetectors 426. The controller 416 includes a processor 418, a memory 420, and instructions 422 stored in the memory 420.

[0116] Similar to processor 404, processor 418 may include one or more processors, such as one or more general-purpose microprocessors and / or one or more special-purpose microprocessors. One or more processors may include, for example, one or more CPUs, one or more microcontrollers, one or more GPUs, one or more TPUs, one or more ASICs, and / or one or more FPGAs. Other types of processors, computers, or devices configured to execute software instructions are also considered herein.

[0117] Similar to memory 406, memory 420 may include computer-readable media, such as non-transitory computer-readable media, such as, but not limited to, ROM, PROM, EPROM, EEPROM, non-volatile random access memory (e.g., flash memory, etc.), SSD, HDD, CD, DVD, digital magnetic tape, R / W CD, R / W DVD, etc.

[0118] Instruction 422 is stored in memory 420 and can be executed by processor 418 to perform functions related to controlling ignition circuit 428 and selector circuit 430, for generating 3D point cloud data, and for processing 3D point cloud data (or may be facilitated by another computing device, such as system controller 402, to process 3D point cloud data).

[0119] The controller 416 can determine 3D point cloud data by emitting light pulses using light emitters 424. An emission time is established for each light emitter, and the relative position at the time of emission is also tracked. The light pulses are reflected from various aspects of the environment surrounding the lidar device 410, such as objects. For example, when the lidar device 410 is in an environment including roads, such objects can include vehicles, signs, pedestrians, road surfaces, building cones, etc. Some objects may be more reflective than others, such that the intensity of the reflected light can indicate the type of object reflecting the light pulse. Furthermore, the surface of an object may be at a different position relative to the lidar device 410, and therefore take more or less time to reflect a portion of the light pulse back to the lidar device 410. Therefore, the controller 416 can track the detection time when the photodetector detects the reflected light pulse and the relative position of the photodetector at that detection time. By measuring the time difference between the emission time and the detection time, the controller 416 can determine how far the light pulse has traveled before being received, and thus determine the relative distance to the corresponding object. By tracking the relative positions at the emission and detection times, controller 416 can determine the orientation of the light pulses and reflected light pulses relative to lidar device 410, and thus determine the relative orientation of the object. By tracking the intensity of the received light pulses, controller 416 can determine how the object reflects light. The 3D point cloud data determined based on this information can therefore indicate the relative positions of the detected reflected light pulses (e.g., in a coordinate system such as a Cartesian coordinate system) and the intensity of each reflected light pulse.

[0120] As further described below, the ignition circuit 428 is used to select the light emitter that emits the light pulse. Similarly, the selector circuit 430 is used to sample the output of the photodetector.

[0121] Figure 5 An optical system 500 according to an example embodiment is shown. As previously described, the optical system 500 may be part of a lidar system, such as lidar unit 528. The optical system 500 may be installed inside the lidar system to monitor the lidar dome window and detect any dirt or damage to the dome. The optical system 500 includes an optical component 510 and one or more light sources 520. In various embodiments, the optical component 510 may include a lens. In this case, the optical component 510 may include one or more plano-convex lenses, prisms, cylindrical lenses, conical lenses, and / or other types of lenses. However, other types of optical components, such as filters, thin films, plane mirrors, windows, diffusers, gratings, and / or prisms, are also considered and are possible.

[0122] The optical system 500 also includes a detector 530. The detector 530 may be a photosensitizing device configured to detect at least a portion of the interacting optical signal 524 as a detected optical signal 526. In some cases, the detector 530 may include at least one of a charge-coupled device (CCD), a portion of a CCD, an image sensor of a camera, or a portion of an image sensor of a camera. Additionally or alternatively, the detector 530 may include a silicon photomultiplier (SiPM), an avalanche photodiode (APD), a single-photon avalanche detector (SPAD), a cryogenic detector, a photodiode, or a phototransistor. Other photosensitizing devices or systems are also possible and are considered herein.

[0123] In some embodiments, the optical system 500 may include an image sensor 540. For example, the image sensor 540 may include multiple charge-coupled device (CCD) elements and / or multiple complementary metal-oxide-semiconductor (CMOS) elements. In some embodiments, the optical system 500 may include multiple image sensors. In an example embodiment, the image sensor 540 may be configured to detect light in the infrared spectrum (e.g., from about 700 nm to about 1000 nm, etc.) and / or in the visible spectrum (e.g., from about 400 nm to about 700 nm, etc.). It is possible to use the image sensor 540 to sense light in other spectral ranges (e.g., long-wavelength infrared (LWIR) light with wavelengths between 8 and 12 micrometers, etc.) and is considered herein.

[0124] Image sensor 540 can be configured according to image sensor format (e.g., size, dimensions, etc.). For example, image sensor 540 may include a full-frame (e.g., 35 mm, etc.) format sensor. Additionally or alternatively, image sensor 540 may include a "cropped sensor" format, such as APS-C (e.g., 28.4 mm diagonal, etc.) or one-inch (e.g., 15.86 mm diagonal, etc.) format. Other image sensor formats are considered and possible within the scope of this disclosure.

[0125] Additionally, the optical system 500 also includes a controller 550. In some embodiments, the controller 550 may be a readout integrated circuit (ROIC) electrically coupled to the image sensor 540. The controller 550 includes at least one of a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). Additionally or alternatively, the controller 550 may include one or more processors 552 and a memory 554. The one or more processors 552 may include general-purpose processors (e.g., those with single or multiple cores) and / or special-purpose processors (e.g., digital signal processors). The one or more processors 552 may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein. The one or more processors 552 may be configured to execute computer-readable program instructions stored in the memory 554. In some embodiments, the one or more processors 552 may execute program instructions to provide at least some of the functions and operations described herein.

[0126] The memory 554 may include or take the form of one or more computer-readable storage media that can be read or accessed by one or more processors 552. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, solid-state memory or other memory or disk storage devices, which may be integrated wholly or partially with at least one of the one or more processors 552. In some embodiments, the memory 554 may be implemented using a single physical device (e.g., a single optical, magnetic, organic solid-state memory or other memory or disk storage device unit), while in other embodiments, the memory 554 may be implemented using two or more physical devices.

[0127] As described above, memory 554 may include computer-readable program instructions related to the operation of optical system 500. At least one processor 552 executes the instructions stored in at least one memory 554 to perform operations.

[0128] The operation includes using a lidar device 128 to determine light intensity information 522 of the surrounding environment 50 of the lidar device 128. The light intensity information 522 may include multiple angles within an exposure threshold range. In some embodiments, at least one high-sensitivity photodiode may integrate the amount of background light sensed within a small time window to determine the average brightness of the solar background in the surrounding environment. For example, the specular reflection of a car may be brighter than the diffuse reflection of a building.

[0129] The operation may also include determining the rotation time associated with each angle within the exposure threshold range.

[0130] The operation may also include determining multiple target image times based on the rotation time associated with each angle within the exposure threshold range.

[0131] The operation may also include capturing multiple images at multiple target image times using a camera system.

[0132] Figure 6A and Figure 6B The environment surrounding the lidar device on the carrier 600 is shown. The surrounding environment is divided into multiple angles 602. The figures also show a first angle 604 with high exposure, a second angle 606 with low exposure, and multiple angles 608 within the exposure threshold range detected by the lidar device. An example embodiment may include using a blur detection camera to capture images at rotation times associated with angles within the exposure threshold range, such that the exposure time is consistent across images. This allows images to be captured using the same exposure settings, thus improving image quality and reducing processing time.

[0133] In the example embodiment, Figure 6A The diagram illustrates a scene where the average background light is very bright in front of the vehicle and very dark behind it. This scenario might occur, for example, when leaving a tunnel on a sunny day. When the blur detection camera is pointed backward into the tunnel, the average background light is darker, so a longer exposure time should be used to avoid underexposed images. When the camera is pointed forward into the sun, a shorter exposure time should be used to avoid overexposed images. However, by taking images within the exposure threshold at an angle associated with the average background brightness, the same exposure time can be used for all images. By attempting to make all images have a consistent exposure time, processing can be reduced. These images can then be used to determine if the LiDAR window is obstructed.

[0134] In an exemplary embodiment, Figure 6B The following scene is shown: the average background light above the vehicle is very bright due to the sun, and the light towards the vehicle is also very bright due to sun reflection. This scene would likely occur on a particularly sunny day. The surrounding environment is divided into multiple elevation angles 610. The accompanying figure also shows a first angle 612 with high exposure, a second angle 614 with low exposure, and multiple angles 616 within the exposure threshold range detected by a lidar device. Similar to... Figure 6A It can capture images when the lidar device and camera are oriented at an angle associated with the average background brightness within the exposure threshold range.

[0135] Figure 7This is a flowchart of method 700 according to an example embodiment. In various embodiments, one or more blocks of method 700 may be... Figure 1 The computer system 112 shown is used to perform this action. In some embodiments, one or more blocks of method 700 may be performed by a computing device (e.g., a controller for one or more components of optical system 500). The computing device may include computing components such as non-volatile memory (e.g., hard disk drive, read-only memory (ROM), etc.), volatile memory (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.), user input devices (e.g., mouse, keyboard, etc.), displays (e.g., LED displays, liquid crystal displays (LCDs), etc.), and / or network communication controllers (e.g., based on the IEEE 802.11 standard). (Controller, Ethernet controller, etc.). For example, a computing device can run instructions stored on a non-transitory computer-readable medium (e.g., a hard disk drive, etc.) to perform one or more of the operations considered herein.

[0136] In box 702, method 700 may include using a LiDAR device to determine light intensity information of the environment surrounding the LiDAR device. The light intensity information may include multiple angles within an exposure threshold range. In box 704, the method may include determining a rotation time associated with each angle within the exposure threshold range. In box 706, the method may include determining multiple target image times based on the rotation time associated with each angle within the exposure threshold range. In box 708, the method may include capturing multiple images at the multiple target image times using a camera system.

[0137] In some embodiments of method 700, the lidar device rotates continuously. The device can maintain a tracking lidar at its current facing angle during rotation, and multiple high-sensitivity photodiodes can be used to determine the light intensity information of the lidar's surrounding environment; these high-sensitivity photodiodes can be integrated into the lidar device. Alternatively, the lidar's inherent photosensitive element can be used to determine the light intensity information. The lidar's inherent photosensitive element can be one or more photodetectors, as previously described, which can be particularly sensitive detectors (e.g., avalanche photodiodes, etc.). In some examples, such photodetectors are capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs), etc.). Furthermore, such photodetectors can be arranged in an array (e.g., in a silicon photomultiplier (SiPM), etc.) (e.g., via series electrical connections, etc.). In some examples, one or more photodetectors are Geiger-mode operating devices, and the lidar includes sub-components designed for this Geiger-mode operation. The lidar device can use photodetectors to sense and guide light pulses returning after emission.

[0138] Light intensity information can include the average brightness of the surrounding environment from multiple angles. For example, as the lidar device rotates, it can sense the average background light of the surrounding environment and correlate the sensed background light value with a specific angle the lidar is facing at that moment. The light intensity information may also include the background light brightness of the surrounding environment, with at least low and high exposure values ​​for the lidar device. Specifically, the light intensity information can include multiple angles facing the lidar device with low exposure and multiple angles facing the lidar device with high exposure. Low exposure can be considered any brightness below approximately 100 lumens per square meter, and high exposure can be considered any brightness above 15,000 lumens per square meter.

[0139] In an example embodiment of method 700, the average brightness of each of the multiple angles can be converted into exposure time. For example, Figure 8 A lookup table is shown that converts average background brightness to exposure time. Different sensors can be used to measure average background light, and different types of sensors can have different levels of light sensitivity. In the table, the units of average background light depend on the device used to collect them and can therefore be arbitrary. In an example embodiment, the photodetector of a LiDAR device can be used to collect the raw average background light in lumens. Using the lookup table, the raw average background light can be associated with an appropriate exposure time for that background light. In other embodiments, different mappings can be used to convert background brightness to exposure time. For example, the data can be fitted to a polynomial.

[0140] In an example embodiment of method 700, the light intensity information may include multiple angles. These multiple angles can be determined by a lidar device. They may be at least one of multiple yaw angles and multiple elevation angles. Specifically, for multiple yaw angles, the lidar device may maintain tracking of the angle it faces as it rotates about its vertical axis. For elevation angles, the lidar device may rotate about its orthogonal axis, and it may maintain tracking of that angle. The multiple angles may be within an exposure threshold range. The exposure range may be multiple exposure times of the camera. For example, most cameras have an available exposure range from less than 1 millisecond to 25 milliseconds. In the example embodiment, the threshold range for exposure time may be from 5 milliseconds to 15 milliseconds. In an alternative embodiment, the threshold range for exposure time may be any value from 1 millisecond to 25 milliseconds.

[0141] To determine the light intensity information of the environment surrounding a lidar device, where the light intensity information includes multiple angles within an exposure threshold range, an example embodiment of method 700 may include measuring the average background light of multiple fixed-size yaw sectors using the lidar device. The multiple fixed-size yaw sectors may be selected portions of a 360-degree rotation of the lidar device. The size of the sector may depend on the lidar rotation frequency and the exposure threshold range, as well as the frame rate of the obstacle detection camera. For example, the size of the yaw sector may be the lidar device rotation frequency multiplied by the threshold exposure time multiplied by 360 degrees. Additionally, the 360-degree rotation of the lidar device may be divided into 14 equally sized sectors. Other numbers of equally sized sectors are also possible, such as four equally divided fixed-size yaw sectors. The sectors may also be divided unequally by varying the lidar device's rate or the obstacle detection camera's exposure time. In another embodiment, the size of the yaw sector may be determined by an interpolation function or dynamically based on the environment surrounding the lidar device. For example, the size of the yaw sector may vary based on the background lighting in the environment.

[0142] The light intensity information of multiple fixed-size yaw sectors can then be converted into exposure times for the multiple fixed-size yaw sectors. As previously described, at least one of a lookup table or mapping technique can be used to convert the light intensity information into exposure time. This embodiment can then include determining a subset of multiple fixed-size yaw sectors within an exposure threshold range. For example, the exposure threshold range can be from 5 milliseconds to 15 milliseconds. Based on a lookup table or mapping technique, any fixed-size yaw sector that includes light intensity information corresponding to a threshold range is a fixed-size yaw sector within a subset of multiple fixed-size yaw sectors. In an additional embodiment, the desired exposure time can be selected based on the light intensity within each fixed-size yaw sector. For example, the light intensity can be summed within each yaw sector, and the exposure time can then be selected such that the image time is aligned with the position of the yaw sector. The light intensity can also be summed over multiple yaw sectors to determine the desired exposure time.

[0143] Some embodiments of method 700 include determining a rotation time associated with each angle within an exposure threshold range. The rotation time can be determined based on a known angle faced by the lidar device and a constant rotation rate of the lidar device. As the lidar device rotates, a computing device associated with the lidar, such as computer system 112 or controller 550, can maintain the angle of the tracking lidar relative to the front of the vehicle. For example, a lidar facing the central front of the vehicle could be at zero degrees and 360 degrees, while a lidar facing the central rear of the vehicle could be at 180 degrees. Similarly, if the lidar has multiple scanning axes, the computing device associated with the lidar can maintain the elevation angle faced by the tracking lidar. For example, an upward-facing lidar could be at zero degrees.

[0144] In the example embodiment, angles can be grouped into sectors of a fixed size, such as 0 to 10 degrees, 10 to 20 degrees, etc. Alternatively, angles may not be grouped together but may be considered individually.

[0145] In addition to the computing system associated with the LiDAR device monitoring the angles faced by the LiDAR device, the camera system can also monitor multiple angles spanned by the rotation of the LiDAR device. The camera system can correspond to the same computing device at the LiDAR device, such as computer system 112 or controller 550. Alternatively, the computer system may include its own computing device. The camera system can monitor the multiple angles spanned by the LiDAR device and use these angles to determine when to capture an image. The multiple angles spanned by the LiDAR device can be angles from 0 to 360 degrees faced by the LiDAR.

[0146] Once multiple angles are determined, and the angles within a threshold exposure range are identified, the rotation time can be correlated with each angle within that threshold exposure range. The LiDAR and the obstacle detection camera attached to it can rotate at a constant rotation rate. For example, the LiDAR can rotate at 10 Hz. Using the constant rotation rate of the LiDAR device and the multiple angles spanned by the LiDAR device's rotation, the rotation time associated with each angle within the exposure threshold range can also be determined.

[0147] Example embodiments of method 700 may include determining a plurality of target image times based on rotation times associated with each angle within an exposure threshold range. Specifically, the camera system may capture images at image times associated with angles within the exposure threshold range. In this way, images can be captured at desired exposure times to attempt to avoid underexposed or overexposed images. Multiple target image times may be set during multiple rotation times associated with multiple angles within the exposure threshold range. For example, the target image times may be any time associated with 45 degrees to 90 degrees, assuming 45 degrees to 90 degrees are within the exposure threshold range.

[0148] The target image time can be further adjusted to avoid times outside the threshold. Multiple target image times can be adjusted by varying the image readout blank lines. The image readout blank lines are parameters in an example embodiment of the camera module. The image readout blank lines do not contain any image data but can be used to change the frames per second. In an example embodiment, varying the image readout blank lines includes at least one of the following: increasing the camera readout time of image frames preceding subsequent image frames, or decreasing the camera readout time of image frames preceding subsequent image frames. By increasing the number of blank lines, the camera readout time increases, and therefore the start time of subsequent frames is delayed. Similarly, by decreasing the number of blank lines, the camera readout time decreases, and the start time of subsequent frames is advanced. By increasing or decreasing the blank lines, the system can manipulate the timing of image capture at angles within the exposure threshold. In an example embodiment, an additional blank line can delay subsequent frames by 54.792 microseconds. Multiple blank frames can be used to achieve the desired position for capturing images. In alternative embodiments, the blank lines may correspond to other times.

[0149] Example embodiments of method 700 may include capturing multiple images at multiple target image times by a camera system. By capturing multiple images at the target image times, images can be captured using a desired exposure time across all images. The time taken to capture images to achieve the target image times can vary as described above, and multiple images can be captured in real time using a streaming camera. In alternative embodiments, the camera system may also be any camera configured to capture still images and / or video. For example, a non-streaming camera may also capture multiple images. A non-streaming camera may include at least one camera that stores the multiple images to be processed later. Multiple images can be captured at multiple target times across all images using a desired exposure time, and these multiple images can be stored to determine at a later time whether the LiDAR window is obstructed. Once multiple images have been captured at the target image times, they can be used to determine whether the LiDAR device is obstructed. Specifically, the multiple images can be used to determine whether there is any dirt on the LiDAR window or whether the LiDAR window is damaged in any way.

[0150] An example embodiment of method 700 may further include determining a second plurality of angles outside the exposure threshold range of the camera system to avoid capturing images at these angles and times. This embodiment may include using a LiDAR device to determine light intensity information, further including the second plurality of angles outside the exposure threshold range. As previously described, the LiDAR device may determine the angles and the average background light for each angle. The average background light may be converted into exposure time. As previously described, the exposure threshold range may be from 5 milliseconds to 15 milliseconds. The second plurality of angles may have exposures outside the threshold range and are therefore not preferred for capturing images at those angles.

[0151] Using the previously described method, the computational system can determine the rotation time associated with all angles faced by the LiDAR device during rotation. This includes determining the rotation time associated with each of a second plurality of angles outside the exposure threshold range. Based on the rotation time associated with each of the second plurality of angles outside the exposure threshold range, the computational system can determine a plurality of undesirable image times. The plurality of undesirable image times can include times during which, if an image is captured, it will produce an image with undesirable exposure. Therefore, in response to determining the plurality of undesirable image times, the computational system associated with the camera system can vary the image readout blank line to avoid exposure at angles outside the threshold.

[0152] Since multiple images were captured at similar angles using similar exposure times in an attempt to coordinate the images, some example embodiments described herein may include using the same signal processing across multiple images. Specifically, the images were captured at a specific angle and have similar average background light and similar fields of view, so they can be processed similarly. For example, sequential frame processing techniques can be used across multiple images. Typically, continuous frame processing is not an option for sensors that rotate 360 ​​degrees over time because multiple images were captured at different angles including different background lights. For example, frame subtraction cannot be used to subtract background effects because the two different parts of the field of view are in the image frame. However, since multiple images were captured at a specific angle, the fields of view between image frames are similar. Therefore, images captured at similar angles can have their background subtracted to determine if the LiDAR window is occluded. Similarly, since the average background light is similar between images, any background illumination can be subtracted from the image. Therefore, as a result of this method, the ordering of illuminator on / off states and background subtraction become more useful.

[0153] This disclosure is not limited to the specific embodiments described herein, but is intended to illustrate various aspects. It will be apparent to those skilled in the art that many modifications and variations can be made without departing from the spirit and scope of the invention. In addition to the methods and apparatus listed herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art based on the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0154] The above detailed description, with reference to the accompanying drawings, illustrates various features and functions of the disclosed systems, devices, and methods. In the drawings, similar symbols generally identify similar components unless the context otherwise indicates. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that, as generally described herein and shown in the accompanying drawings, aspects of this disclosure can be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are expressly taken into account herein.

[0155] With respect to any or all message flowcharts, scenarios, and processes shown in the accompanying drawings and discussed herein, each step, block, operation, and / or communication may represent information processing and / or information transmission according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed (including substantially simultaneously or in reverse order), depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be associated with any message flowcharts, scenarios, and processes discussed herein. Figure 1 They can be used together, and these message flow diagrams, scenarios, and flowcharts can be combined with each other, either partially or entirely.

[0156] A step, block, or operation representing information processing may correspond to a circuit that can be configured to perform a specific logical function of the method or technique described herein. Alternatively or additionally, a step or block representing information processing may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions executable by a processor for implementing a specific logical operation or action in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including RAM, a disk drive, a solid-state drive, or other storage media.

[0157] Furthermore, steps, blocks, or operations representing one or more information transfers can correspond to information transfers between software and / or hardware modules within the same physical device. However, other information transfers can occur between software and / or hardware modules in different physical devices.

[0158] The specific arrangements shown in the accompanying drawings should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in the given drawings. Furthermore, some of the shown elements may be combined or omitted. Additionally, example embodiments may include elements not shown in the figures.

[0159] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes and not for limitation, and the true scope is indicated by the appended claims.

Claims

1. A method for camera orientation control, comprising: A lidar device is used to determine information related to the light intensity of the surrounding environment of the lidar device, wherein the information includes multiple angles at which the light brightness is related to the exposure within an exposure threshold range; Determine the rotation time associated with each of the plurality of angles; Based on the rotation time associated with each of the plurality of angles, a plurality of target image times are determined; and Multiple images are captured at the multiple target image times using a camera system.

2. The method according to claim 1, wherein, The timing of the multiple target images is adjusted by changing the blank lines read from the image.

3. The method according to claim 2, wherein, The variable image readout blank line includes at least one of increasing the camera readout time of the image frame preceding the subsequent image frame, or decreasing the camera readout time of the image frame preceding the subsequent image frame.

4. The method according to claim 1, wherein, Determining the plurality of target image times includes setting the plurality of target image times during rotation time associated with the plurality of angles.

5. The method according to claim 1, wherein, The camera system is a streaming camera.

6. The method according to claim 1, further comprising determining, based on the plurality of images, that the lidar device is obstructed.

7. The method according to claim 1, wherein, The information includes multiple angles with low exposure and multiple angles with high exposure, wherein the low exposure and high exposure include the background brightness of the surrounding environment.

8. The method according to claim 1, wherein, The lidar device is used to determine information related to the light intensity of the surrounding environment, wherein the information includes the plurality of angles at which the light brightness is related to exposure within an exposure threshold range, and further includes: The average background light of multiple fixed-size yaw sectors was measured using a lidar device. The information from multiple fixed-size yaw sectors is converted into the exposure time of multiple fixed-size yaw sectors; and Determine a subset of multiple fixed-size yaw sectors within the exposure threshold range.

9. The method according to claim 1, wherein, The multiple angles include at least one of multiple yaw angles and multiple pitch angles.

10. The method of claim 1, further comprising monitoring, via the camera system, multiple angles spanned by the rotation of the lidar device.

11. The method according to claim 10, wherein, The rotation time associated with each of the plurality of angles is based on the constant rotation rate of the lidar device and the plurality of angles spanned by the rotation of the lidar device.

12. The method according to claim 1, wherein, Information related to the light intensity of the surrounding environment of the lidar device includes the average brightness of the surrounding environment from multiple angles.

13. The method of claim 12, further comprising converting the average brightness of each of the plurality of angles into an exposure time.

14. The method of claim 13, further comprising capturing multiple images using exposure time.

15. The method according to claim 1, wherein: Determining the information using the lidar device also includes a second plurality of angles, at which the brightness of the light is related to exposure outside the exposure threshold range; Determine the rotation time associated with each of the second plurality of angles; Based on the rotation time associated with each of the second plurality of angles, a plurality of unwanted image times are determined; as well as In response to determining the multiple unwanted image times, the image readout blank lines are changed.

16. A non-transitory computer-readable medium having instructions stored thereon, wherein, The instructions, when executed by the processor, cause the processor to perform a method, the method comprising: A lidar device is used to determine information related to the light intensity of the surrounding environment of the lidar device, wherein the information includes multiple angles at which the light brightness is related to the exposure within an exposure threshold range; Determine the rotation time associated with each of the plurality of angles; Based on the rotation time associated with each of the plurality of angles, a plurality of target image times are determined; and Multiple images are captured at the multiple target image times using a camera system.

17. The non-transitory computer-readable medium according to claim 16, wherein, The timing of the plurality of target images is adjusted by varying the image readout blank line, wherein varying the image readout blank line includes at least one of increasing the camera readout time of image frames preceding subsequent image frames or decreasing the camera readout time of image frames preceding subsequent image frames.

18. The non-transitory computer-readable medium according to claim 16, wherein, Using the lidar device to determine information related to the light intensity of the surrounding environment of the lidar device also includes: The average background light of multiple fixed-size yaw sectors was measured using a lidar device. The information from multiple fixed-size yaw sectors is converted into the exposure time of multiple fixed-size yaw sectors; and Determine a subset of fixed-size yaw sectors within the exposure threshold range.

19. The non-transitory computer-readable medium according to claim 16, wherein, The multiple angles include at least one of multiple yaw angles and multiple pitch angles.

20. An optical system comprising: Optical components; An image sensor is configured to receive light from the scene via imaging optics; as well as The controller is configured to execute an imaging routine, wherein the imaging routine includes: A lidar device is used to determine information related to the light intensity of the surrounding environment of the lidar device, wherein the information includes multiple angles at which the light brightness is related to the exposure within an exposure threshold range; Determine the rotation time associated with each of the plurality of angles; Based on the rotation time associated with each of the plurality of angles, a plurality of target image times are determined; and Multiple images are captured at the multiple target image times using a camera system.

Images