Camera placement for vehicle object detection and avoidance

Abstract

An exemplary embodiment relates to a camera arrangement for vehicle object detection and avoidance. [Solution] An exemplary system includes a vehicle and at least one camera of a first camera type mounted on the vehicle. The system also includes a plurality of cameras of a second camera type mounted on the vehicle. Furthermore, the system includes a plurality of cameras of a third camera type mounted on the vehicle. The system also includes a computing device communicatively coupled to the at least one camera of the first camera type, the plurality of cameras of the second camera type, and the plurality of cameras of the third camera type. The computing device is configured to identify objects located within a first distance range from the vehicle, objects located within a second distance range from the vehicle, and objects located within a third distance range from the vehicle.

Description

Background Art

[0001] Unless otherwise stated in this specification, the descriptions in this chapter are not prior art for the claims of this application and should not be considered prior art by inclusion in this chapter.

[0002] Cameras and image sensors are devices that can be used to capture images of scenes. Some cameras (e.g., film cameras) chemically capture images on film. Other cameras (e.g., digital cameras) electrically capture image data (e.g., using a charge-coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) sensor). The images captured by a camera can be analyzed to determine their content. For example, a processor can execute a machine learning algorithm to identify objects in a scene based on a library of previously classified objects that includes, for example, the shape, color, size, etc. of the objects (such machine learning algorithms can be applied to computer vision in robotics or other applications).

[0003] Cameras can have various features that can distinguish one camera from another. For example, a camera and / or the images captured by a camera can be identified by values such as aperture size, f-number, exposure time, shutter speed, depth of field, focal length, International Organization for Standardization (ISO) sensitivity (or gain), pixel size, sensor resolution, exposure distance, etc. These features can be based on the lens, the image sensor, and / or additional facets of the camera. Further, these features can also be adjustable within a single camera (e.g., the aperture of the lens on the camera can be adjusted between photographs).

Summary of the Invention

[0004] This disclosure relates to a technology that can be used to detect and identify objects, such as objects around a vehicle. Such object detection and identification may utilize data, such as images, collected from different types of sensors and / or cameras. These sensors and / or cameras may be located on the vehicle. The type of camera and / or sensor, as well as the choice of location for the camera and / or sensor on the vehicle, may affect the information captured by the camera and / or sensor.

[0005] In one embodiment, a system is provided. The system includes a vehicle. The system also includes at least one camera of a first camera type mounted on the vehicle. Each camera of the first camera type has a first field of view. Furthermore, the system includes a plurality of cameras of a second camera type mounted on the vehicle. Each camera of the second camera type has a second field of view, the second field of view extending at least 170 degrees in the yaw direction relative to the vehicle. Furthermore, the system includes a plurality of cameras of a third camera type mounted on the vehicle. Each camera of the third camera type has a third field of view. The combined field of view of the plurality of cameras of the third camera type extends 360 degrees in the yaw direction relative to the vehicle. The first field of view extends at a smaller angle in the yaw direction relative to the vehicle than the third field of view, and the third field of view extends at a smaller angle in the yaw direction relative to the vehicle than the second field of view. Furthermore, the system includes a computing device communicatively coupled to at least one camera of the first camera type, a plurality of cameras of the second camera type, and a plurality of cameras of the third camera type. The computing device is configured to identify objects located within a first distance range from the vehicle based on one or more first images captured by at least one camera of a first camera type. The computing device is also configured to identify objects located within a second distance range from the vehicle based on one or more second images captured by multiple cameras of a second camera type. Furthermore, the computing device is configured to identify objects located within a third distance range from the vehicle based on one or more third images captured by multiple cameras of a third camera type. The third distance range includes distances further from the vehicle than those included in the second distance range, and the first distance range includes distances further from the vehicle than those included in the third distance range.

[0006] In another embodiment, a method is provided. The method includes receiving one or more first images captured by at least one camera of a first camera type using a computing device. Each camera of the first camera type is mounted on a vehicle and has a first field of view. The method also includes receiving one or more second images captured by a plurality of cameras of a second camera type using a computing device. Each camera of the second camera type is mounted on a vehicle and has a second field of view. The second field of view extends at least 170 degrees in the yaw direction relative to the vehicle. The method also includes receiving one or more third images captured by a plurality of cameras of a third camera type using a computing device. Each camera of the third camera type is mounted on a vehicle and has a third field of view, and the combined field of view of the plurality of cameras of the third camera type extends 360 degrees in the yaw direction relative to the vehicle. The first field of view extends at a smaller angle in the yaw direction relative to the vehicle than the third field of view, and the third field of view extends at a smaller angle in the yaw direction relative to the vehicle than the second field of view. Furthermore, the method includes, by means of a computing device, identifying an object located within a first distance range from the vehicle based on one or more first images. The method also includes, by means of a computing device, identifying an object located within a second distance range from the vehicle based on one or more second images. Furthermore, the method includes, by means of a computing device, identifying an object located within a third distance range from the vehicle based on one or more third images. The third distance range includes distances further from the vehicle than those included in the second distance range, and the first distance range includes distances further from the vehicle than those included in the third distance range. Furthermore, the method includes, by means of a computing device, determining a driving decision for the vehicle based on an object located within the first distance range from the vehicle, an object located within the second distance range from the vehicle, or an object located within the third distance range from the vehicle.

[0007] In yet another embodiment, a non-temporary computer-readable medium stored thereon is provided. Program instructions cause the processor to perform an action when executed by the processor. The action includes receiving one or more first images captured by at least one camera of a first camera type. Each camera of the first camera type is mounted on the vehicle and has a first field of view. The action also includes receiving one or more second images captured by a plurality of cameras of a second camera type. Each camera of the second camera type is mounted on the vehicle and has a second field of view, the second field of view extending at least 170 degrees in the yaw direction relative to the vehicle. Furthermore, the action includes receiving one or more third images captured by a plurality of cameras of a third camera type. Each camera of the third camera type is mounted on the vehicle and has a third field of view, the combined field of view of the plurality of cameras of the third camera type extending 360 degrees in the yaw direction relative to the vehicle. The first field of view extends to an angle smaller than that of the third field of view in the yaw direction relative to the vehicle, and the third field of view extends to an angle smaller than that of the second field of view in the yaw direction relative to the vehicle. Furthermore, the operation includes identifying objects located within a first distance range from the vehicle based on one or more first images. The operation also includes identifying objects located within a second distance range from the vehicle based on one or more second images. Furthermore, the operation includes identifying objects located within a third distance range from the vehicle based on one or more third images. The third distance range includes distances further from the vehicle than those included in the second distance range, and the first distance range includes distances further from the vehicle than those included in the third distance range. Furthermore, the operation includes determining a driving decision for the vehicle based on objects located within the first distance range from the vehicle, objects located within the second distance range from the vehicle, or objects located within the third distance range from the vehicle.

[0008] These, as well as other embodiments, advantages, and alternatives, will become apparent to those skilled in the art by reading the following detailed description with appropriate reference to the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a functional block diagram illustrating a vehicle according to an exemplary embodiment. [Figure 2] Figure 2A is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2B] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2C] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2D] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2E] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2F] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2G] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2H] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2I] This is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2J] This is an illustrative diagram showing the field of view of various sensors according to an exemplary embodiment. [Figure 2K] This is an illustrative diagram of beam steering for a sensor according to an exemplary embodiment. [Figure 3] This is a conceptual diagram illustrating wireless communication between various computing systems related to autonomous or semi-autonomous vehicles, according to an exemplary embodiment. [Figure 4A] This is a block diagram of a system including a LiDAR device according to an exemplary embodiment. [Figure 4B] This is a block diagram of a LiDAR device according to an exemplary embodiment. [Figure 5A] This is an illustrative diagram of a first camera type camera and associated first field of view according to an exemplary embodiment. [Figure 5B]An exemplary diagram of a camera of a second camera type and a related second field of view according to an exemplary embodiment. [Figure 5C] An exemplary diagram of a camera of a third camera type and a related third field of view according to an exemplary embodiment. [Figure 5D] An exemplary diagram of a camera of a fourth camera type and a related fourth field of view according to an exemplary embodiment. [Figure 6A] An exemplary diagram of a camera system according to an exemplary embodiment. [Figure 6B] A simplified exemplary diagram of the camera system of FIG. 6A showing a camera of a first camera type of the camera system according to an exemplary embodiment. [Figure 6C] A simplified exemplary diagram of the camera system of FIG. 6A showing a camera of a second camera type of the camera system according to an exemplary embodiment. [Figure 6D] A simplified exemplary diagram of the camera system of FIG. 6A showing a camera of a third camera type of the camera system according to an exemplary embodiment. [Figure 7A] An exemplary diagram of a camera system according to an exemplary embodiment. [Figure 7B] A simplified exemplary diagram of the camera system of FIG. 7A showing a camera of a first camera type of the camera system according to an exemplary embodiment. [Figure 7C] A simplified exemplary diagram of the camera system of FIG. 7A showing a camera of a second camera type of the camera system according to an exemplary embodiment. [Figure 7D] A simplified exemplary diagram of the camera system of FIG. 7A showing a camera of a third camera type of the camera system according to an exemplary embodiment. [Figure 8A] An exemplary diagram of a camera system according to an exemplary embodiment. [Figure 8B] A simplified exemplary diagram of the camera system of FIG. 8A showing a camera of a first camera type of the camera system according to an exemplary embodiment. [Figure 8C]A simplified exemplary diagram of the camera system of FIG. 8A showing a camera of a second camera type of the camera system according to an exemplary embodiment. [Figure 8D] A simplified exemplary diagram of the camera system of FIG. 8A showing a camera of a third camera type of the camera system according to an exemplary embodiment. [Figure 9A] An exemplary diagram of a camera system according to an exemplary embodiment. [Figure 9B] A simplified exemplary diagram of the camera system of FIG. 9A showing a camera of a first camera type of the camera system according to an exemplary embodiment. [Figure 9C] A simplified exemplary diagram of the camera system of FIG. 9A showing a camera of a second camera type of the camera system according to an exemplary embodiment. [Figure 9D] A simplified exemplary diagram of the camera system of FIG. 9A showing a camera of a third camera type of the camera system according to an exemplary embodiment. [Figure 9E] A simplified exemplary diagram of the camera system of FIG. 9A showing a camera of a fourth camera type of the camera system according to an exemplary embodiment. [Figure 10A] An exemplary diagram of a camera system according to an exemplary embodiment. [Figure 10B] A simplified exemplary diagram of the camera system of FIG. 10A showing a camera of a first camera type of the camera system according to an exemplary embodiment. [Figure 10C] A simplified exemplary diagram of the camera system of FIG. 10A showing a camera of a second camera type of the camera system according to an exemplary embodiment. [Figure 10D] A simplified exemplary diagram of the camera system of FIG. 10A showing a camera of a third camera type of the camera system according to an exemplary embodiment. [Figure 10E] A simplified exemplary diagram of the camera system of FIG. 10A showing a camera of a fourth camera type of the camera system according to an exemplary embodiment. [Figure 11A]This is an illustrative diagram of a camera system according to an exemplary embodiment. [Figure 11B] Figure 11A is a simplified illustrative diagram of the camera system, showing a camera of a first camera type according to an exemplary embodiment. [Figure 11C] Figure 11A is a simplified illustrative diagram of the camera system, showing a second camera type of camera in an exemplary embodiment of the camera system. [Figure 11D] Figure 11A is a simplified illustrative diagram of the camera system, showing a third camera type of camera in an exemplary embodiment of the camera system. [Figure 11E] Figure 11A is a simplified illustrative diagram of the camera system, showing a fourth camera type of camera in the camera system according to an exemplary embodiment. [Figure 12] This is an illustrative diagram of an exemplary embodiment of object recognition technology. [Figure 13] This is an illustrative diagram of a first modified third camera type camera according to an exemplary embodiment. [Figure 14] This is a flowchart of the method according to an exemplary embodiment. [Modes for carrying out the invention]

[0010] This disclosure describes exemplary methods and systems. Any exemplary embodiment or configuration described herein should not be construed as necessarily preferable or advantageous to other embodiments or configurations. Furthermore, exemplary embodiments described herein are not intended to be limiting. Certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, and it will be readily apparent that all of these configurations are envisioned herein. In addition, specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Furthermore, some of the exemplary elements may be combined or omitted. Moreover, exemplary embodiments may include elements not illustrated in the figures.

[0011] The LiDAR devices described herein may include one or more optical emitters and one or more detectors used to detect light emitted by the optical emitters and reflected by one or more objects in the environment surrounding the LiDAR device. For example, the surrounding environment may include an interior or exterior environment, such as the inside or outside of a building. Additionally or alternatively, the surrounding environment may include the interior of a vehicle. Furthermore, the surrounding environment may include the periphery of a road and / or on a road. Examples of objects in the surrounding environment include, but are not limited to, other vehicles, traffic signs, pedestrians, cyclists, road surfaces, buildings, and terrain. Additionally, one or more optical emitters may emit light into the local environment of the LiDAR itself. For example, light emitted from one or more optical emitters may interact with the housing of the LiDAR and / or surfaces or structures coupled to the LiDAR. In some cases, LiDAR may be mounted on a vehicle, in which case one or more optical emitters may be configured to emit light that interacts with objects in the vicinity of the vehicle. Furthermore, the optical emitters may include, among other possibilities, optical fiber amplifiers, laser diodes, and light-emitting diodes (LEDs).

[0012] Information (e.g., images) may be collected from multiple cameras and / or sensors. This information may be combined and used for machine vision on a vehicle (e.g., a vehicle operating in autonomous or semi-autonomous mode). For example, a camera system may use the information collected from these cameras and other sensors to provide warnings to the user or to make other decisions about the vehicle.

[0013] The camera placement in a camera system used for machine vision on a vehicle (e.g., a vehicle operating in autonomous or semi-autonomous mode) can be complex. Such a system may need to integrate multiple cameras, sensors, and processors to provide real-time information about the vehicle's surroundings. The cameras and other sensors may be of different types, including those that capture high-resolution images, those configured to detect objects at short distances from the vehicle, and those configured to detect objects at long distances from the vehicle.

[0014] Therefore, when designing a camera system, multiple design parameters may be considered. One such set of design parameters is the type of camera and / or sensor to be used in the camera system, and the location of such camera and / or sensor. When determining the type of camera and / or sensor to use, and the location of those cameras and / or sensors on the vehicle, there may be multiple, and possibly competing, objectives. One such objective may be improved safety, e.g., the vehicle's ability to avoid or warn occupants of potentially dangerous situations. Another objective may be improved efficiency, e.g., the ability to make more complex decisions based on the same amount of information. This may include decisions that take into account conditions that could lead to a decrease in sensor data quality (e.g., different weather conditions). A further objective may be to reduce the overall cost of creating, training, using, and / or maintaining the camera system. In some embodiments described herein, the selection of the type of camera and / or sensor, and their location on the vehicle, may advance these objectives simultaneously. In some embodiments described herein, the selection of the type of camera and / or sensor, and their location on the vehicle, may involve prioritizing one objective over one or more of the other objectives.

[0015] Multiple different types of cameras may be mounted on the vehicle and used for computer vision (for example, to capture images used for object detection and / or classification and avoidance). For example, three long-range cameras may be mounted on the roof of the vehicle and oriented forward (e.g., in the vehicle's primary direction of travel). Furthermore, such a system may include four cameras configured to detect objects located close to the vehicle. Each of these cameras may have a field of view of at least 170 degrees (e.g., a fisheye camera). In addition, such cameras may be mounted on the front bumper or side fender bumper of the vehicle (e.g., above the front or rear wheel wells of the vehicle) and / or oriented forward, laterally, or rearward with respect to the vehicle's direction of travel.

[0016] Some embodiments may also include five cameras, which, when combined, can detect objects located at intermediate distances from the vehicle and at any angle in the yaw direction relative to the vehicle (e.g., any angle from 0 to 360 degrees). These five cameras may be mounted on the vehicle and / or on the roof of the vehicle above one or more of the front wheels (e.g., adjacent to one of the 170-degree view cameras). Furthermore, the five cameras, which can observe 360 ​​degrees around the vehicle, may each be oriented in one or more lateral, forward, and / or rear directions relative to the direction of travel of the vehicle. Given this camera arrangement, the system may be able to simultaneously capture images that can be used to identify objects that are (1) close to the vehicle, (2) at intermediate distances from the vehicle and within the entire 360-degree range in the yaw direction, and (3) at long distances from the vehicle and forward relative to the direction of travel of the vehicle.

[0017] In some embodiments described herein, images captured from one or more cameras may be combined with data collected by other sensors (e.g., light detection and ranging (LiDAR) devices). Combining information from multiple sources can enable improved detection of certain types of objects, such as traffic lights and headlights. For example, a LiDAR device may be mounted on a vehicle and configured to generate a point cloud indicating the distance to objects in the environment surrounding the vehicle.

[0018] In some embodiments, one of the cameras configured to detect objects located close to the vehicle may include a neutral density filter through which an image is captured. Such a camera may be oriented relative to the vehicle to capture an image from a pitch angle greater than 20 degrees relative to the vehicle. Furthermore, the camera may be configured to capture a pair of images. The first of these two images may be captured with an exposure time set by the camera's automatic exposure setting. The second of these two images may be captured with an exposure time longer than a predetermined exposure time long enough to ensure the capture of flashing light sources (e.g., a predetermined exposure time of 1 / 60 second or 1 / 50 second), or with an exposure time longer than the exposure time set by the camera's automatic exposure setting. These two images may be aligned and overlapped with a point cloud generated by a LiDAR device to create an overlapping image. Detection of objects close to the vehicle may then be performed based on the overlapping image.

[0019] The following description and accompanying drawings illustrate the features of various exemplary embodiments. The embodiments provided are illustrative and not intended to be limiting. Accordingly, the dimensions in the drawings are not necessarily to scale.

[0020] Herein, exemplary systems within the scope of this disclosure will be described in more detail. Exemplary systems may be implemented in or take the form of automobiles. Additionally, exemplary systems may also be implemented in or take the form of various vehicles, such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor-trailers), motorcycles, buses, airplanes, helicopters, drones, lawnmowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, agricultural machinery or agricultural vehicles, construction machinery or construction vehicles, warehouse equipment or warehouse vehicles, factory equipment or factory vehicles, trams, golf carts, electric trains, trolleys, pedestrian transport vehicles, and robotic devices. Other vehicles are similarly possible. Furthermore, in some embodiments, exemplary systems may not include a vehicle.

[0021] Referring here to the figure, Figure 1 is a functional block diagram illustrating an exemplary vehicle 100, which may be configured to operate fully or partially in autonomous mode. More specifically, the vehicle 100 may operate in autonomous mode without human interaction by receiving control commands from a computing system. As part of its operation in autonomous mode, the vehicle 100 may use sensors to detect, and possibly detect, objects in its surrounding environment to enable safe navigation. Additionally, the exemplary vehicle 100 may operate in a partially autonomous (i.e., semi-autonomous) mode, in which some functions of the vehicle 100 are controlled by a human driver of the vehicle 100, and some functions of the vehicle 100 are controlled by a computing system. For example, the vehicle 100 may also include subsystems that allow the driver to control the operation of the vehicle 100, such as steering, acceleration, and braking, while the computing system implements assistance functions, such as lane departure warning / lane keeping assist or adaptive cruise control, based on other objects (e.g., other vehicles) in the surrounding environment.

[0022] As described herein, in a partially autonomous driving mode, the vehicle assists with one or more driving actions (e.g., steering, braking, and / or acceleration for lane centering, adaptive cruise control, advanced driver-assistance systems (ADAS), and emergency braking), but the human driver is expected to situationally perceive the vehicle's surroundings and supervise the assisted driving actions. Here, the vehicle may perform all driving tasks in certain situations, but the human driver is expected to take control as needed.

[0023] For the sake of simplification and brevity, various systems and methods are described below in conjunction with autonomous vehicles, but these or similar systems and methods may be used in various driver assistance systems that do not reach the level of fully autonomous driving systems (i.e., partially autonomous driving systems). In the United States, the Society of Automotive Engineers (SAE) defines different levels of automated driving behavior to indicate how much or how little a vehicle controls the driving, but different organizations in the United States or other countries may classify the levels differently. More specifically, the systems and methods of this disclosure may be used in SAE Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver assistance. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems that enable autonomous driving under limited conditions (e.g., highways). Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 automated driving systems that operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such systems, accurate lane estimation is performed automatically without driver input or control (e.g., while the vehicle is moving), resulting in improved reliability of vehicle positioning and navigation, as well as overall safety of autonomous, semi-autonomous, and other driver assistance systems. In addition to the way SAE classifies levels of autonomous driving operations as described above, other organizations in the United States or other countries may classify levels of autonomous driving operations differently. The systems and methods disclosed herein may be used in driver assistance systems defined by the levels of autonomous driving operations of these other organizations, but are not limited to these.

[0024] As shown in Figure 1, the vehicle 100 may be configured to include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112 (which may also be called a computing system) having data storage 114, and a user interface 116. In other examples, the vehicle 100 may be configured to include more or fewer subsystems, each of which may include multiple elements. The subsystems and components of the vehicle 100 may be interconnected in various ways. In addition, the functions of the vehicle 100 described herein may be divided into additional functional or physical components, or combined into fewer functional or physical components within an embodiment. For example, the control system 106 and the computer system 112 may be combined into a single system that operates the vehicle 100 according to various operations.

[0025] The propulsion system 102 may be configured to include one or more components that are capable of providing powered motion to the vehicle 100, and among other possible components, it may be configured to include an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121. For example, the engine / motor 118 may be configured to convert the energy source 119 into mechanical energy, and among other possible options, it may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine. For example, in some embodiments, the propulsion system 102 may be configured to include a number of types of engines and / or motors, such as a gasoline engine and an electric motor.

[0026] The energy source 119 represents an energy source that can, all or partly, power one or more systems of the vehicle 100 (e.g., engine / motor 118). For example, the energy source 119 may be gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, the energy source 119 may be configured to include a combination of a fuel tank, batteries, a capacitor, and / or a flywheel.

[0027] The transmission 120 may be configured to transmit mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. Therefore, the transmission 120 may include, among other possible components, a gearbox, clutch, differential, and drive shaft. The drive shaft may include an axle connected to one or more wheels / tires 121.

[0028] The wheels / tires 121 of the vehicle 100 may have various configurations within the exemplary embodiment. For example, the vehicle 100 may exist in the form of a unicycle, bicycle / motorcycle, tricycle, or four-wheeled automobile / truck, among other possible configurations. Thus, the wheels / tires 121 may be attached to the vehicle 100 in various ways and may exist in different materials such as metal and rubber.

[0029] The sensor system 104 may include various types of sensors, among other possible sensors, such as a Global Positioning System (GPS) 122, an Inertial Measurement Unit (IMU) 124, radar 126, lidar 128, camera 130, steering sensor 123, and throttle / brake sensor 125. In some embodiments, the sensor system 104 may also include sensors (e.g., O2 monitor, fuel gauge, engine oil temperature, and brake wear) configured to monitor the internal systems of the vehicle 100.

[0030] The GPS 122 may be configured to include a transceiver capable of operating to provide information regarding the position of the vehicle 100 relative to the Earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes, and may sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 while the vehicle 100 is stationary or in motion.

[0031] The radar 126 may represent one or more systems configured to sense objects in the surrounding environment of the vehicle 100, including their speed and orientation, using radio signals. Therefore, the radar 126 may include antennas configured to transmit and receive radio signals. In some embodiments, the radar 126 may correspond to a mountable radar configured to obtain measurements of the surrounding environment of the vehicle 100.

[0032] Lidar 128 may be configured to include, among other system components, one or more laser sources, a laser scanner, and one or more detectors, and may be configured to operate in coherent mode (e.g., using heterodyne detection) or incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of Lidar 128 may be particularly sensitive detectors (e.g., avalanche photodiodes), and may be configured to include one or more photodetectors. In some embodiments, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Furthermore, such photodetectors may be arranged in an array (e.g., like silicon photomultiplier tubes (SiPMs)) (e.g., through series electrical connections). In some embodiments, one or more photodetectors are devices operating in Geiger mode, and Lidar includes subcomponents designed for such Geiger mode operation.

[0033] The camera 130 may include one or more devices (e.g., a still camera, a video camera, a thermal imaging camera, a stereo camera, and a night vision camera) configured to capture images of the surrounding environment of the vehicle 100.

[0034] The steering sensor 123 may be configured to sense the steering angle of the vehicle 100, and the sensing may include measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, the steering sensor 123 may be configured to measure the angle of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the front axle of the vehicle 100. The steering sensor 123 may also be configured to measure a combination (or subset) of the angle of the steering wheel, an electrical signal representing the angle of the steering wheel, and the angle of the wheels of the vehicle 100.

[0035] The throttle / brake sensor 125 may be configured to detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angles of both the accelerator pedal (throttle) and the brake pedal, or it may be configured to measure an electrical signal representing, for example, the angle of the accelerator pedal (throttle) and / or the angle of the brake pedal. The throttle / brake sensor 125 may also measure the angle of the throttle body of the vehicle 100, which may include part of a physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., a butterfly valve and a carburetor). Additionally, the throttle / brake sensor 125 may measure the pressure of one or more brake pads on the rotor of the vehicle 100, or a combination (or subset) of the angles of the accelerator pedal (throttle) and the brake pedal, an electrical signal representing the angles of the accelerator pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied by at least one brake pad to the rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure the pressure applied to a vehicle pedal, such as the throttle or brake pedal.

[0036] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a brake unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / route search system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the direction of the vehicle 100, and the throttle 134 may control the acceleration of the vehicle 100 by controlling the operating speed of the engine / motor 118. The brake unit 136 can decelerate the vehicle 100, which may be configured to decelerate the wheels / tires 121 using friction. In some embodiments, the brake unit 136 may be configured to convert the kinetic energy of the wheels / tires 121 into an electric current for subsequent use by the vehicle 100's system or multiple systems.

[0037] The sensor fusion algorithm 138 may be configured to include a Kalman filter, a Bayesian network, or other algorithms capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may be configured to provide evaluations based on the received sensor data, such as evaluations of individual objects and / or features, evaluations of specific situations, and / or evaluations of possible effects within a given situation.

[0038] The computer vision system 140 may be configured to include hardware and software (e.g., a general-purpose processor such as a central processing unit (CPU), a dedicated processor such as a graphics processing unit (GPU) or tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) that can operate to process and analyze images to determine objects that are in motion (e.g., other vehicles, pedestrians, cyclists, or animals) and objects that are not in motion (e.g., streetlights, road boundaries, speed bumps, or potholes). Thus, the computer vision system 140 may be configured to use object recognition, structure from motion (SFM), video tracking, and other algorithms used in computer vision, such as recognizing objects, mapping environments, tracking objects, and estimating the speed of objects.

[0039] The navigation / route search system 142 can determine the driving route of the vehicle 100, and this may include a configuration that dynamically adjusts the navigation during operation. Thus, the navigation / route search system 142 may be configured to navigate the vehicle 100 using, among other sources, the sensor fusion algorithm 138, GPS 122, and data from a map. The obstacle avoidance system 144 evaluates potential obstacles based on sensor data and can cause the vehicle 100's system to avoid or otherwise navigate around potential obstacles.

[0040] As shown in Figure 1, the vehicle 100 may also include peripherals 108, such as a wireless communication system 146, a touchscreen 148, an internal microphone 150, and / or a speaker 152. The peripherals 108 may provide control or other elements for the user to interact with the user interface 116. For example, the touchscreen 148 may be configured to provide information to the user of the vehicle 100. The user interface 116 may also be configured to accept input from the user via the touchscreen 148. The peripherals 108 may also be configured to allow the vehicle 100 to communicate with devices, such as devices in other vehicles.

[0041] The wireless communication system 146 may be configured to communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may be configured to use 3G cellular communication such as Code Division Multiple Access (CDMA), Evolution Data Optimization (EVDO), Global System for Mobile Communications (GSM) / General-Purpose Packet Radio Service (GPRS), or cellular communication such as 4G Worldwide Interoperability for Microwave Access (WiMAX) or Long-Term Evolution (LTE), or 5G. Alternatively, the wireless communication system 146 may be configured to communicate with a wireless local area network (WLAN) using Wi-Fi® or other possible connections. The wireless communication system 146 may also be configured to communicate directly with devices using, for example, an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are possible within the context of this disclosure. For example, the wireless communication system 146 may be configured to include one or more dedicated narrow-area communication (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0042] The vehicle 100 may include a power supply 110 for supplying power to its components. In some embodiments, the power supply 110 may include a rechargeable lithium-ion or lead-acid battery. For example, the power supply 110 may include one or more batteries configured to provide power. The vehicle 100 may also use other types of power supplies. In an exemplary embodiment, the power supply 110 and the energy source 119 may be integrated to form a single energy source.

[0043] Vehicle 100 may also include a computer system 112 for performing operations such as those described herein. Accordingly, the computer system 112 may include at least one processor 113 (which may include at least one microprocessor) capable of operating to execute instructions 115 stored in a non-temporary computer-readable medium, such as data storage 114. In some embodiments, the computer system 112 may represent a plurality of computing devices capable of functioning to control individual components or subsystems of vehicle 100 in a distributed manner.

[0044] In some embodiments, the data storage 114 may be configured to include instructions 115 (e.g., program logic) executable by the processor 113 for performing various functions of the vehicle 100, including those described above in relation to Figure 1. The data storage 114 may also be configured to include additional instructions, including instructions for transmitting, receiving, interacting with, and / or controlling data to one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0045] In addition to instruction 115, the data storage 114 may be configured to store data such as road maps and route information, among other information. Such information may be used by the vehicle 100 and the computer system 112 during the operation of the vehicle 100 in autonomous mode, semi-autonomous mode, and / or manual mode.

[0046] Vehicle 100 may be configured to include a user interface 116 for providing information to or receiving input from the user of vehicle 100. The user interface 116 may be configured to control or enable control of the layout of content and / or interactive images that may be displayed on the touchscreen 148. Furthermore, the user interface 116 may be configured to include one or more input / output devices within a set of peripherals 108, such as a wireless communication system 146, a touchscreen 148, a microphone 150, and a speaker 152.

[0047] The computer system 112 may be configured to control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, or the control system 106) and from the user interface 116. For example, the computer system 112 may be configured to utilize inputs from the sensor system 104 to estimate the outputs generated by the propulsion system 102 and the control system 106. Depending on the embodiment, the computer system 112 may be operable to monitor many aspects of the vehicle 100 and its subsystems. In some embodiments, the computer system 112 may be configured to disable some or all of the functions of the vehicle 100 based on signals received from the sensor system 104.

[0048] The components of the vehicle 100 may be configured to function in a manner that interconnects with other components, either internally or externally, to each of their respective systems. For example, in an exemplary embodiment, the camera 130 may be configured to capture multiple images that can represent information about the state of the surrounding environment of the vehicle 100 operating in autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, the computer vision system 140 may be configured to recognize incline (gradient) or other features based on multiple images of the road. Additionally, the combination of the GPS 122 and the features recognized by the computer vision system 140 may be used together with map data stored in the data storage 114 to determine specific road parameters. Furthermore, radar 126, and / or lidar 128, and / or several other environmental mapping, range, and / or positioning sensor systems may also be configured to provide information about the vehicle's surroundings.

[0049] In other words, a combination of various sensors (which can be called input indicator sensors and output indicator sensors) and the computer system 112 can interact to provide an indicator of input or an indicator of the vehicle's surroundings, which is provided for controlling the vehicle.

[0050] In some embodiments, the computer system 112 may be configured to perform detections of various objects based on data provided by systems other than the wireless system. For example, the vehicle 100 may be configured to have lasers or other optical sensors configured to sense objects within the vehicle's field of view. The computer system 112 may be configured to use the outputs from various sensors to detect information about objects within the vehicle's field of view, and may be configured to measure distance and directional information to various objects. The computer system 112 may also be configured to determine whether an object is desirable or undesirable based on the outputs from various sensors.

[0051] Figure 1 shows various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) integrated into vehicle 100, but one or more of these components may be mounted on vehicle 100 or provided separately. For example, data storage 114 may be configured to exist partially or completely separately from vehicle 100. Thus, vehicle 100 may be provided in the form of device elements located in different locations, or in the form of device elements located together in the same location. The device elements constituting vehicle 100 may be coupled together in a wired and / or wireless manner so as to be able to communicate together.

[0052] Figures 2A–2E show an exemplary vehicle 200 (e.g., a fully autonomous vehicle or a semi-autonomous vehicle) which may include some or all of the functions described in relation to vehicle 100 with reference to Figure 1. For illustrative purposes, vehicle 200 is illustrated in Figures 2A–2E as a van with side mirrors, but the disclosure is not limited thereto. For example, vehicle 200 may represent a truck, passenger car, semi-trailer truck, motorcycle, golf cart, off-road vehicle, agricultural vehicle, or any other vehicle described elsewhere in this specification (e.g., bus, boat, airplane, helicopter, drone, lawnmower, bulldozer, submarine, all-terrain vehicle, snowmobile, aircraft, recreation vehicle, amusement park vehicle, farm equipment, construction machinery or construction vehicle, warehouse equipment or warehouse vehicle, factory equipment or factory vehicle, tram, train, trolley, pedestrian transport vehicle, and robotic device).

[0053] An exemplary vehicle 200 may include one or more sensor systems, 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more LiDAR, one or more radar, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding the vehicle 200. In other words, any sensor system currently known or to be created in the future may be coupled to the vehicle 200 and / or used in conjunction with various operations of the vehicle 200. For example, LiDAR may be used for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of the vehicle 200. In addition, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent combinations of sensors described herein (e.g., one or more lidar and radar, one or more lidar and camera, one or more camera and radar, one or more lidar, camera, and radar).

[0054] It should be noted that the number, location, and type of sensor systems (e.g., 202 and 204) depicted in Figures 2A-E are intended as non-limiting embodiments of the location, number, and type of such sensor systems in autonomous or semi-autonomous vehicles. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to reduce vehicle size, shape, aerodynamics, fuel economy, aesthetics, or cost, or to adapt to special environments or other conditions for application). For example, sensor systems (e.g., 202 and 204) may be placed in various other locations on the vehicle (e.g., at location 216) and may have a field of view corresponding to the interior of the vehicle 200 and / or the surrounding environment.

[0055] The sensor system 202 may include one or more sensors mounted on top of the vehicle 200 and configured to detect information about the environment surrounding the vehicle 200 and output an index of that information. For example, the sensor system 202 may include any combination of cameras, radar, lidar, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). The sensor system 202 may include one or more movable mounts that are operable to adjust the orientation of one or more sensors within the sensor system 202. In one embodiment, the movable mounts may include a rotating platform that can scan the sensors to acquire information from each direction around the vehicle 200. In another embodiment, the movable mounts of the sensor system 202 may be movable in a scanning manner within a range of a specific angle and / or azimuth and / or elevation. The sensor system 202 may be configured to be mounted on the roof of the vehicle, although other mounting locations are also possible.

[0056] Furthermore, the sensors of the sensor system 202 may be configured to be distributed in different locations and do not need to be placed side by side in a single location. In addition, each sensor of the sensor system 202 may be configured to move or scan independently of the other sensors of the sensor system 202. Additionally or alternatively, multiple sensors may be mounted on one or more of the sensor locations 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two LiDAR devices mounted on the sensor locations, and / or one LiDAR device and one radar mounted on the sensor locations.

[0057] One or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to include one or more LiDAR devices. For example, a LiDAR device may include multiple light emitter devices arranged over a range of angles with respect to a given plane (e.g., the xy-plane). For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot around an axis perpendicular to a given plane (e.g., the z-axis) to illuminate the environment surrounding the vehicle 200 with light pulses. Information about the surrounding environment can be determined based on the detection of various aspects of the reflected light pulses (e.g., elapsed time of flight, polarization, and intensity).

[0058] In exemplary embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide point cloud information that may relate to physical objects in the surrounding environment of the vehicle 200. While the vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are shown as including certain features, it will be understood that other types of sensor systems are contemplated within the scope of this disclosure. Furthermore, the exemplary vehicle 200 may include any of the components described in relation to the vehicle 100 in Figure 1.

[0059] In exemplary configurations, one or more radars may be located on the vehicle 200. Similar to radar 126 described above, one or more radars may include antennas configured to transmit and receive radio waves (e.g., electromagnetic waves having frequencies between 30 Hz and 300 GHz). Such radio waves may be used to determine the distance and / or speed of one or more objects in the environment surrounding the vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some examples, one or more radars may be located near the rear of the vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of the vehicle 200 for the presence of radio wave reflecting objects. Similarly, one or more radars may be positioned near the front of the vehicle 200 (e.g., sensor systems 212, 214) to actively scan the environment near the front of the vehicle 200. The radars may be positioned in a location suitable for illuminating an area including the forward path of the vehicle 200 without being obstructed by other features of the vehicle 200. For example, the radars may be embedded in the front bumper, front headlights, cowl, and / or hood, and / or mounted on or near them. Furthermore, one or more additional radars may be positioned on or near the rear bumper, side panels, rocker panels, and / or lower running gear, for example, by including such devices, to actively scan the sides and / or rear of the vehicle 200 for the presence of radio wave reflecting objects.

[0060] The vehicle 200 may include one or more cameras. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more cameras. The cameras may be photosensitive devices such as still cameras, video cameras, thermal imaging cameras, stereo cameras, and night vision cameras, configured to capture multiple images of the environment surrounding the vehicle 200. For this purpose, the cameras may be configured to detect visible light, and additionally or alternatively, to detect light from other parts of the spectrum, such as infrared or ultraviolet light. The cameras may be two-dimensional detectors, and optionally, may be configured to have a sensitivity range in three-dimensional space. In some embodiments, the cameras may be configured to include range detectors, configured to produce a two-dimensional image showing, for example, the distance from the camera to several points in the surrounding environment. For this purpose, the cameras may use one or more range detection techniques. For example, the camera can provide range information by using a structured light technique, in which the vehicle 200 illuminates objects in the surrounding environment with a predetermined light pattern, such as a grid or checkerboard pattern, and the camera is used to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion aberration of the reflected light pattern, the vehicle 200 can determine the distance to a point on the object. The predetermined light pattern may include infrared light or radiation of other wavelengths suitable for such measurements. In some examples, the camera may be mounted inside the windshield of the vehicle 200. Specifically, the camera may be positioned to capture an image from a forward view relative to the orientation of the vehicle 200. Other mounting positions and field of view of the camera may be used either inside or outside the vehicle 200. The camera may also have associated optical elements that are operable to provide an adjustable field of view. Furthermore, the camera may be mounted on the vehicle 200 using a movable mount to change the camera's directional angle via a pan / tilt mechanism, etc.

[0061] Vehicle 200 may also include one or more acoustic sensors used to sense the environment surrounding Vehicle 200 (for example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). The acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, and microelectromechanical system (MEMS) microphones) used to sense acoustic waves (i.e., pressure differences) in the fluid (e.g., air) of the environment surrounding Vehicle 200. Such acoustic sensors may be used to detect sounds in the surrounding environment (e.g., sirens, human speech, animal sounds, and alarms) on which the control strategy of Vehicle 200 may be based. For example, if an acoustic sensor detects a siren (e.g., an ambulance siren and / or a fire truck siren), Vehicle 200 may slow down and / or navigate to the edge of the road.

[0062] Although not shown in Figures 2A-2E, the vehicle 200 may include a wireless communication system (e.g., similar to and / or in addition to the wireless communication system 146 in Figure 1). The wireless communication system may include a wireless transmitter and a wireless receiver, which may be configured to communicate with devices outside or inside the vehicle 200. Specifically, the wireless communication system may include, for example, a vehicle communication system or a transceiver configured to communicate with other vehicles and / or computing devices at a road station. Examples of such vehicle communication systems include DSRC, radio frequency detection (RFID), and other communication standards proposed for intelligent transport systems.

[0063] Vehicle 200 may include, in addition to or instead of, these indicated components, one or more other components. These additional components may include electrical or mechanical functions.

[0064] The control system of the vehicle 200 may be configured to control the vehicle 200 according to a control strategy selected from among several possible control strategies. The control system may also be configured to receive information from sensors coupled to the vehicle 200 (on or outside the vehicle 200), modify the control strategy (and associated driving behavior) based on that information, and control the vehicle 200 according to the modified control strategy. The control system may further be configured to monitor the information received from the sensors and continuously evaluate the driving conditions, and may also be configured to modify the control strategy and driving behavior based on changes in the driving conditions. For example, the route taken by the vehicle from one destination to another may be modified based on the driving conditions. Additionally or alternatively, speed, acceleration, turning angle, inter-vehicle distance (i.e., distance to the vehicle in front of the current vehicle), lane selection, etc., may all be modified in response to changes in driving conditions.

[0065] As described above, in some embodiments, the vehicle 200 may take the form of a van, but alternative forms are also possible and intended herein. Accordingly, Figures 2F to 2I illustrate embodiments in which the vehicle 250 takes the form of a semi-track. For example, Figure 2F illustrates a front view of the vehicle 250, and Figure 2G illustrates an isometric view of the vehicle 250. In embodiments in which the vehicle 250 is a semi-track, the vehicle 250 may include a tractor portion 260 and a trailer portion 270 (illustrated in Figure 2G). Figures 2H and 2I provide a side view and a top view of the tractor portion 260, respectively. Similar to the vehicle 200 illustrated above, the vehicle 250 illustrated in Figures 2F to 2I may also include various sensor systems (for example, sensor systems 202, 206, 208, 210, 212, and 214 shown and described with reference to Figures 2A to 2E). In some embodiments, the vehicle 200 in Figures 2A-2E may include only a single copy of several sensor systems (e.g., sensor system 204), while the vehicle 250 illustrated in Figures 2F-2I may include multiple copies of its sensor system (e.g., sensor systems 204A and 204B as illustrated).

[0066] While the drawings and overall description may refer to a given vehicle configuration (e.g., a semi-truck vehicle 250 or a van vehicle 200), it should be understood that the embodiments described herein are equally applicable in the context of various vehicles (e.g., with modifications adopted to take into account the vehicle's form factor). For example, sensors and / or other components described or illustrated as part of a van vehicle 200 may also be used in a semi-truck vehicle 250 (e.g., for navigation and / or obstacle detection and avoidance).

[0067] Figure 2J illustrates various sensor fields of view (e.g., associated with the vehicle 250 described above). As described above, the vehicle 250 may contain multiple sensors / sensor units. The locations of the various sensors may correspond, for example, to the sensor locations disclosed in Figures 2F to 2I. However, in some examples, sensors may be located elsewhere. For the sake of simplicity in the drawings, sensor location reference numbers are omitted from Figure 2J. For each sensor unit of the vehicle 250, Figure 2J illustrates typical fields of view (e.g., fields of view labeled as 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The sensor field of view may include angular regions (e.g., azimuth and / or elevation regions) in which the sensor can detect objects.

[0068] Figure 2K illustrates beam steering for sensors in a vehicle (e.g., vehicle 250 shown and described with reference to Figures 2F-2J) according to an exemplary embodiment. In various embodiments, the sensor unit of vehicle 250 may be radar, lidar, sonar, etc. Furthermore, in some embodiments, while the sensor is operating, the sensor may be configured to perform scanning within the sensor's field of view. Various different scanning angles for the exemplary sensor are shown as regions 272, each indicating the angular region in which the sensor is operating. The sensor may be configured to periodically or iteratively change the region in which it is operating. In some embodiments, multiple sensors may be used by vehicle 250 to measure regions 272. In other embodiments, other regions may be included in the configuration. For example, one or more sensors may measure the aspect of the trailer 270 of vehicle 250 and / or the region in front of vehicle 250.

[0069] At certain angles, the sensor's operating area 275 may be configured to include the rear wheels 276A and 276B of the trailer 270. Therefore, the sensor may measure the rear wheels 276A and / or 276B during operation. For example, the rear wheels 276A and 276B may reflect lidar or radar signals transmitted by the sensor. The sensor may be configured to receive signals reflected from the rear wheels 276A and 276. Thus, the data collected by the sensor may be configured to include data from reflections from the wheels.

[0070] In some cases, such as when the sensor is radar, reflections from the rear wheels 276A and 276B may appear as noise in the received radar signal. As a result, the radar may operate with an enhanced signal-to-noise ratio in cases where the rear wheels 276A and 276B direct the radar signal away from the sensor.

[0071] Figure 3 is a conceptual diagram illustrating wireless communication between various computing systems related to autonomous or semi-autonomous vehicles, according to an exemplary embodiment. In particular, wireless communication may be configured to occur between a remote computing system 302 and a vehicle 200 via a network 304. Alternatively, wireless communication may occur between a server computing system 306 and a remote computing system 302, and between the server computing system 306 and a vehicle 200.

[0072] Vehicle 200 can accommodate various types of vehicles capable of transporting passengers or objects between locations, and can take any one or more forms of the vehicles considered above. In some examples, vehicle 200 may be configured to operate in autonomous or semi-autonomous mode, where a control system uses sensor measurements to enable safe navigation of vehicle 200 between destinations. When operating in autonomous or semi-autonomous mode, vehicle 200 may be configured to navigate with or without passengers. As a result, vehicle 200 can pick up and drop off passengers between desired destinations.

[0073] The remote computing system 302 may represent any type of device relating to remote assistance technology, including but not limited to those described herein. In examples, the remote computing system 302 may represent any type of device configured to (i) receive information relating to the vehicle 200, (ii) provide an interface through which a human operator can then become aware of the information and input a response relating to the information, and (iii) transmit the response to the vehicle 200 or to another device. The remote computing system 302 can take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some examples, the remote computing system 302 may be configured to include multiple computing devices operating together in a network configuration.

[0074] The remote computing system 302 may consist of one or more subsystems and components similar to, or identical to, those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface, such as input / output devices, including a touchscreen and speakers. Other embodiments are equally possible.

[0075] Network 304 represents the infrastructure that enables wireless communication between the remote computing system 302 and the vehicle 200. Network 304 also enables wireless communication between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0076] The location of the remote computing system 302 can vary within the scope of the embodiment. For example, the remote computing system 302 may be located remotely from the vehicle 200, having wireless communication via the network 304. In another embodiment, the remote computing system 302 may correspond to a computing device within the vehicle 200, separate from the vehicle 200, that a human operator can interact with the passengers or driver of the vehicle 200. In some embodiments, the remote computing system 302 may be a computing device equipped with a touchscreen that can be operated by the passengers of the vehicle 200.

[0077] In some embodiments, the operations described herein, which are performed by the remote computing system 302, may be performed by the vehicle 200 (i.e., by any system or subsystem of the vehicle 200) as an additional or alternative configuration. In other words, the vehicle 200 may be configured to provide a remote assistance mechanism that can be interacted with by the vehicle's driver or passengers.

[0078] The server computing system 306 may be configured to communicate wirelessly with the remote computing system 302 and the vehicle 200 (or, optionally, directly with the remote computing system 302 and / or the vehicle 200) via the network 304. The server computing system 306 may represent any computing device configured to receive, store, detect, and / or transmit information about the vehicle 200 and its remote assistance. Thus, the server computing system 306 may be configured to perform any operation or part of such operation described herein as being performed by the remote computing system 302 and / or the vehicle 200. The server computing system 306 may be available in some embodiments of the wireless communication related to remote assistance, but not in other embodiments.

[0079] The server computing system 306 may be configured to include one or more subsystems and components similar to or identical to those of the remote computing system 302 and / or vehicle 200, such as a processor configured to perform the various operations described in this disclosure, and a wireless communication interface for receiving and providing information to the remote computing system 302 and the vehicle 200.

[0080] The various systems described above may be configured to perform a variety of operations. These operations and their associated characteristics are described below.

[0081] In line with the above considerations, the computing system (e.g., remote computing system 302, server computing system 306, or computing system local to vehicle 200) may be configured to operate using cameras to capture images of the environment surrounding the autonomous or semi-autonomous vehicle. Generally, at least one computing system can analyze the images and, if possible, control the autonomous or semi-autonomous vehicle.

[0082] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data (also referred to in this disclosure as “environmental data”) representing objects in the environment surrounding the vehicle in various ways. The vehicle’s sensor system may be configured to provide environmental data representing objects in the surrounding environment. For example, the vehicle may be configured to include various sensors such as cameras, radar, lidar, microphones, wireless units, and other sensors. Each of these sensors may be configured to communicate environmental data about the information it has detected to a processor in the vehicle.

[0083] In one embodiment, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have two or more cameras positioned in different orientations. Also in some embodiments, the cameras may be movable to capture images and / or video in different orientations. The cameras may be configured to store the captured images and video in memory for subsequent processing by the vehicle's processing system. The captured images and / or video may be environmental data. Furthermore, the cameras may be configured to include an image sensor, as described herein.

[0084] In another embodiment, the radar may be configured to transmit electromagnetic signals reflected by various objects near the vehicle and then capture electromagnetic signals reflected from those objects. The captured reflected electromagnetic signals may enable the radar (or processing system) to perform various detections about the objects that reflected the electromagnetic signals. For example, it may be configured to measure the distance and position to various reflective objects. In some embodiments, the vehicle may be configured to have two or more radars oriented in different directions. The radar may be configured to store the captured information in memory for subsequent processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0085] In another embodiment, the lidar may be configured to transmit electromagnetic signals (e.g., infrared light, such as from a gas or diode laser, or other possible light source) reflected by target objects near the vehicle. The lidar may also be capable of capturing the reflected electromagnetic (e.g., infrared) signals. The captured reflected electromagnetic signals may enable a ranging system (or processing system) to measure the distance to various objects. The lidar can also determine the velocity or speed of target objects and store this as environmental data.

[0086] Additionally, in one embodiment, the microphone may be configured to capture audio from the vehicle's surrounding environment. The sounds captured by the microphone may include emergency vehicle sirens and sounds from other vehicles. For example, the microphone may be configured to capture the sounds of ambulance, fire engine, or police vehicle sirens. The processing system may be able to detect that the captured audio signal indicates an emergency vehicle. In another embodiment, the microphone may be configured to capture the sounds of exhaust from another vehicle, such as exhaust from a motorcycle. The processing system may be able to detect that the captured audio signal indicates a motorcycle. The data captured by the microphone may form part of the environmental data.

[0087] In yet another embodiment, the radio unit may be configured to transmit an electromagnetic signal, which may take the form of a Bluetooth signal, an 802.11 signal, and / or other radio technology signal. The first electromagnetic radiation signal may be configured to be transmitted via one or more antennas located on the radio unit. Furthermore, the first electromagnetic radiation signal may be configured to be transmitted in one of many different radio signal modes. However, in some embodiments, it is desirable to transmit the first electromagnetic radiation signal in a signal mode that requests a response from a device located near an autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on the response returned to the radio unit and use this communicated information as part of the environmental data.

[0088] In some embodiments, the processing system may be able to combine information from various sensors to further detect the vehicle's surrounding environment. For example, the processing system may be configured to combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, other combinations of sensor data may be used by the processing system to make determinations about the surrounding environment.

[0089] While operating in autonomous (or semi-autonomous) mode, a vehicle may be configured to control its actions with little or no human input. For example, if a human operator enters an address into the vehicle, the vehicle may be able to drive to the designated destination without further human input (e.g., without the human needing to operate or touch the brake / accelerator pedals). Furthermore, while the vehicle is operating autonomously or semi-autonomously, the sensor system may be configured to receive environmental data. The vehicle's processing system may be configured to modify the vehicle's control based on environmental data received from various sensors. In some embodiments, the vehicle may be configured to change its speed in response to environmental data from various sensors. The vehicle may be configured to change its speed to avoid obstacles, comply with traffic laws, etc. When the processing system in the vehicle detects an object near the vehicle, the vehicle may be able to change its speed or otherwise alter its movement.

[0090] If a vehicle detects an object but lacks sufficient confidence in its detection, the vehicle may be configured to request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object is actually present in the surrounding environment (e.g., whether there is actually a stop sign or not), (ii) verifying whether the vehicle's object detection is correct, (iii) correcting the detection if it was incorrect, and / or (iv) providing supplementary instructions for the autonomous or semi-autonomous vehicle (or modifying the current instructions). Remote assistance tasks may also include providing instructions for the human operator to control the vehicle's actions (e.g., if the human operator detects that the object is a stop sign, they instruct the vehicle to stop at the stop sign), although in some scenarios, the vehicle itself may control its own actions based on human operator feedback related to object detection.

[0091] To facilitate this, the vehicle may be configured to analyze environmental data representing objects in the surrounding environment and detect at least one object with a detection confidence level below a threshold. The vehicle's processor may be configured to detect various objects in the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, cyclists, street signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the acquired environmental data.

[0092] The detection confidence level may be configured to indicate the likelihood that the detected object is correctly detected or present in the surrounding environment. For example, the processor may perform object detection on objects in the image data of the received environmental data, and if at least one object cannot be detected, it may be configured to determine that the object has a detection confidence level below the threshold. If the result of object detection or object recognition is inconclusive, the detection confidence level may be low or below the set threshold.

[0093] Depending on the source of the environmental data, the vehicle may detect objects in its surroundings in various ways. In some embodiments, the environmental data may be image or video data coming from a camera. In other embodiments, the environmental data may come from LiDAR. The vehicle may be configured to analyze the captured image or video data to detect objects in the image or video data. The method and apparatus may be configured to monitor the image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in the surrounding environment based on radar, audio, or other data.

[0094] In some embodiments, the technique used by the vehicle to detect objects may be based on a set of known data. For example, data related to environmental objects may be stored in memory located in the vehicle. The vehicle may be configured to compare the received data with the stored data to make a determination about the object. In other embodiments, the vehicle may be configured to make a determination about the object based on the context of the data. For example, road signs related to construction may generally be orange in color. Therefore, the vehicle may be configured to detect orange objects located near the side of the road as road signs related to construction. The vehicle's processing system may also be configured to calculate the confidence level of each object once it has detected an object in the captured data.

[0095] Furthermore, the vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object being detected. For example, the confidence threshold may be lower for objects that may require a quick response action from the vehicle, such as the brake lights of another vehicle. However, in other embodiments, the confidence threshold may be the same for all detected objects. If the confidence associated with a detected object is higher than the confidence threshold, the vehicle may assume that the object has been correctly recognized and, based on that assumption, responsively adjust the vehicle's control accordingly.

[0096] If the confidence level associated with a detected object is lower than the confidence threshold, the action taken by the vehicle may change. In some embodiments, the vehicle may be configured to react as if the detected object exists, despite the low confidence level. In other embodiments, the vehicle may be configured to react as if the detected object does not exist.

[0097] When a vehicle detects an object in its surrounding environment, it can also calculate a confidence score to associate it with a specific detected object. The confidence score can be calculated in various ways depending on the embodiment. In one embodiment, when an object is detected in the surrounding environment, the vehicle may be configured to compare environmental data with predetermined data associated with known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence score. In another embodiment, the vehicle may be configured to calculate the confidence score associated with an object using a mathematical analysis of the environmental data.

[0098] In response to detection that an object has a detection confidence level below a threshold, the vehicle may be configured to send a request for remote assistance to a remote computing system along with the detection of the object. As considered above, the remote computing system can take various forms. For example, the remote computing system may be a computing device located in the vehicle, separate from the vehicle itself, but which may be a touchscreen interface for displaying remote assistance information, allowing a human operator to interact with the vehicle's passengers or driver. As an additional or alternative configuration, in another embodiment, the remote computing system may be a remote computer terminal or other device located not near the vehicle.

[0099] Remote assistance requests may include environmental data, such as image data and audio data, which may include objects. The vehicle may be configured to transmit the environmental data to a remote computing system via a network (e.g., network 304) and, in some embodiments, via a server (e.g., server computing system 306). A human operator in the remote computing system may then be configured to use the environmental data as a basis for responding to the request.

[0100] In some embodiments, if an object is detected as having a confidence level below a confidence threshold, the object may be given a preliminary detection, and the vehicle may be configured to adjust its operation in response to the preliminary detection. Such adjustments to operation may take the form of stopping the vehicle, switching the vehicle to a human-controlled mode, or changing the vehicle's speed (e.g., speed and / or direction), among other possible adjustments.

[0101] In other embodiments, even if the vehicle detects an object with a confidence level that meets or exceeds a threshold, the vehicle may act according to the detected object (for example, stop if the object is detected with high confidence as a stop sign), but the vehicle may be configured to request remote assistance at the same time as (or after) acting according to the detected object.

[0102] Figure 4A is a block diagram of a system according to an exemplary embodiment. In particular, Figure 4A shows a system 400 including a system controller 402, a lidar device 410, a number of sensors 412, and a number of controllable components 414. The system controller 402 includes a processor(s) 404, a memory 406, and instructions 408 stored on the memory 406 and executable by the processor(s) 404 to perform functions.

[0103] The processor 404 may include one or more processors, such as one or more general-purpose microprocessors (e.g., having single-core or multi-core) and / or one or more dedicated microprocessors. The one or more processors may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more image processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein.

[0104] Memory 406 may include, but is not limited to, computer-readable media such as non-temporary computer-readable media, including read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), solid-state drives (SSDs), hard disk drives (HDDs), compact discs (CDs), digital video discs (DVDs), digital tapes, read / write (R / W) CDs, R / W DVDs, etc.

[0105] The LiDAR device 410, further described below, includes a plurality of light emitters configured to emit light (e.g., in light pulses), and one or more photodetectors configured to detect light (e.g., the reflected portion of the light pulses). The LiDAR device 410 may generate three-dimensional (3D) point cloud data from the output of the photodetector(s) and provide the 3D point cloud data to the system controller 402. The system controller 402 may then perform operations on the 3D point cloud data to determine the characteristics of the surrounding environment (e.g., relative positions of objects in the surrounding environment, edge detection, object detection, and / or proximity sensing).

[0106] Similarly, the system controller 402 may use the outputs from multiple sensors 412 to determine the characteristics of the system 400 and / or the surrounding environment. For example, the sensors 412 may include one or more of a GPS, an IMU, an image capture device (e.g., a camera), a light sensor, a thermal sensor, and other sensors that indicate parameters related to the system 400 and / or the surrounding environment. The lidar device 410 is depicted separately from the sensors 412 for illustrative purposes and may, in some embodiments, be considered as part of or as part of the sensors 412.

[0107] Based on the characteristics of the surrounding environment determined by the system controller 402 based on the outputs from system 400 and / or the LiDAR device 410 and sensor 412, the system controller 402 may control the controllable components 414 to perform one or more actions. For example, system 400 may correspond to a vehicle, in which case the controllable components 414 may include the vehicle's braking system, turning system, and / or acceleration system, and the system controller 402 may change the configuration of these controllable components based on the characteristics determined from the LiDAR device 410 and / or sensor 412 (for example, when the system controller 402 controls the vehicle in autonomous or semi-autonomous mode). In this example, the LiDAR device 410 and sensor 412 are also controllable by the system controller 402.

[0108] Figure 4B is a block diagram of a lidar device according to an exemplary embodiment. In particular, Figure 4B shows a lidar device 410 having a controller 416 configured to control a plurality of optical emitters 424 and one or more photodetectors, for example, a plurality of photodetectors 426. The lidar device 410 further includes a firing circuit 428 configured to select and provide power to each of the plurality of optical emitters 424, and may also include a selector circuit 430 configured to select each of the plurality of photodetectors 426. The controller 416 includes a processor 418, a memory 420, and instructions 422 stored on the memory 420.

[0109] Similar to processor(s) 404, processor(s) 418 may include one or more processors, such as one or more general-purpose microprocessors and / or one or more dedicated microprocessors. One or more processors may include, for example, one or more CPUs, one or more microcontrollers, one or more GPUs, one or more TPUs, one or more ASICs, and / or one or more FPGAs. Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein.

[0110] Similar to memory 406, memory 420 may include, but is not limited to, computer-readable media such as ROM, PROM, EPROM, EEPROM, non-volatile random access memory (e.g., flash memory), SSD, HDD, CD, DVD, digital tape, R / WCD, R / WDVD, and other non-temporary computer-readable media.

[0111] Instruction 422 is stored in memory 420 and is executable by processor 418, and performs functions related to controlling firing circuit 428 and selector circuit 430 for generating 3D point cloud data and processing the 3D point cloud data (or possibly to facilitate processing of the 3D point cloud data by another computing device, such as system controller 402).

[0112] The controller 416 can determine 3D point cloud data by using the light emitters 424 to emit pulses of light. The emission time is established for each light emitter, and the relative position at the emission time is also tracked. Various aspects of the environment surrounding the LiDAR device 410, such as objects, reflect the pulses of light. For example, if the LiDAR device 410 is in an environment that includes a road, such objects may include vehicles, signs, pedestrians, road surfaces, construction cones, etc. Some objects may be more reflective than others, such that the intensity of the reflected light may indicate the type of object reflecting the light pulse. Furthermore, the surfaces of objects may be in different positions relative to the LiDAR device 410, and therefore it may take some time for them to reflect a portion of the light pulse back to the LiDAR device 410. Thus, the controller 416 can track the detection time at which the reflected light pulse is detected by the photodetector, and the relative position of the photodetector at the detection time. By measuring the time difference between the emission time and the detection time, the controller 416 can determine how far the light pulse travels before being received, and therefore the relative distance to the corresponding object. By tracking the relative positions at the emission and detection times, the controller 416 can determine the orientation of the light pulse and the reflected light pulse relative to the LiDAR device 410, and therefore the relative orientation of the object. By tracking the intensity of the received light pulse, the controller 416 can determine how reflective the object is. Based on this information, the 3D point cloud data can therefore show the relative position of the detected reflected light pulse (e.g., in a coordinate system such as a Cartesian coordinate system) and the intensity of each reflected light pulse.

[0113] The firing circuit 428 is used to select an optical emitter to emit an optical pulse. Similarly, the selector circuit 430 is used to sample the output from the photodetector.

[0114] As described above, a camera system may include multiple cameras and / or sensors, each having different characteristics based on differences in one or more components of the camera. For example, each camera may include one or more lenses housed within the camera barrel. Each camera may also include one or more image sensors. Each image sensor may receive light from the scene through one or more lenses and mirrors housed within the lens barrel. The camera may also include additional components (e.g., a shutter button, a viewfinder, a flash, a battery, an electronic storage device for recording captured images, a display screen, and selection buttons).

[0115] In some embodiments, one or more camera barrels may be configured to rotate around their axis to correct the relative positions of one or more lenses within the barrel, thereby adjusting the camera's field of view and / or zoom. For example, one or more camera components may be electronically controlled (e.g., a camera controller may adjust one or more lenses within the barrel to correct the camera's zoom, for example, during an autofocus procedure).

[0116] The camera may include an image sensor only behind one or more lenses (e.g., a telecentric lens). Other arrangements are also possible. In some embodiments, for example, the camera may include one or more optical filters (e.g., a polarizing filter, a color filter, or a neutral density filter), and / or one or more motorized stages and / or motors configured to adjust the position of one or more components of the camera.

[0117] As described above, in some embodiments, cameras may be used for object detection and avoidance within an autonomous vehicle (for example, camera 130 as illustrated and described with reference to Figure 1). Figures 5A to 5D show exemplary cameras that may be incorporated (for example, in various numbers and / or positions) into the camera systems disclosed herein.

[0118] Figure 5A is an illustrative diagram of a first camera type 502 camera having an associated first field of view 503 according to an exemplary embodiment. In some embodiments, the first camera type 502 camera may be mounted on a vehicle such as vehicle 100. In some embodiments, the image captured by the first camera type 502 camera can be used to identify objects within a first distance range from the vehicle.

[0119] Figure 5B is an illustrative diagram of a second camera type 504 camera having an associated second field of view 505 according to an exemplary embodiment. The second camera type 504 camera may be mounted on a vehicle such as vehicle 100. The second field of view 505 may extend to a larger angle (e.g., angular direction around the z-axis) in the yaw direction relative to vehicle 100 than the first field of view 503. In some embodiments, the second field of view 505 extends to at least 170 degrees in the yaw direction relative to vehicle 100. Images captured by the second camera type 504 camera may be used to identify objects in a second distance range from vehicle 100. In some embodiments, the first distance range includes distances further from vehicle 100 than the distances included in the second distance range.

[0120] Figure 5C is an illustrative diagram of a third camera type 506 camera having an associated third field of view 507 according to an exemplary embodiment. The third camera type 506 camera may be mounted on a vehicle such as vehicle 100. In some embodiments, the combined field of view of multiple cameras of the third camera type 506 may extend to 360 degrees in yaw relative to vehicle 100. The first field of view 503 may extend to a smaller angle in the yaw direction relative to vehicle 100 than the third field of view 507. The third field of view 507 may extend to a smaller angle in the yaw direction relative to vehicle 100 than the second field of view 505. Images captured by the third camera type 506 camera may be used to identify objects in a third distance range from vehicle 100. In some embodiments, the third distance range may include distances further from vehicle 100 than the distances included in the second distance range. The first distance range may include distances further from vehicle 100 than the distances included in the third distance range.

[0121] The cameras of the first camera type 502, the second camera type 504, and / or the third camera type 506 may have camera lenses with a focal length in the range of approximately 1.5 mm to approximately 25 mm. Furthermore, the cameras of the first camera type 502, the second camera type 504, and / or the third camera type 506 may have a total dynamic range of less than 120 decibels. The camera of the first camera type 502 may have an image resolution of approximately 17 megapixels. The camera of the second camera type 504 may have an image resolution of approximately 2 megapixels. The camera of the third camera type 506 may have an image resolution of approximately 8 megapixels.

[0122] Figure 5D is an illustrative diagram of a camera of a fourth camera type 508 having an associated fourth field of view 509, according to an exemplary embodiment. The camera of the fourth camera type 508 may be mounted on a vehicle such as vehicle 100. The fourth field of view 509 may extend to a larger angle in the yaw direction relative to vehicle 100 than the first field of view 503. The fourth field of view 509 may extend to a smaller angle in the yaw direction relative to vehicle 100 than the third field of view 507. Images captured by the camera of the fourth camera type 508 may be used to identify objects in a fourth distance range from vehicle 100. In some embodiments, the fourth distance range may include distances further from vehicle 100 than the distances included in the third distance range. The first distance range may include distances further from vehicle 100 than the distances included in the fourth distance range.

[0123] Figure 6A is an illustrative diagram of a camera system 600 according to an exemplary embodiment. As shown, multiple cameras may be mounted on the vehicle 100. As seen in Figure 6A, in some exemplary embodiments, there may be three cameras of the first camera type 502. Each camera of the first camera type 502 may have an associated first field of view 503. Each of these cameras of the first camera type 502 may be oriented forward with respect to the direction of travel of the vehicle 100. The direction of travel shown in Figure 6A is the negative x-direction, but other directions of travel are also possible. For example, the direction of travel may alternatively be the positive x-direction (e.g., the vehicle 100 may be traveling in the opposite direction). Furthermore, other numbers of cameras of the first camera type 502 (e.g., one, two, four, five, etc.) and / or other mounting locations for the cameras of the first camera type 502 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0124] As used herein, the phrase "oriented forward" means that the central optical axis of each field of view is oriented approximately forward with respect to the direction of travel of the vehicle, or at least parallel to the direction of travel of the vehicle. Similarly, as used herein, the phrase "oriented rearward" means that the central optical axis of each field of view is oriented approximately rearward with respect to the direction of travel of the vehicle, or at least parallel to the direction of travel of the vehicle. Furthermore, as used herein, "one or more lateral directions" and "one or more transverse directions" mean that the central optical axis of each field of view is oriented at approximately 90 degrees in the yaw direction with respect to the direction of travel of the vehicle. Furthermore, as used herein, the phrase "at least partially oriented" in that direction means that at least a portion of the field of view overlaps with that direction (but not necessarily with respect to the central axis of each field of view). For example, "at least partially oriented forward" means that at least a portion of the field of view overlaps with the direction of travel of the vehicle.

[0125] As shown in Figure 6A, in some embodiments, the camera system 600 may include four cameras of a second camera type 504. Each camera of the second camera type 504 may have an associated second field of view 505. One of the cameras of the second camera type 504 may be oriented forward with respect to the direction of travel of the vehicle 100. Another camera of the second camera type 504 may be oriented rearward with respect to the direction of travel of the vehicle 100. Two of the cameras of the second camera type 504 may each be oriented in one or more lateral directions (e.g., parallel to the y-axis) with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the second camera type 504 (e.g., one, two, three, or five) and / or other mounting locations for the cameras of the second camera type 504 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0126] As shown in Figure 6A, in some embodiments, the camera system 600 may have five cameras of a third camera type 506. Each camera of the third camera type 506 may have an associated third field of view 507. In some embodiments, the cameras of the third camera type 506 may be oriented forward with respect to the direction of travel of the vehicle 100. Two cameras of the third camera type 506 may be oriented at least parallel to the rearward direction with respect to the direction of travel of the vehicle 100. Furthermore, two cameras of the third camera type 506 may be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the third camera type 506 (e.g., one, two, three, four, six, etc.) and / or other mounting locations for the cameras of the third camera type 506 (e.g., under the vehicle 100 or inside the vehicle 100) are also possible.

[0127] The camera of the first camera type 502 may be mounted on the vehicle adjacent to the camera of the third camera type 506. As used herein, the term “adjacent” means within 10 centimeters. However, in some embodiments, the cameras may be closer or further apart from each other (e.g., within 1 meter or 1 centimeter from each other). Furthermore, the two cameras of the second camera type 504 may each be mounted on the vehicle 100 adjacent to the camera of the third camera type 506. Using fewer cameras in a camera system may increase the speed at which such a camera system can detect and / or recognize objects (e.g., this is because the camera system will process less data to make decisions from capturing images using fewer cameras). Figure 6A contains a considerable amount of information. For this reason, for clarity, Figures 6B–6D are provided to show the camera system 600 of Figure 6A, but all but one of the camera types have been removed from the drawings (e.g., to remove clutter).

[0128] Figure 6B is a simplified illustrative diagram of the camera system 600 of Figure 6A, showing the cameras of the first camera type 502 of the camera system 600 according to an exemplary embodiment. Each of the cameras of the first camera type 502 may be mounted on the roof of the vehicle 100. This may allow the camera system 600 to capture objects that are far away from the vehicle 100. Other mounting locations (e.g., above or inside the tire wells, or near the side mirrors) are also possible. The two cameras of the first camera type 502 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. This may provide a composite field of view that extends over a larger angle in the yaw direction relative to the vehicle 100, thereby facilitating the detection of objects that may enter the forward travel path of the vehicle 100.

[0129] Figure 6C is a simplified illustrative diagram of the camera system 600 of Figure 6A, showing a camera of the second camera type 504 of the camera system 600 according to an exemplary embodiment. According to Figure 6C, there may be four cameras of the second camera type 504. As can be seen from Figure 6A, the combined field of view of the cameras of the second camera type 504 may extend to 360 degrees in the yaw direction relative to the vehicle 100. This makes it easier to detect objects located near the vehicle 100 in the area surrounding the vehicle 100. One of the cameras of the second camera type 504 may be mounted on the front bumper of the vehicle 100. One of the cameras of the second camera type 504 may be mounted on the rear bumper of the vehicle 100. Two of the cameras of the second camera type 504 may be mounted on one or more side bumpers of the vehicle 100 (for example, above the front wheel wells of the vehicle 100). Positioning the second camera type 504 in this manner can facilitate the detection of objects close to the road surface (e.g., less than 1 meter above the road surface) that may cross the direction of travel of the vehicle 100. Other mounting locations (e.g., inside the tire well or near the side mirror) are also possible.

[0130] Figure 6D is a simplified illustrative diagram of the camera system 600 of Figure 6A, showing the cameras of the third camera type 506 of the camera system 600 according to an exemplary embodiment. As can be seen from Figure 6D, there may be five cameras of the third camera type 506, and the combined field of view of the cameras of the third camera type 506 may extend to 360 degrees in the yaw direction relative to the vehicle 100. Three of the cameras of the third camera type 506 may be mounted on the roof of the vehicle 100. Two of the cameras of the third camera type 506 may each be mounted on one or more side bumpers of the vehicle 100 (for example, above the front wheel wells of the vehicle 100). Such a configuration can facilitate the detection of objects within a third distance range that may cross the direction of travel of the vehicle 100. Other mounting locations (e.g., inside the tire wells or near the side mirrors) are also possible.

[0131] Figure 7A is an illustrative diagram of a camera system 700 according to an exemplary embodiment. As can be seen from Figure 7A, there may be four cameras of a first camera type 502 mounted on the vehicle 100, each having a first field of view 503. Each of the cameras of the first camera type 502 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. Including multiple cameras of the first camera type 502 can facilitate the detection of objects in the forward direction with respect to the direction of travel of the vehicle 100, since each camera of the first camera type 502 may be able to independently detect objects in the forward direction with respect to the direction of travel of the vehicle 100. The direction of travel shown in Figure 7A is the negative x direction, but other directions of travel are also possible. For example, the direction of travel may alternatively be the positive x direction (for example, the vehicle 100 may be traveling in the opposite direction). Furthermore, other numbers of cameras of the first camera type 502 (e.g., one, two, four, five, etc.) and / or other mounting locations for the camera(s) of the first camera type 502 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0132] As shown in Figure 7A, there may be five cameras of the second camera type 504 mounted on the vehicle 100, each having a second field of view 505. One camera of the second camera type 504 may be oriented forward with respect to the direction of travel of the vehicle 100. Two cameras of the second camera type 504 may each be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. Two cameras of the second camera type 504 may each be oriented rearward with respect to the direction of travel of the vehicle 100. In such a configuration, the fields of view of three cameras of the second camera type 504 may overlap in one or more areas behind the vehicle 100, and each camera of the second camera type 504 may be able to independently detect objects in the rearward direction with respect to the direction of travel of the vehicle 100, thus facilitating the detection of objects within a second distance range in the rearward direction with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the second camera type 504 (e.g., one, two, three, four, or six), and / or other mounting locations for the second camera type 504 cameras (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0133] Furthermore, as shown in Figure 7A, there may be five cameras of the third camera type 506 mounted on the vehicle 100, each having a third field of view 507. The cameras of the third camera type 506 may be oriented forward with respect to the direction of travel of the vehicle 100. Two cameras of the third camera type 506 may each be oriented at least partially rearward with respect to the direction of travel of the vehicle 100. Also, two cameras of the third camera type 506 may each be oriented at least partially laterally in one or more directions with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the third camera type 506 (e.g., one, two, three, four, or six) and / or other mounting locations for the cameras of the third camera type 506 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0134] Furthermore, as shown in Figure 7A, at least three cameras of the first camera type 502 may each be mounted on the vehicle 100 adjacent to at least three cameras of the third camera type 506. Furthermore, at least two cameras of the second camera type 504 may each be mounted on the vehicle 100 adjacent to at least two cameras of the third camera type 506. Figure 7A contains a considerable amount of information. For this reason, for clarity, Figures 7B to 7D are provided to illustrate the camera system 700 of Figure 7A, but all but one of the camera types have been removed from the drawings (for example, to eliminate clutter).

[0135] Figure 7B is a simplified illustrative diagram of the camera system 700 of Figure 7A, showing the cameras of the first camera type 502 of the camera system 700 according to an exemplary embodiment. As shown in Figure 7B, each of the four cameras of the first camera type 502 may be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0136] Figure 7C is a simplified illustrative diagram of the camera system 700 of Figure 7A, showing the cameras of the second camera type 504 of the camera system 700 according to an exemplary embodiment. As shown in Figure 7C, one camera of the second camera type 504 may be mounted on the front bumper of the vehicle 100. Furthermore, as shown in Figure 7C, each of the four cameras of the second camera type 504 may be mounted on the roof of the vehicle 100. Furthermore, as shown in Figure 7C, two cameras of the second camera type 504 may be mounted on the vehicle above the rear wheel wells of the vehicle 100. Such a configuration may facilitate the detection of objects within a second distance range in the rearward direction relative to the direction of travel of the vehicle 100, as the fields of view of the three cameras of the second camera type 504 may overlap in one or more areas behind the vehicle 100, and each camera of the second camera type 504 may be able to independently detect objects within a second distance range from the vehicle 100. Other mounting locations are also possible (for example, above or inside the tire well, or near the side mirror).

[0137] Figure 7D is a simplified illustrative diagram of the camera system 700 of Figure 7A, showing the cameras of the third camera type 506 of the camera system 700 according to an exemplary embodiment. As shown in Figure 7D, the five cameras of the third camera type 506 may be mounted on the roof of the vehicle 100. Mounting all of the cameras of the third camera type 506 on the roof makes it easier to detect objects within a third distance range from the vehicle 100 and covering 360 degrees in the yaw direction, because all of the cameras of the third camera type 506 may be at approximately the same distance from the road surface. When all of the cameras of the third camera type 506 are at approximately the same distance from the road surface, computing devices such as the system controller 402 may have less need to compensate for differences in the distance from the road surface across the cameras of the third camera type 506 when using images captured by the cameras of the third camera type 506 to detect objects within a third distance range from the vehicle 100. Other mounting locations (e.g., above or inside the tire wells, or near the side mirrors) are also possible.

[0138] Figure 8A is an illustrative diagram of a camera system 800 according to an exemplary embodiment. As shown in Figure 8A, there may be three cameras of a first camera type 502, each oriented at least partially forward with respect to the direction of travel of the vehicle 100. Each camera of the first camera type 502 may have an associated first field of view 503. The direction of travel shown in Figure 8A is the negative x-direction, but other directions of travel are also possible. For example, the direction of travel may alternatively be the positive x-direction (e.g., the vehicle 100 may be traveling in the opposite direction). Furthermore, other numbers of cameras of the first camera type 502 (e.g., one, two, four, or five), and / or other mounting locations for the cameras of the first camera type 502 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0139] Furthermore, as shown in Figure 8A, there may be four cameras of the second camera type 504, each having its own associated second field of view 505. One of the cameras of the second camera type 504 may be oriented forward with respect to the direction of travel of the vehicle 100. Another camera of the second camera type 504 may be oriented rearward with respect to the direction of travel of the vehicle 100. Each of the two cameras of the second camera type 504 may be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the second camera type 504 (e.g., one, two, three, or five), and / or other mounting locations for the cameras of the second camera type 504 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0140] Furthermore, as shown in Figure 8A, there may be six cameras of the third camera type 506, each having an associated third field of view 507. One camera of the third camera type 506 may be oriented forward with respect to the direction of travel of the vehicle 100. Two cameras of the third camera type 506 may be oriented at least partially backward with respect to the direction of travel of the vehicle 100. Furthermore, each of the three cameras of the third camera type 506 may be oriented at least partially laterally in one or more directions with respect to the direction of travel of the vehicle 100. Such a configuration can facilitate the detection of objects within a third distance range from the vehicle 100 located on one side of the vehicle 100 (e.g., the right side of the vehicle 100, the left side of the vehicle 100, the driver's side of the vehicle 100, or the occupant's side of the vehicle 100). The reason for this is that the fields of view of the three cameras of the third camera type 506 may overlap in one or more areas on one side of the vehicle 100, and each camera of the third camera type 506 may be able to independently detect objects within a third distance range from the vehicle 100. Facilitating the detection of objects within a third distance range from the vehicle 100 may, in turn, facilitate the vehicle 100 to perform turning maneuvers (e.g., turning left or right). Other numbers of cameras of the third camera type 506 (e.g., one, two, three, four, five, or seven), and / or other mounting locations for the cameras of the third camera type 506 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0141] Furthermore, as can be seen in Figure 8A, at least one camera of the first camera type 502 can be mounted on the vehicle 100 adjacent to at least one camera of the third camera type 506. In addition, at least two cameras of the second camera type 504 can be mounted on the vehicle 100 adjacent to at least two cameras of the third camera type 506.

[0142] Figure 8B is a simplified illustrative diagram of the camera system 800 of Figure 8A, showing the cameras of the first camera type 502 of the camera system 800 according to an exemplary embodiment. As shown in Figure 8B, each of the cameras of the first camera type 502 may be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0143] Figure 8C is a simplified illustrative diagram of the camera system 800 of Figure 8A, showing a camera of the second camera type 504 of the camera system 800 according to an exemplary embodiment. One camera of the second camera type 504 may be mounted on the front bumper of the vehicle 100. Furthermore, each of the two cameras of the second camera type 504 may be mounted on one or more side bumpers of the vehicle 100 (for example, above one or more wheel wells of the vehicle 100). Also, one camera of the second camera type 504 may be mounted on the roof of the vehicle 100. Other mounting locations (for example, inside the tire well or near the side mirror) are also possible.

[0144] Figure 8D is a simplified illustrative diagram of the camera system 800 of Figure 8A, showing the cameras of the third camera type 506 of the camera system 800 according to an exemplary embodiment. As shown in Figure 8D, the five cameras of the third camera type 506 can be mounted on the roof of the vehicle 100. Mounting the five cameras of the third camera type 506 on the roof of the vehicle 100 can facilitate the detection of objects within a third distance range from the vehicle 100 and over 360 degrees in the yaw direction, because these cameras of the third camera type 506 can be at approximately the same distance from the road surface. When these cameras of the third camera type 506 are at approximately the same distance from the road surface, a computing device such as the system controller 402 may not need to compensate for the difference in distance from the road surface across the cameras of the third camera type 506 when using images captured by the cameras of the third camera type 506 to detect objects within a third distance range from the vehicle 100. Furthermore, one camera of the second camera type 504 can be mounted on the side bumper of the vehicle 100 (for example, above the left front wheel well of the vehicle 100). Other mounting locations are also possible (for example, inside the tire well or near the side mirror).

[0145] Figure 9A is an illustrative diagram of a camera system 900 according to an exemplary embodiment. As shown in Figure 9A, there may be two cameras of the first camera type 502, each having an associated first field of view 503. Each camera of the first camera type 502 may be oriented forward with respect to the direction of travel of the vehicle 100. The direction of travel shown in Figure 9A is the negative x-direction, but other directions of travel are also possible. For example, the direction of travel may alternatively be the positive x-direction (e.g., the vehicle 100 may be traveling in the opposite direction). Furthermore, other numbers of cameras of the first camera type 502 (e.g., one, three, four, or five), and / or other mounting locations for the cameras of the first camera type 502 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0146] Furthermore, there may be four cameras of the second camera type 504, each having its associated second field of view 505. One camera of the second camera type 504 may be oriented forward with respect to the direction of travel of the vehicle 100. Another camera of the second camera type 504 may be oriented rearward with respect to the direction of travel of the vehicle 100. Furthermore, two cameras of the second camera type 504 may be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. Furthermore, other numbers of cameras of the second camera type 504 (e.g., one, two, three, or five) and / or other mounting locations for the cameras of the second camera type 504 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0147] As shown in Figure 9A, there may be eight cameras of the third camera type 506, each having an associated third field of view 507. Of these, two cameras of the third camera type 506 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. Furthermore, two cameras of the third camera type 506 may be oriented at least partially rearward with respect to the direction of travel of the vehicle 100. Also, each of the four cameras of the third camera type 506 may be oriented in one or more lateral directions with respect to the direction of travel of the vehicle 100. Furthermore, there may be two cameras of the fourth camera type 508, each having an associated fourth field of view 509. These cameras of the fourth camera type 508 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. Including both a first camera type 502 camera and a fourth camera type 508 camera oriented forward or at least partially forward with respect to the direction of travel of the vehicle 100 can facilitate the detection of objects that are at least partially forward with respect to the direction of travel of the vehicle 100. Other numbers of third camera type 506 cameras (e.g., one, two, three, four, five, six, seven, or nine) and / or other mounting locations for third camera type 506 cameras (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0148] As shown in Figure 9A, each of the two cameras of the first camera type 502 may be mounted on the vehicle 100 adjacent to the camera of the fourth camera type 508. Each of the cameras of the fourth camera type 508 may also be mounted on the vehicle 100 adjacent to the camera of the third camera type 506. Furthermore, each of the two cameras of the second camera type 504 may be mounted on the vehicle 100 adjacent to the two cameras of the third camera type 506.

[0149] Figure 9B is a simplified illustrative diagram of the camera system 900 of Figure 9A, showing the cameras of the first camera type 502 of the camera system 900 according to an exemplary embodiment. As shown in Figure 9B, the two cameras of the first camera type can be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0150] Figure 9C is a simplified illustrative diagram of the camera system 900 of Figure 9A, showing a camera of the second camera type 504 of the camera system 900 according to an exemplary embodiment. One camera of the second camera type 504 can be mounted on the roof of the vehicle 100. In addition, one camera of the second camera type 504 can be mounted on the front bumper of the vehicle 100. In addition, one camera of the second camera type 504 can be mounted on the rear bumper of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0151] As shown in Figure 9C, there may be a first predetermined camera 504A among a plurality of cameras of a second camera type 504 mounted on the roof of the vehicle 100, having a related first predetermined camera field of view 505A. The first predetermined camera 504A of the plurality of cameras of the second camera type 504 may be a camera of the second camera type 504 that includes a neutral density filter through which images are captured. The first predetermined camera 504A of the plurality of cameras of the second camera type 504 may be oriented relative to the vehicle 100 to capture images from a pitch angle greater than 5 degrees relative to the vehicle 100 in the forward direction relative to the direction of travel of the vehicle 100 (e.g., rotation around the z-axis). Such orientation can facilitate the detection of objects such as traffic signals, as will be described in more detail below. Other mounting locations (e.g., above the tire well or near the side mirrors) are also possible.

[0152] Figure 9D is a simplified illustrative diagram of the camera system 900 of Figure 9A, showing the cameras of the third camera type 506 of the camera system 900 according to an exemplary embodiment. As shown in Figure 9D, the eight cameras of the third camera type 506 may be mounted on the roof of the vehicle 100. Mounting the eight cameras of the third camera type 506 on the roof of the vehicle 100 can facilitate the detection of objects within a third distance range from the vehicle 100 and over 360 degrees in the yaw direction, because these cameras of the third camera type 506 may be at approximately the same distance from the road surface. When these cameras of the third camera type 506 are at approximately the same distance from the road surface, a computing device such as the system controller 402 may not need to compensate for differences in the distance from the road surface across the cameras of the third camera type 506 when using images captured by the cameras of the third camera type 506 to detect objects within a third distance range from the vehicle 100. Other mounting locations (e.g., above or inside the tire wells, or near the side mirrors) are also possible. Such a configuration may allow the fields of view of two or more cameras of the third camera type 506 to overlap over an area spanning 360 degrees in the yaw direction relative to the vehicle 100, and each camera of the third camera type 506 may be able to independently detect objects within a third distance range from the vehicle 100, thus facilitating the detection of objects within a third distance range from the vehicle 100, spanning 360 degrees in the yaw direction relative to the vehicle 100.

[0153] Figure 9E is a simplified illustrative diagram of the camera system 900 of Figure 9A, showing the cameras of the fourth camera type 508 of the camera system 900 according to an exemplary embodiment. As shown in Figure 9E, the two cameras of the fourth camera type 508 may be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0154] Figure 10A is an illustrative diagram of a camera system 1000 according to an exemplary embodiment. As shown in Figure 10A, the camera system 1000 may include two cameras of a first camera type 502, each having an associated first field of view 503. Each camera of the first camera type 502 may be oriented forward with respect to the direction of travel of the vehicle 100. The direction of travel shown in Figure 10A is the negative x-direction, but other directions of travel are also possible. For example, the direction of travel may alternatively be the positive x-direction (e.g., the vehicle 100 may be traveling in the opposite direction). Furthermore, other numbers of cameras of the first camera type 502 (e.g., one, three, four, or five), and / or other mounting locations for the cameras of the first camera type 502 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0155] Furthermore, as shown in Figure 10A, the camera system 1000 may include four cameras of the second camera type 504, each having an associated second field of view 505. One camera of the second camera type 504 may be oriented forward with respect to the direction of travel of the vehicle 100. Furthermore, one camera of the second camera type 504 may be oriented rearward with respect to the direction of travel of the vehicle 100. Furthermore, two cameras of the second camera type 504 may be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. Furthermore, other numbers of cameras of the second camera type 504 (e.g., one, two, three, or five) and / or other mounting locations for the cameras of the second camera type 504 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0156] As shown in Figure 10A, there may be eight cameras of the third camera type 506, each having its own associated third field of view 507. Of these, two cameras of the third camera type 506 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. Furthermore, two cameras of the third camera type 506 may be oriented at least partially rearward with respect to the direction of travel of the vehicle 100. In addition, each of the four cameras of the third camera type 506 may be oriented one or more lateral directions with respect to the direction of travel of the vehicle 100. Such a configuration may allow the fields of view of three cameras of the third camera type 506 to overlap in one or more areas behind the vehicle 100, and each camera of the third camera type 506 may be able to independently detect objects within a third distance range from the vehicle 100, thus facilitating the detection of objects within a third distance range from the vehicle 100 that are entering the rearward travel path of the vehicle 100. Other numbers of cameras of the third camera type 506 (e.g., one, two, three, four, five, six, seven, or nine), and / or other mounting locations for the third camera type 506 cameras (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0157] In some exemplary embodiments, there may be two cameras of the fourth camera type 508, each having an associated fourth field of view 509. These cameras of the fourth camera type 508 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. Including both cameras of the first camera type 502 and cameras of the fourth camera type 508 oriented forward or at least partially forward with respect to the direction of travel of the vehicle 100 can facilitate the detection of objects that are at least partially forward with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the fourth camera type 508 (e.g., one, three, or four), and / or other mounting locations for cameras of the fourth camera type 508 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0158] Furthermore, as shown in Figure 10A, at least two cameras of the first camera type 502 may each be mounted on the vehicle 100 adjacent to a camera of the fourth camera type 508. Also, at least two cameras of the first camera type 502 may each be mounted on the vehicle 100 adjacent to a camera of the third camera type 506. Furthermore, at least two cameras of the fourth camera type 508 may each be mounted on the vehicle adjacent to a camera of the third camera type 506. Furthermore, at least two cameras of the third camera type 506 may each be mounted on the vehicle adjacent to a camera of the second camera type 504.

[0159] Figure 10B is a simplified illustrative diagram of the camera system 1000 of Figure 10A, showing the cameras of the first camera type 502 of the camera system 1000 according to an exemplary embodiment. As shown in Figure 10B, the two cameras of the first camera type may each be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0160] Figure 10C is a simplified illustrative diagram of the camera system 1000 of Figure 10A, showing a camera of the second camera type 504 of the camera system 1000 according to an exemplary embodiment. As shown in Figure 10C, one camera of the second camera type 504 may be mounted on the front bumper of the vehicle 100. Furthermore, one camera of the second camera type 504 may be mounted on the rear bumper of the vehicle 100. Furthermore, two cameras of the second camera type 504 may each be mounted on the side bumper of the vehicle 100 (for example, above the front wheel well of the vehicle 100). Such a configuration may facilitate the detection of objects within a second distance range that may cross the direction of travel of the vehicle 100, because the fields of view of the three cameras of the second camera type 504 may overlap in one or more areas in front of the vehicle 100, and each camera of the second camera type 504 may be able to independently detect objects within a second distance range from the vehicle 100. Other mounting locations are also possible (for example, inside the tire well or near the side mirror).

[0161] As further shown in Figure 10C, the camera system 1000 may include a second predetermined camera 504B of a plurality of cameras of a second camera type 504 mounted on the roof of the vehicle 100, having a related second predetermined camera field of view 505B. The second predetermined camera 504B of the plurality of cameras of the second camera type 504 may be a camera of the second camera type 504 that includes a neutral density filter through which images are captured. Furthermore, the second predetermined camera 504B of the plurality of cameras of the second camera type 504 may be oriented relative to the vehicle 100 to capture images from a pitch angle greater than 20 degrees relative to the vehicle 100 in the forward direction relative to the direction of travel of the vehicle 100 (e.g., rotation around the z-axis). Such orientation can facilitate the detection of objects such as traffic signals, as will be described in more detail below. Other mounting locations (e.g., above the tire well or near the side mirror) are also possible.

[0162] Figure 10D is a simplified illustrative diagram of the camera system 1000 of Figure 10A, showing the cameras of the third camera type 506 of the camera system 1000 according to an exemplary embodiment. As shown in Figure 10D, the six cameras of the third camera type 506 may be mounted on the roof of the vehicle 100. Furthermore, as shown in Figure 10D, two cameras of the third camera type 506 may each be mounted on one or more side bumpers of the vehicle 100 (for example, above the front wheel wells of the vehicle 100). Other mounting locations (for example, inside the tire wells or near the side mirrors) are also possible.

[0163] Figure 10E is a simplified illustrative diagram of the camera system 1000 of Figure 10A, showing the cameras of the fourth camera type 508 of the camera system 1000 according to an exemplary embodiment. As shown in Figure 10E, the two cameras of the fourth camera type 508 may be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0164] Figure 11A is an illustrative diagram of a camera system 1100 according to an exemplary embodiment. As shown in Figure 11A, the camera system 1100 may include two cameras of a first camera type 502, each having an associated first field of view 503. These two cameras of the first camera type 502 may be oriented forward with respect to the direction of travel of the vehicle 100. The direction of travel shown in Figure 11A is the negative x-direction, but other directions of travel are also possible. For example, the direction of travel may alternatively be the positive x-direction (e.g., the vehicle 100 may be traveling in the opposite direction). Furthermore, other numbers of cameras of the first camera type 502 (e.g., one, three, four, or five), and / or other mounting locations for the cameras of the first camera type 502 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0165] Furthermore, there may be four cameras of the second camera type 504, each having its own associated second field of view 505. One camera of the second camera type 504 may be oriented forward with respect to the direction of travel of the vehicle 100. Another camera of the second camera type 504 may be oriented rearward with respect to the direction of travel of the vehicle 100. Furthermore, two cameras of the second camera type 504 may be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. In addition, other numbers of cameras of the second camera type 504 (e.g., one, two, three, or five) and / or other mounting locations for the camera(s) of the second camera type 504 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0166] As shown in Figure 11A, the camera system 1100 may include five cameras of the third camera type 506, each having an associated third field of view 507. One of the cameras of the third camera type 506 may be oriented forward with respect to the direction of travel of the vehicle 100. Furthermore, four of the cameras of the third camera type 506 may be oriented laterally in one or more directions with respect to the direction of travel of the vehicle 100. Other numbers of cameras of the third camera type 506 (e.g., one, two, three, four, or six), and / or other mounting locations for the cameras of the third camera type 506 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0167] Furthermore, as shown in Figure 11A, four cameras of the fourth camera type 508, each having an associated fourth field of view 509, may be mounted on the vehicle 100. Two of the cameras of the fourth camera type 508 may be oriented at least partially forward with respect to the direction of travel of the vehicle 100. Alternatively, two of the cameras of the fourth camera type 508 may be oriented backward with respect to the direction of travel of the vehicle 100. Such a configuration facilitates the detection of objects within a fourth distance range that may enter the forward or backward travel path of the vehicle 100. Other numbers of cameras of the fourth camera type 508 (e.g., one, two, three, or five), and / or other mounting locations for the cameras of the fourth camera type 508 (e.g., on the underside of the vehicle 100 or inside the vehicle 100) are also possible.

[0168] As shown in Figure 11A, each of the two cameras of the second camera type 504 may be mounted on the vehicle 100 adjacent to one or more cameras of the third camera type 506. Furthermore, the two cameras of the first camera type 502 may be mounted on the vehicle 100 adjacent to one or more cameras of the fourth camera type 508.

[0169] Figure 11B is a simplified illustrative diagram of the camera system 1100 of Figure 11A, showing the cameras of the first camera type 502 of the camera system 1100 according to an exemplary embodiment. As shown in Figure 11B, the two cameras of the first camera type 502 may be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above or inside the tire well, or near the side mirror) are also possible.

[0170] Figure 11C is a simplified illustrative diagram of the camera system 1100 of Figure 11A, showing a camera of the second camera type 504 of the camera system 1100 according to an exemplary embodiment. As shown in Figure 11C, one camera of the second camera type 504 may be mounted on the front bumper of the vehicle 100. Furthermore, one camera of the second camera type 504 may be mounted on the rear bumper of the vehicle 100. In addition, each of the two cameras of the second camera type 504 may be mounted on the side bumper of the vehicle 100 (for example, above the front wheel well of the vehicle 100). Other mounting locations (for example, inside the tire well or near the side mirror) are also possible.

[0171] Figure 11D is a simplified illustrative diagram of the camera system 1100 of Figure 11A, showing the cameras of the third camera type 506 of the camera system 1100 according to an exemplary embodiment. As shown in Figure 11D, the two cameras of the third camera type 506 may be mounted on the roof of the vehicle 100. Furthermore, each of the two cameras of the third camera type 506 may be mounted on the side bumper of the vehicle 100 (for example, above the front wheel well of the vehicle 100). Other mounting locations (for example, inside the tire well or near the side mirror) are also possible.

[0172] As further shown in Figure 11D, the camera system 1100 may include a predetermined camera 506A of a plurality of cameras of a third camera type 506 mounted on the roof of the vehicle 100, having a third predetermined camera field of view 507A. The predetermined camera 506A of the plurality of cameras of the third camera type 506 may be a camera of the third camera type 506 that includes a neutral density filter through which images are captured. Furthermore, the predetermined camera 506A of the plurality of cameras of the third camera type 506 may be oriented relative to the vehicle 100 to capture images from a pitch angle greater than 20 degrees relative to the vehicle 100 in the forward direction relative to the direction of travel of the vehicle 100 (e.g., rotation around the z-axis). Such orientation can facilitate the detection of objects such as traffic signals, as will be described in more detail below. Other mounting locations (e.g., above the tire well or near the side mirrors) are also possible.

[0173] Figure 11E is a simplified illustrative diagram of the camera system 1100 of Figure 11A, showing the cameras of the fourth camera type 508 of the camera system 1100 according to an exemplary embodiment. As shown in Figure 11E, the four cameras of the fourth camera type 508 may be mounted on the roof of the vehicle 100. Other mounting locations (e.g., above the tire well or near the side mirrors) are also possible.

[0174] As mentioned above, information from multiple sensors and / or cameras may be collected and combined. Figure 12 is an illustrative diagram of an object recognition technology 1200 according to an exemplary embodiment. In some embodiments, the LiDAR device may be mounted on a vehicle. For example, the LiDAR device may be mounted on a vehicle 100. This LiDAR device may generate a point cloud 1202 indicating the distance to objects in the environment around the vehicle 100. At least some of these objects may also be included in an image frame 1204 captured by a camera, such as a second camera type 504 camera, which may also be mounted on the vehicle 100. The LiDAR device and the camera may occupy different locations on the vehicle 100. Therefore, the fields of view of the camera and the fields of view of the LiDAR device may not inherently be perfectly aligned. However, portions of the fields of view of the camera and the fields of view of the LiDAR device may overlap with each other. Therefore, a computing device may align the image frame 1204 with the point cloud 1202 to create a aligned image frame. By doing so, the LiDAR device and the camera may each contain information having complementary characteristics about the object, thereby improving the detection and / or recognition of an object 1206 within the overlapping portion 1208 of the image frame 1204 and the point cloud 1202. Information about the recognized object may be used by multiple systems of the vehicle 100 in various embodiments. For example, information about the recognized object may be used by an obstacle avoidance system 144 to evaluate whether the recognized object is a potential obstacle.

[0175] Figure 13 is an illustrative diagram of a predetermined camera 506A of a plurality of cameras of a third camera type 506 according to an exemplary embodiment. As shown in Figure 13, the camera of the predetermined camera 506A of the plurality of cameras of the third camera type 506 may be oriented relative to the vehicle 100 to capture images from a pitch angle greater than 20 degrees relative to the vehicle 100 (e.g., rotation around the z-axis). Doing so may facilitate the detection of objects, such as traffic signals 1302, located above the vehicle 100 in the surrounding environment. The orientation of the camera of the predetermined camera 506A of the plurality of cameras of the third camera type 506 relative to the vehicle 100 may be adjusted to detect objects closer to or further from the road surface. For example, to facilitate the detection of objects closer to the road surface, the orientation of the predetermined camera 506A of the plurality of cameras of the third camera type 506 relative to the vehicle 100 may be adjusted to capture images from a pitch angle greater than 10 degrees or 5 degrees relative to the vehicle 100. As another example, to facilitate the detection of objects far from the road surface, the orientation of a given camera 506A of a plurality of cameras of a third camera type 506 relative to the vehicle 100 may be adjusted to capture images from a pitch angle greater than 25 degrees or 20 degrees relative to the vehicle 100. Doing so may facilitate the detection of objects closer to or farther from the road surface. The pitch angle values ​​relative to the vehicle 100 are provided as examples only; other pitch angles are possible.

[0176] In some exemplary embodiments, a given camera of a plurality of cameras of the second camera type 504 may include a neutral density filter through which images are captured. In such embodiments, a given camera of a plurality of cameras of the second camera type 504 may be oriented relative to the vehicle 100 to capture images from a pitch angle greater than 20 degrees relative to the vehicle. Furthermore, a given camera of a plurality of cameras of the second camera type 504 may be configured to capture a pair of image frames, the first image frame of the pair of image frames being captured with a first exposure time, which is set by an automatic exposure setting, and the second image frame of the pair of image frames being captured with a second exposure time, which is longer than 1 / 60 second. This second exposure time may be longer than 1 / 50 second. The second exposure time may be shorter than 1 / 60 second. The use of such cameras in a camera system can reduce the maintenance associated with such a camera system by reducing the total number of cameras in the camera system (for example, because a single camera can represent multiple imaging modalities).

[0177] In some exemplary embodiments, the LiDAR device may be mounted on a vehicle 100 and may be configured to generate a point cloud indicating the distance to objects in the environment surrounding the vehicle 100. In such embodiments, identifying objects located within a second distance range from the vehicle 100 based on one or more second images captured by a plurality of cameras of a second camera type 504 may include aligning the first image frame with the point cloud, superimposing the aligned first and second image frames to generate an overlapping image, and performing object recognition using the overlapping image and the point cloud.

[0178] In some exemplary embodiments, the multiple cameras of the second camera type 504 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, a camera oriented backward with respect to the direction of travel of the vehicle 100, and two cameras, each oriented laterally with respect to the direction of travel of the vehicle 100.

[0179] In some exemplary embodiments, at least one camera of the first camera type 502 may include two cameras, each oriented forward with respect to the direction of travel of the vehicle 100.

[0180] In some exemplary embodiments, at least one camera of the first camera type 502 may include three cameras, each oriented forward with respect to the direction of travel of the vehicle 100. The plurality of cameras of the second camera type 504 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, a camera oriented rearward with respect to the direction of travel of the vehicle 100, and two cameras, each oriented laterally with respect to the direction of travel of the vehicle 100. Furthermore, the plurality of cameras of the third camera type 506 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, a camera oriented at least partially rearward with respect to the direction of travel of the vehicle 100, and two cameras, each oriented laterally with respect to the direction of travel of the vehicle 100.

[0181] In some exemplary embodiments, at least one camera of the first camera type 502 may be mounted on the vehicle 100 adjacent to at least one camera of the third camera type 506.

[0182] In some exemplary embodiments, at least one camera of the first camera type 502 may include four cameras, each oriented at least partially forward with respect to the direction of travel of the vehicle 100. In such embodiments, the plurality of cameras of the second camera type 504 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, two cameras oriented rearward with respect to the direction of travel of the vehicle 100, and two cameras oriented laterally with respect to one or more of the direction of travel of the vehicle 100. Furthermore, in such embodiments, the plurality of cameras of the third camera type 506 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, two cameras oriented at least partially rearward with respect to the direction of travel of the vehicle 100, and two cameras oriented laterally with respect to at least partially one or more of the direction of travel of the vehicle 100.

[0183] In some exemplary embodiments, at least three cameras of at least one camera of the first camera type 502 may be mounted on the vehicle 100 adjacent to at least three cameras of the plurality of cameras of the third camera type 506. In such embodiments, at least two cameras of the plurality of cameras of the second camera type 504 may be mounted on the vehicle 100 adjacent to at least two cameras of the plurality of cameras of the third camera type 506.

[0184] In some exemplary embodiments, at least one camera of the first camera type 502 may include three cameras, each oriented at least partially forward with respect to the direction of travel of the vehicle 100. In such embodiments, the plurality of cameras of the second camera type 504 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, a camera oriented rearward with respect to the direction of travel of the vehicle 100, and two cameras, each oriented one or more laterally with respect to the direction of travel of the vehicle 100. Furthermore, in such embodiments, the plurality of cameras of the third camera type 506 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, two cameras, each oriented at least partially rearward with respect to the direction of travel of the vehicle 100, and three cameras, each oriented at least partially one or more laterally with respect to the direction of travel of the vehicle 100.

[0185] In some exemplary embodiments, at least one camera of the first camera type 502 may be mounted on the vehicle 100 adjacent to at least one camera of the third camera type 506, and at least two cameras of the second camera type 504 may each be mounted on the vehicle 100 adjacent to at least two cameras of the third camera type 506.

[0186] In some exemplary embodiments, at least one camera of the first camera type 502 may include two cameras, each oriented forward with respect to the direction of travel of the vehicle 100. In such embodiments, a plurality of cameras of the second camera type 504 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, a camera oriented rearward with respect to the direction of travel of the vehicle 100, and two cameras, each oriented laterally with respect to the direction of travel of the vehicle 100. Furthermore, in such embodiments, a plurality of cameras of the third camera type 506 may include a camera, each oriented at least partially forward with respect to the direction of travel of the vehicle 100, two cameras, each oriented at least partially rearward with respect to the direction of travel of the vehicle 100, and four cameras, each oriented laterally with respect to the direction of travel of the vehicle 100.

[0187] In some exemplary embodiments, the camera system may include a plurality of cameras of a fourth camera type 508. The plurality of cameras of the fourth camera type 508 may include two cameras mounted on the vehicle 100. Each camera of the fourth camera type 508 may have a fourth field of view 509, which may extend to a larger angle in the yaw direction relative to the vehicle 100 than the first field of view 503. Furthermore, the fourth field of view 509 may extend to a smaller angle in the yaw direction relative to the vehicle 100 than the third field of view 507. In such embodiments, a computing device may be configured to identify objects located within a fourth distance range based on one or more images captured by the plurality of cameras of the fourth camera type 508. The fourth distance range may include distances further from the vehicle 100 than the distances included in the third distance range, and the first distance range may include distances further from the vehicle 100 than the distances included in the fourth distance range.

[0188] In some exemplary embodiments, at least one of the multiple cameras of the second camera type 504 may include a neutral density filter through which images are captured. Such a camera may be oriented relative to the vehicle 100 to capture images from a pitch angle greater than 5 degrees relative to the vehicle. Furthermore, such a camera may have a field of view that extends over a larger angle in the yaw direction relative to the vehicle 100 than at least one other camera of the second camera type 504.

[0189] In some exemplary embodiments, at least two cameras of at least one camera of the first camera type 502 may be mounted on the vehicle 100 adjacent to at least two cameras of a plurality of cameras of the fourth camera type 508. In such embodiments, at least two cameras of at least one camera of the first camera type 502 may be mounted on the vehicle 100 adjacent to at least two cameras of a plurality of cameras of the third camera type 506. Also in such embodiments, at least two cameras of a plurality of cameras of the fourth camera type 508 may be mounted on the vehicle 100 adjacent to at least two cameras of a plurality of cameras of the third camera type 506. Furthermore, in such embodiments, at least two cameras of a plurality of cameras of the third camera type 506 may be mounted on the vehicle 100 adjacent to at least two cameras of a plurality of cameras of the second camera type 502.

[0190] In some exemplary embodiments, at least one camera of the first camera type 502 may include two cameras, each oriented forward with respect to the direction of travel of the vehicle 100. In such embodiments, a plurality of cameras of the second camera type 504 may include a camera oriented forward with respect to the direction of travel of the vehicle 100, a camera oriented rearward with respect to the direction of travel of the vehicle 100, and two cameras, each oriented laterally with respect to the direction of travel of the vehicle 100. In such embodiments, a plurality of cameras of the third camera type 506 may include one camera oriented forward with respect to the direction of travel of the vehicle 100, and four cameras, each oriented laterally with respect to the direction of travel of the vehicle 100.

[0191] In some exemplary embodiments, at least one of the multiple cameras of the third camera type 506 may include a neutral density filter through which an image is captured and oriented relative to the vehicle 100 to capture an image from a pitch angle greater than 20 degrees relative to the vehicle.

[0192] In some exemplary embodiments, the camera system may also include a plurality of cameras of a fourth camera type 508. In such embodiments, the plurality of cameras of the fourth camera type 508 may include four cameras mounted on the vehicle 100. Each camera of the fourth camera type 508 may have a fourth field of view 509, the fourth field of view 509 may extend to a larger angle in the yaw direction relative to the vehicle 100 than the first field of view 503. Furthermore, the fourth field of view 509 may extend to a smaller angle in the yaw direction relative to the vehicle than the third field of view 507. The computing device may also be further configured to identify objects located within a fourth distance range based on one or more images captured by the plurality of cameras of the fourth camera type 508. Such a fourth distance range may include distances further from the vehicle 100 than the distances included in the third distance range, and the first distance range may include distances further from the vehicle 100 than the distances included in the fourth distance range.

[0193] A single device may have features of at least one camera of a first camera type 502, at least one camera of a second camera type 504, at least one camera of a third camera type 506, at least one camera of a fourth camera type 508, and / or a LiDAR device. For example, a single device may acquire images used to identify objects within a second distance range from a vehicle 100 and to generate a point cloud 1202 indicating the distance to the objects in the environment around the vehicle 100. As another example, a single device may acquire images used to identify objects both within a first distance range from the vehicle 100 and within a second distance range from the vehicle 100. Other types of sensors may also be used in addition to, or instead of, cameras or LiDAR devices within a single device. For example, a radar sensor may be incorporated into a single device.

[0194] Figure 14 is a flowchart of Method 1400 according to an exemplary embodiment. Method 1400 may be carried out using images captured by camera systems 600, 700, 800, 900, 1000, or 1100. In an alternative embodiment, Method 1400 may be carried out using an alternative camera system.

[0195] In block 1402, method 1400 may also include receiving one or more first images captured by at least one camera of a first camera type using a computing device, each camera of the first camera type being mounted on a vehicle and having a first field of view.

[0196] In block 1404, method 1400 may include receiving one or more second images captured by a plurality of cameras of a second camera type by a computing device, each camera of the second camera type being mounted on a vehicle and having a second field of view, the second field of view extending at least 170 degrees in the yaw direction relative to the vehicle.

[0197] In block 1406, method 1400 may include receiving one or more third images captured by a plurality of cameras of a third camera type by a computing device, each camera of the third camera type being mounted on a vehicle and having a third field of view, the combined field of view of the plurality of cameras of the third camera type extending 360 degrees in the yaw direction relative to the vehicle, the first field of view extending at a smaller angle in the yaw direction relative to the vehicle than the third field of view, and the third field of view extending at a smaller angle in the yaw direction relative to the vehicle than the second field of view.

[0198] In block 1408, method 1400 may include, by a computing device, identifying an object located within a first distance range from a vehicle based on one or more first images.

[0199] In block 1410, method 1400 may include, by means of one or more second images, using a computing device to identify an object located within a second distance range from the vehicle.

[0200] In block 1412, method 1400 may include, by a computing device, identifying an object located within a third distance range from a vehicle based on one or more third images, wherein the third distance range includes distances further from the vehicle than those included in the second distance range, and the first distance range includes distances further from the vehicle than those included in the third distance range.

[0201] In block 1414, method 1400 may include determining a driving decision for a vehicle based on an object located within a first distance range from the vehicle, an object located within a second distance range from the vehicle, or an object located within a third distance range from the vehicle, using a computing device.

[0202] In some embodiments, method 1400 may also include causing a computing device to cause a vehicle (e.g., vehicle 100) to perform driving operations based on driving decisions.

[0203] This disclosure is not limited to the specific embodiments described herein, and the specific embodiments are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many changes and modifications can be made without departing from the spirit and scope of this disclosure. In addition to the methods and apparatus enumerated herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and modifications are intended to fall within the scope of the appended claims.

[0204] The above detailed description, with reference to the accompanying drawings, describes various features and functions of the disclosed systems, devices, and methods. In the drawings, unless the context otherwise indicates, similar symbols typically represent similar components identically. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized and other modifications may be made without departing from the scope of the subject matter presented herein. It will be readily apparent that the aspects of the disclosure described herein and illustrated in the drawings can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly assumed.

[0205] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and those considered herein, each step, block, action, and / or communication may represent the processing of information and / or the transmission of information according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, actions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in an order different from those shown or discussed, such as substantially simultaneously or in reverse order, depending on the relevant function. Furthermore, more or fewer blocks and / or actions may be used in any of the message flow diagrams, scenarios, and flowcharts considered herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other in part or as a whole.

[0206] Steps, blocks, or operations corresponding to the processing of information may correspond to a network of circuits, which may be configured to perform a specific logical function of the method or technique described herein. Alternatively or additionally, steps or blocks corresponding to the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions that are executable by a processor to perform a specific logical operation or action in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including RAM, a disk drive, a Solid State Drive, or another storage medium.

[0207] Furthermore, one or more steps, blocks, or operations corresponding to information transmission may correspond to information transmission between software modules and / or hardware modules on the same physical device. However, other information transmissions may be between software modules and / or hardware modules on different physical devices.

[0208] The specific arrangement shown in the figure should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in the given figure. Also, some of the illustrated elements may be combined or omitted. Furthermore, exemplary embodiments may include elements not illustrated in the figure.

[0209] Various aspects and embodiments are disclosed herein, but other aspects and embodiments will be obvious to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to limit the scope, and the true scope is indicated by the following claims.

Claims

1. It is a system, Vehicles and A first camera type camera mounted on the vehicle, wherein each camera of the first camera type has a first field of view, and comprises at least one camera. A plurality of cameras of a second camera type mounted on the vehicle, wherein each of the second camera type cameras has a second field of view, and the second field of view extends at least 170 degrees in the yaw direction relative to the vehicle, A plurality of third camera-type cameras mounted on the vehicle, wherein each of the third camera-type cameras has a third field of view, the combined field of view of the plurality of third camera-type cameras extends 360 degrees in the yaw direction relative to the vehicle, the first field of view extends at a smaller angle than the third field of view in the yaw direction relative to the vehicle, and the third field of view extends at a smaller angle than the second field of view in the yaw direction relative to the vehicle, A computing device communicatively coupled to the at least one camera of the first camera type, the plurality of cameras of the second camera type, and the plurality of cameras of the third camera type, wherein the computing device is Based on one or more first images captured by the at least one camera of the first camera type, an object located within a first distance range from the vehicle is identified. Based on one or more second images captured by the plurality of cameras of the second camera type, an object located within a second distance range from the vehicle is identified. Identifying an object located within a third distance range from the vehicle based on one or more third images captured by the plurality of cameras of the third camera type, wherein the third distance range includes distances further from the vehicle than the distances included in the second distance range, and the first distance range includes distances further from the vehicle than the distances included in the third distance range. A computing device configured to perform the following: A system equipped with these features.

2. A predetermined camera among the plurality of cameras of the second camera type described above is Equipped with a neutral density filter, the image is captured through the filter, Oriented relative to the vehicle to capture images from a pitch angle greater than 20 degrees relative to the vehicle, The system according to claim 1, configured to capture a pair of image frames, wherein the first image frame of the pair of image frames is captured with a first exposure time, the first exposure time is set by an automatic exposure setting, and the second image frame of the pair of image frames is captured with a second exposure time, the second exposure time is longer than 1 / 60 second.

3. The vehicle further comprises a light detection and ranging (lidar) device mounted on the vehicle and configured to generate a point cloud indicating the distance to an object in the environment surrounding the vehicle. Identifying an object located within the second distance range from the vehicle based on one or more second images captured by the plurality of cameras of the second camera type is: Aligning the first image frame with the point cloud, The matching first image frame and the second image frame are superimposed to generate an overlapping image, The process involves performing object recognition using the overlapping images and the point cloud, The system according to claim 2, including the above.

4. The plurality of cameras of the second camera type described above are: A camera oriented forward with respect to the direction of travel of the vehicle, A camera oriented in the rearward direction with respect to the vehicle's direction of travel, The system according to claim 1, comprising two cameras, each oriented laterally with respect to the direction of travel of the vehicle, one or more of them.

5. The system according to claim 1, wherein the at least one camera of the first camera type comprises two cameras, each oriented forward with respect to the direction of travel of the vehicle.

6. The first camera type, the at least one camera, comprises three cameras, each oriented forward with respect to the direction of travel of the vehicle, The plurality of cameras of the second camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, A camera oriented in the rearward direction with respect to the vehicle's direction of travel, Each comprises two cameras, one or more of which are oriented laterally with respect to the vehicle's direction of travel, The plurality of cameras of the third camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, Two cameras, each at least partially oriented in the rearward direction with respect to the vehicle's direction of travel, The system according to claim 1, each comprising two cameras oriented laterally with respect to the direction of travel of the vehicle, one or more of the cameras.

7. The system according to claim 6, wherein at least one camera of the first camera type is mounted on the vehicle adjacent to at least one camera of the plurality of cameras of the third camera type.

8. The first camera type, the at least one camera, comprises four cameras, each oriented at least partially forward with respect to the direction of travel of the vehicle, The plurality of cameras of the second camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, Two cameras, each oriented in the rearward direction with respect to the vehicle's direction of travel, Each comprises two cameras, one or more of which are oriented laterally with respect to the vehicle's direction of travel, The plurality of cameras of the third camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, Two cameras, each at least partially oriented in the rearward direction with respect to the vehicle's direction of travel, The system according to claim 1, comprising two cameras, each at least partially oriented in the one or more lateral directions with respect to the direction of travel of the vehicle.

9. Each of the at least three cameras of the first camera type described above is mounted on the vehicle adjacent to at least three cameras of the plurality of cameras of the third camera type described above. The system according to claim 8, wherein at least two cameras of the plurality of cameras of the second camera type are mounted on the vehicle adjacent to at least two cameras of the plurality of cameras of the third camera type.

10. The first camera type described above comprises three cameras, each oriented at least partially forward with respect to the direction of travel of the vehicle, The plurality of cameras of the second camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, A camera oriented in the rearward direction with respect to the vehicle's direction of travel, Each comprises two cameras, one or more of which are oriented laterally with respect to the vehicle's direction of travel, The plurality of cameras of the third camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, Two cameras, each at least partially oriented in the rearward direction with respect to the vehicle's direction of travel, The system according to claim 1, comprising three cameras, each at least partially oriented in the one or more lateral directions with respect to the direction of travel of the vehicle.

11. At least one camera of the first camera type is mounted on the vehicle adjacent to at least one camera of the plurality of cameras of the third camera type. The system according to claim 10, wherein at least two cameras of the plurality of cameras of the second camera type are mounted on the vehicle adjacent to at least two cameras of the plurality of cameras of the third camera type.

12. The first camera type, the at least one camera, comprises two cameras, each oriented forward with respect to the direction of travel of the vehicle, The plurality of cameras of the second camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, A camera oriented in the rearward direction with respect to the vehicle's direction of travel, Each comprises two cameras, one or more of which are oriented laterally with respect to the vehicle's direction of travel, The plurality of cameras of the third camera type described above are: Two cameras, each at least partially oriented in the forward direction with respect to the vehicle's direction of travel, Two cameras, each at least partially oriented in the rearward direction with respect to the vehicle's direction of travel, The system according to claim 1, comprising, each of four cameras, one or more cameras oriented laterally with respect to the direction of travel of the vehicle.

13. It further includes multiple cameras of a fourth camera type, The plurality of cameras of the fourth camera type comprises two cameras mounted on the vehicle, Each of the fourth camera types described above has a fourth field of view, The fourth field of view extends at a larger angle than the first field of view in the yaw direction relative to the vehicle. The fourth field of view extends to a smaller angle than the third field of view in the yaw direction relative to the vehicle. The computing device is further configured to identify objects located within a fourth distance range based on one or more images captured by the plurality of cameras of the fourth camera type. The fourth distance range includes distances that are further from the vehicle than the distances included in the third distance range, The system according to claim 12, wherein the first distance range includes a distance from the vehicle that is further than the distance included in the fourth distance range.

14. At least one of the plurality of cameras of the second camera type described above is Equipped with a neutral density filter, the image is captured through the filter, Oriented relative to the vehicle to capture images from a pitch angle greater than 5 degrees relative to the vehicle, The system according to claim 12, wherein the field of view extends over a larger angle in the yaw direction relative to the vehicle than that of at least one other camera of the second camera type.

15. Each of the at least two cameras of the first camera type is mounted on the vehicle adjacent to at least two cameras of the plurality of cameras of the fourth camera type. Each of the at least two cameras of the first camera type is mounted on the vehicle adjacent to at least two cameras of the plurality of cameras of the third camera type. At least two of the plurality of cameras of the fourth camera type described above are mounted on the vehicle adjacent to at least two of the plurality of cameras of the third camera type described above. The system according to claim 13, wherein at least two cameras of the plurality of cameras of the third camera type are mounted on the vehicle adjacent to at least two cameras of the plurality of cameras of the second camera type.

16. The first camera type, the at least one camera, comprises two cameras, each oriented forward with respect to the direction of travel of the vehicle, The plurality of cameras of the second camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, A camera oriented in the rearward direction with respect to the vehicle's direction of travel, Each comprises two cameras, one or more of which are oriented laterally with respect to the vehicle's direction of travel, The plurality of cameras of the third camera type described above are: A camera oriented in the forward direction with respect to the vehicle's direction of travel, The system according to claim 1, comprising, each of four cameras, one or more cameras oriented laterally with respect to the direction of travel of the vehicle.

17. At least one of the plurality of cameras of the third camera type described above is: Equipped with a neutral density filter, the image is captured through the filter, The system according to claim 16, which is oriented with respect to the vehicle so as to capture images from a pitch angle greater than 20 degrees with respect to the vehicle.

18. It further includes multiple cameras of a fourth camera type, The aforementioned fourth camera type comprises four cameras mounted on the vehicle, Each of the fourth camera types described above has a fourth field of view, The fourth field of view extends at a larger angle than the first field of view in the yaw direction relative to the vehicle. The fourth field of view extends to a smaller angle than the third field of view in the yaw direction relative to the vehicle. The computing device is further configured to identify objects located within a fourth distance range based on one or more images captured by the plurality of cameras of the fourth camera type. The fourth distance range includes distances that are further from the vehicle than the distances included in the third distance range, The system according to claim 16, wherein the first distance range includes a distance further from the vehicle than the distance included in the fourth distance range.

19. It is a method, A computing device receives one or more first images captured by at least one camera of a first camera type, wherein each camera of the first camera type is mounted on a vehicle and has a first field of view. The computing device receives one or more second images captured by a plurality of cameras of a second camera type, each of which is mounted on the vehicle and has a second field of view, the second field of view extending at least 170 degrees in the yaw direction relative to the vehicle. The computing device receives one or more third images captured by a plurality of cameras of a third camera type, wherein each camera of the third camera type is mounted on the vehicle and has a third field of view, the combined field of view of the plurality of cameras of the third camera type extends 360 degrees in the yaw direction with respect to the vehicle, the first field of view extends at a smaller angle than the third field of view in the yaw direction with respect to the vehicle, and the third field of view extends at a smaller angle than the second field of view in the yaw direction with respect to the vehicle. Based on the one or more first images, the computing device identifies an object located within a first distance range from the vehicle. Based on the one or more second images, the computing device identifies an object located within a second distance range from the vehicle. Based on the one or more third images, the computing device identifies an object located within a third distance range from the vehicle, wherein the third distance range includes distances further from the vehicle than those included in the second distance range, and the first distance range includes distances further from the vehicle than those included in the third distance range. The computing device determines a driving decision for the vehicle based on the object located within the first distance range from the vehicle, the object located within the second distance range from the vehicle, or the object located within the third distance range from the vehicle. Methods that include...

20. A non-temporary computer-readable storage medium, which, when executed by a processor, allows the processor to: Receiving one or more first images captured by at least one camera of a first camera type, wherein each camera of the first camera type is mounted on a vehicle and has a first field of view. Receiving one or more second images captured by a plurality of cameras of a second camera type, wherein each camera of the second camera type is mounted on the vehicle and has a second field of view, the second field of view extending at least 170 degrees in the yaw direction relative to the vehicle. Receiving one or more third images captured by a plurality of cameras of a third camera type, wherein each camera of the third camera type is mounted on the vehicle and has a third field of view, the combined field of view of the plurality of cameras of the third camera type extends 360 degrees in the yaw direction with respect to the vehicle, the first field of view extends at a smaller angle than the third field of view in the yaw direction with respect to the vehicle, and the third field of view extends at a smaller angle than the second field of view in the yaw direction with respect to the vehicle. Based on the one or more first images, to identify an object located within a first distance range from the vehicle, Based on the one or more second images, an object located within a second distance range from the vehicle is identified. Identifying an object located within a third distance range from the vehicle based on one or more third images, wherein the third distance range includes distances further from the vehicle than those included in the second distance range, and the first distance range includes distances further from the vehicle than those included in the third distance range. A driving decision for the vehicle is made based on the object located within the first distance range from the vehicle, the object located within the second distance range from the vehicle, or the object located within the third distance range from the vehicle. A non-temporary computer-readable medium that stores program instructions that cause actions including [specific actions].

Images