Three-dimensional time-of-flight vision system

Abstract

Example embodiments relate to devices, systems, and methods related to a three-dimensional time-of-flight / visible image sensor. Example embodiments include an apparatus that includes a plurality of time-of-flight sensor (ToF) pixels and a plurality of visible image sensor (VIS) pixels. The apparatus also includes a ToF communication channel configured to provide ToF image data indicative of one or more ToF pixels, and a VIS communication channel configured to provide VIS image data indicative of one or more VIS pixels. The plurality of ToF pixels and the plurality of VIS pixels are arranged along a plane. The plurality of ToF pixels are disposed in an arrangement between the plurality of VIS pixels.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Patent Application No. 63 / 536,174, filed on September 1, 2023, the contents of which are incorporated herein by reference. Background Art

[0003] Unless otherwise indicated herein, the description in this section is not prior art to the claims in this application and is not admitted to be prior art by inclusion in this section.

[0004] Image sensors convert light into electrical signals. They are used in a wide variety of applications, including cameras, smartphones, and medical imaging equipment.

[0005] A time-of-flight (ToF) system is a type of imaging system that measures the time it takes for light to travel from the system to an object and back by using a time-modulated light emitter and a ToF image sensor with a certain amount of temporal resolution. This information can be used to calculate the distance to an object in the field of view.

[0006] Compared to passive 3D imaging without a modulation source, ToF sensors can be used in low light conditions. Different instances of ToF sensors can have different modulation formats that affect their sensitivity to ambient light. For example, indirect ToF (iToF) systems typically use high duty cycle modulated light emitters and fast electronic shutters on iToF imager pixels. Direct ToF (dToF) systems typically use low duty cycle pulsed light emitters and single photon avalanche detectors (e.g., SPAD, GmAPD). As a result, ToF sensors are becoming increasingly popular in various applications such as augmented reality, robotics, and self-driving cars. However, ToF sensors also have some limitations. They can be expensive, bulky, and can be susceptible to interference from other objects. Many ToF systems can collect low-quality passive images, but they tend to have poor spatial resolution, limited passive low light sensitivity, and cannot collect visible color images compared to traditional passive imagers (e.g., CMOS imagers). Therefore, system designers for robots, autonomous vehicles, and other applications typically have ToF systems and camera (passive imaging) systems with overlapping fields of view.

[0007] Therefore, there is a need for systems and methods that combine the beneficial aspects of visible and ToF image sensors in a compact, inexpensive hardware package. Summary of the invention

[0008] The present disclosure relates to systems and methods that can advantageously improve sensing performance while potentially reducing cost and weight in visible and depth sensing applications. For example, in the field of autonomous vehicles, the systems and methods described herein can reduce accidents, such as vehicle-vehicle collisions, by integrating two important sensing functions within a single device, compared to conventional solutions involving separate systems. Other potential benefits include reduced complexity of interactions between different systems, utilization of shared optical axes and fields of view, and a more compact hardware footprint.

[0009] In a first aspect, a device is provided. The device includes a plurality of time-of-flight sensor (ToF) pixels and a plurality of visible image sensor (VIS) pixels. The device also includes a ToF communication channel, which is configured to provide ToF image data indicating one or more ToF pixels. The device also includes a VIS communication channel, which is configured to provide VIS image data indicating one or more VIS pixels. A plurality of ToF pixels and a plurality of VIS pixels are arranged along a plane. A plurality of ToF pixels are arranged in an arrangement between a plurality of VIS pixels.

[0010] In a second aspect, a device is provided. The device includes a device including a plurality of time-of-flight sensor (ToF) pixels configured to detect light within a predetermined wavelength range. The device also includes a plurality of visible image sensor (VIS) pixels. A plurality of ToF pixels and a plurality of VIS pixels are arranged along a first plane. A plurality of ToF pixels are arranged in an arrangement between a plurality of VIS pixels. The device also includes one or more actuators configured to shift the device along the first plane. The device further includes one or more vertical cavity surface emitting lasers (VCSELs) configured to emit light within a predetermined wavelength range. Additionally or alternatively, the device may include an edge emitting laser diode.

[0011] In a third aspect, a system is provided. The system includes a sensing layer. The sensing layer includes a plurality of time-of-flight sensor (ToF) pixels configured to detect light from a field of view to form ToF image data. The sensing layer also includes a plurality of visible image sensor (VIS) pixels configured to detect light from a field of view to form VIS image data. The system also includes a readout layer including a readout circuit configured to receive ToF image data and VIS image data. The system also includes a processing layer having a processing circuit configured to provide at least one composite image based on at least one of the ToF image data or the VIS image data. The sensing layer and the readout layer are communicatively linked. The readout layer and the processing layer are communicatively linked.

[0012] In a fourth aspect, a method is provided. The method includes causing one or more vertical cavity surface emitting lasers (VCSELs) to emit light pulses. The light pulses include light within a predetermined wavelength range. The method also includes determining first ToF image data based on reflected light pulses received by one or more ToF pixels at a first position. The received light is within a predetermined wavelength range. The method also includes determining first VIS image data based on light received by one or more VIS pixels at a first position. The method also includes causing an actuator to shift one or more ToF pixels and one or more VIS pixels from a first position to a second position. The method further includes determining second ToF image data based on reflected light pulses received by one or more ToF pixels at a second position. The method includes determining second VIS image data based on light received by one or more VIS pixels at a second position. The method further includes determining composite VIS image data from the first VIS image data and the second VIS image data. The method also includes determining composite ToF image data from the first ToF image data and the second ToF image data. The method also includes generating at least one object depth map estimate from the composite ToF image data. The method includes providing data indicating the composite VIS image data and at least one object depth map estimate via a communication interface.

[0013] These and other aspects, advantages and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a functional block diagram illustrating a vehicle according to an example embodiment.

[0015] Figure 2A is a diagram of the physical configuration of a vehicle according to an example embodiment.

[0016] Figure 2B is a diagram of the physical configuration of a vehicle according to an example embodiment.

[0017] Figure 2C is a diagram of the physical configuration of a vehicle according to an example embodiment.

[0018] Figure 2D is a diagram of the physical configuration of a vehicle according to an example embodiment.

[0019] Figure 2E is a diagram of the physical configuration of a vehicle according to an example embodiment.

[0020] Figure 3 is a conceptual illustration of wireless communications between various computing systems associated with an autonomous or semi-autonomous vehicle according to an example embodiment.

[0021] Figure 4A is a block diagram of a system including a lidar device according to an example embodiment.

[0022] Figure 4B is a block diagram of a lidar device according to an example embodiment.

[0023] Figure 5 is a block diagram of an apparatus according to an example embodiment.

[0024] Fig. 6A is a cross-sectional view of a time-of-flight sensor pixel according to example embodiments.

[0025] Figure 6B is a cross-sectional view of a visible sensor pixel according to an example embodiment.

[0026] Fig. 7A is a top view of an arrangement of ToF sensor pixels and VIS sensor pixels according to example embodiments.

[0027] Figure 7B is an illustration of a color filter array overlaying a sensor array according to an example embodiment.

[0028] Figure 8 is an illustration of an apparatus according to an example embodiment.

[0029] Fig. 9 is an illustration of a system according to an example embodiment.

[0030] Fig. 10A is an information flow diagram according to an example embodiment.

[0031] Fig. 10B is a system according to an example embodiment.

[0032] Fig.11 is a method according to an example embodiment. DETAILED DESCRIPTION

[0033] Example methods and systems are contemplated herein. Any example embodiment or feature described herein is not necessarily to be construed as being preferred or advantageous over other embodiments or features. In addition, the example embodiments described herein are not meant to be limiting. It will be readily appreciated that certain aspects of the disclosed systems and methods may be arranged and combined in a variety of different configurations, all of which are contemplated herein. In addition, the specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or less of each element shown in a given figure. In addition, some of the elements shown may be combined or omitted. In addition, example embodiments may include elements not shown in the figures.

[0034] As described herein, a laser radar device may include one or more light emitters and one or more detectors for detecting light emitted by one or more light emitters and reflected by one or more objects in the environment surrounding the laser radar device. As an example, the surrounding environment may include an internal or external environment, such as the interior of a building or the exterior of a building. Additionally or alternatively, the surrounding environment may include the interior of a vehicle. In addition, the surrounding environment may include a nearby area around and / or on a road. Examples of objects in the surrounding environment include, but are not limited to, other vehicles, traffic signs, pedestrians, cyclists, road surfaces, buildings, and terrain. In addition, one or more light emitters may emit light into the local environment of the laser radar itself. For example, light emitted from one or more light emitters may interact with the housing of the laser radar and / or a surface or structure coupled to the laser radar. In some cases, the laser radar may be mounted to a vehicle, in which case one or more light emitters may be configured to emit light that interacts with objects near the vehicle. In addition, the light emitter may include a fiber amplifier, a laser diode, a light emitting diode (LED), and the like.

[0035] The present disclosure is generally described at four levels of abstraction.

[0036] First, the present invention describes a sensor pixel layout. The layout combines visible (VIS) and time-of-flight (ToF) sensor elements along a common surface of a substrate in a predetermined patterned arrangement. In an example embodiment, the VIS and ToF pixels may be arranged in a grid. In some embodiments, the VIS pixels may be significantly smaller than the ToF pixels. In this case, within the grid, the ToF pixels may be interspersed between subarrays of VIS pixels. For example, a given row of the patterned arrangement may alternate between a 5×5 grid of VIS pixels and ToF pixels. The patterned arrangement may be extended to a “chessboard” pattern across the entire surface of the substrate. The sensor elements are communicatively coupled to two virtual channels for data readout: one for the VIS pixels and one for the TOF pixels. The patterned arrangement is disposed on a movable substrate. In such a scenario, the sensor pixels may capture a first set of VIS and ToF images when the movable substrate is in a first position. The actuator may move (e.g., shift) the movable substrate along the plane of the substrate by a predetermined distance. Subsequently, the sensor pixels may capture a second set of VIS and ToF images when in a second position. In some embodiments, the patterned arrangement is configured to allow a microlens or other optical element (e.g., a lens or lens group) and / or a color filter array (CFA) to be fixed above the translatable surface, but optionally not attached to the surface.

[0037] Second, the present disclosure includes an actuated camera device. The device includes three components: a VIS-ToF sensor, a vertical cavity surface emitting laser (VCSEL) array, and an actuator. Although the example embodiments describe a VCSEL array, it should be understood that other types of light emitter devices (such as edge emitter lasers and / or light emitting diodes (LEDs)) are possible and expected. The VIS-ToF sensor is a sensor that provides a VIS data channel and a ToF data channel, which can provide visible and time-of-flight information that can substantially overlap or interleave along a shared field of view. The VCSEL array is configured to operate at a wavelength detectable by the ToF sensor. The actuator is configured to precisely shift the VIS-ToF sensor to reverse the interleaving of the VIS image and the ToF image. That is, the actuator converts the VIS-ToF sensor from a first state to a second state, wherein the second state positions the VIS sensor at the position when the TOF sensor is in the first state, and positions the TOF sensor at the position when the VIS sensor is in the first state. Note that along the edge of the VIS-ToF sensor, there may be some displaced sensors where there were no VIS or ToF sensors previously.

[0038] In some embodiments, the purpose of moving the substrate by a predetermined distance and direction is to achieve two sets of VIS / ToF image frames that are shifted relative to each other and can be combined into full-resolution images of the field of view indicating visible and time-of-flight information, respectively. In other words, for each type of sensor, actuation allows the "gaps" in the VIS or ToF data from the first state caused by the other type of sensor to be "filled" by complementary spatial information. In some embodiments, a lens or color filter array can be constructed across the VIS-ToF sensor field of view. In some examples, the color filter array can include a Bayer CFA or an RGGB CFA. The Bayer CFA uses a pattern of red, green, and blue color filters, where each pixel on the sensor is covered by a single color filter. The Bayer RGGB CFA uses a pattern of red, green, green, and blue color filters. RGGC and RGGB CFA color filters are also considered. Transparent color filters allow the sensor below to capture more light, which can improve image quality in low light conditions. Other types of CFA are possible and expected.

[0039] Third, the present disclosure includes a device configured to capture VIS and ToF information and process it for use by other systems. The device includes three physical substrate layers. The first layer includes a VIS-ToF image sensor. The image sensor is configured to capture visible and ToF information from a given field of view. The second layer includes units for image processing, such as an ADC for converting analog signals to digital counterparts, HDR processing for drawing details in low contrast situations, and a cache (i.e., memory) for temporarily storing data. Other functions are possible and expected, such as charge integration, signal amplification, signal multiplexing, timing control, and data formatting. In some embodiments, the third layer may include units for processing images and generating user-consumable data. For example, the third layer may include one or more neural networks that generate object lists (with high recall and lower precision). It should be understood that many different types of neural networks are expected and possible. As an example, one or more of a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer, and / or a base model may represent a type of deep learning algorithm that can be used for image analysis and classification. They can be suitable for tasks such as segmentation and classification because they can be trained to recognize patterns in images. In some embodiments, the CNN may be physically instantiated, such as in an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Additionally or alternatively, the CNN may be instantiated in software. In some cases, the CNN may be implemented by a computing device and may be allocated in memory with dynamic memory allocation or static memory allocation.

[0040] In addition, the third layer may include one or more recursive neural networks (RNNs) that acquire object list metadata and capture timing information associated with potential objects to improve the accuracy of object classification (using oversampled time information). In some embodiments, the RNN may provide an artificial neural network for processing sequential data. The RNN can be configured to learn long-term dependencies between inputs, which makes them suitable for tasks such as natural language processing and speech recognition. The RNN provides a loop return in the neural network structure. In contrast to a feedforward neural network, the loop allows the RNN to remember past inputs, which can be used to influence future outputs. Similar to CNN, the RNN can also be physically instantiated in an ASIC or FPGA. Additionally or alternatively, the RNN can be instantiated in software. In some cases, the RNN can be implemented by a computing device and can be allocated in memory with dynamic memory allocation or static memory allocation.

[0041] Additionally or alternatively, the third layer may include a sliding window transformer, a sparse window transformer, an attention mechanism, a generative adversarial network (GAN), a self-supervised learning neural network, and the like.

[0042] In addition, the third layer may include a processor configured to trigger image capture and cause the above-mentioned actuator to move the position of the image sensor substrate. Additionally or alternatively, the processor may be configured to combine the spatially shifted ToF and VIS information generated by the VIS-ToF image sensor as described in the second part of the present disclosure. In addition, in various examples, the third layer may include computing resources and / or other hardware configured to perform image processing, which includes time denoising, 3D noise filtering, and super-resolution of ToF information, VIS images, or a combination thereof. In addition, the third layer may include units, such as automotive safety integrity level (ASIL) circuits, to indicate the safety level of a given field of view and / or driving scene based on the data generated above. These layers can be implemented in many different configurations. One such configuration may include each functional layer on its own physical substrate and a corresponding substrate stacked on top of each other.

[0043] In addition, the present disclosure includes a method for operating various systems and devices described herein. The method may include a processor to simultaneously capture VIS and ToF images from a VIS-ToF sensor, wherein the VIS-ToF sensor is a sensor that provides VIS information via a VIS channel from a shared field of view and provides ToF information via a ToF channel. In some embodiments, the method may include a processor to cause a VCSEL to emit one or more light pulses, wherein the VCSEL is configured to provide illumination light to a ToF pixel on the VIS-ToF sensor. The method may also include a processor causing an actuator to adjust (e.g., shift) the position of the VIS-ToF sensor by a predetermined distance. In some embodiments, a position shift of the VIS-ToF sensor may be performed so as to position the sensor to obtain complementary VIS and ToF information. The method may also include a processor to simultaneously capture a second group of VIS and ToF images from a VIS-ToF sensor. The method also includes causing the processor to generate a combined VIS image from a first VIS image and a second VIS image and to generate a combined TOF image from a first TOF image and a second TOF image. In some embodiments, the method also includes the processor applying various image processing techniques to the VIS image, including time denoising, 3D noise filtering, super-resolution processing, and the like. In some examples, the method includes the processor generating an object depth map estimate using the TOF image. In addition, the method may include the processor applying various image processing techniques to the TOF image, including super-resolution and the like.

[0044] The following description and accompanying drawings will illustrate the features of various example embodiments. The embodiments provided are by way of example and are not intended to be limiting. Therefore, the dimensions of the accompanying drawings are not necessarily drawn to scale.

[0045] The example systems within the scope of the present disclosure will now be described in more detail. The example systems may be implemented in an automobile or may take the form of an automobile. In addition, the example systems may also be implemented in various vehicles or take the form of various vehicles, such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor trailers), motorcycles, buses, airplanes, helicopters, drones, lawn mowers, earthmoving bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment or vehicles, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, trams, golf carts, trains, carts, sidewalk delivery vehicles, and robotic devices. Other vehicles are also possible. In addition, in some embodiments, the example system may not include a vehicle.

[0046] Now referring to the accompanying drawings, Figure 1 is a functional block diagram illustrating an example vehicle 100, which may be configured to operate in full or in part in an autonomous mode. More specifically, the vehicle 100 may operate in an autonomous mode without human interaction by receiving control instructions from a computing system. As part of operating in an autonomous mode, the vehicle 100 may use sensors to detect and possibly identify objects of the surrounding environment to enable safe navigation. In addition, the example vehicle 100 may operate in a partially autonomous (i.e., semi-autonomous) mode, in which some functions of the vehicle 100 are controlled by a human driver of the vehicle 100 and some functions of the vehicle 100 are controlled by a computing system. For example, the vehicle 100 may also include a subsystem that enables the driver to control the operation of the vehicle 100, such as steering, acceleration, and braking, while the computing system performs auxiliary functions such as lane departure warning / lane keeping assist or adaptive cruise control based on other objects (e.g., vehicles) in the surrounding environment.

[0047] As described herein, in a partially autonomous driving mode, a human driver is expected to be situationally aware of the vehicle's surroundings and oversee the assisted driving operations even if the vehicle assists with one or more driving operations (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), and emergency braking). Here, even if the vehicle can perform all driving tasks in certain situations, the human driver is expected to be responsible for taking control as needed.

[0048] Although various systems and methods are described below in conjunction with autonomous vehicles for the sake of brevity and simplicity, these or similar systems and methods may be used in various driver assistance systems that do not reach the level of fully autonomous driving systems (i.e., partially autonomous driving systems). In the United States, the Society of Automotive Engineers (SAE) has defined different levels of automated driving operation to indicate the degree to which the vehicle controls driving more or less, although different organizations in the United States or other countries may classify the levels differently. More specifically, the disclosed systems and methods may be used to implement SAE Level 2 driver assistance systems for steering, braking, acceleration, lane centering, adaptive cruise control, etc., as well as other driver support. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems that are capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 automated driving systems that operate autonomously in most conventional driving situations and require only occasional attention from a human operator. In all such systems, accurate lane estimation can be performed automatically without driver input or control (e.g., while the vehicle is in motion) and results in improvements in the reliability of vehicle positioning and navigation and the overall safety of autonomous, semi-autonomous, and other driver assistance systems. As previously described, in addition to the way the SAE classifies levels of automated driving operation, other organizations in the United States or other countries may classify levels of automated driving operation differently. Without limitation, the systems and methods disclosed herein may be used in driver assistance systems defined by these other organizations' levels of automated driving operation.

[0049] like Figure 1 As shown, the vehicle 100 may include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power source 110, a computer system 112 (which may also be referred to as a computing system) having a data storage device 114, and a user interface 116. In other examples, the vehicle 100 may include more or fewer subsystems, each of which may include multiple elements. The subsystems and components of the vehicle 100 may be interconnected in various ways. In addition, the functions of the vehicle 100 described herein may be divided into additional functions or physical components, or combined into fewer functions or physical components within an embodiment. For example, the control system 106 and the computer system 112 may be combined into a single system that operates the vehicle 100 according to various operations.

[0050] The propulsion system 102 may include one or more components operable to provide powered movement for the vehicle 100, and may include an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121, among other possible components. For example, the engine / motor 118 may be configured to convert the energy source 119 into mechanical energy, and may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine, among other possible options. For example, in some embodiments, the propulsion system 102 may include multiple types of engines and / or motors, such as a gasoline engine and an electric motor.

[0051] Energy source 119 represents an energy source that can fully or partially power one or more systems (e.g., engine / motor 118) of vehicle 100. For example, energy source 119 can correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other sources of electricity. In some embodiments, energy source 119 can include a combination of a fuel tank, a battery, a capacitor, and / or a flywheel.

[0052] The transmission 120 can transfer mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. Thus, the transmission 120 can include a gearbox, a clutch, a differential, and a drive shaft, among other possible components. The drive shaft can include an axle connected to one or more wheels / tires 121.

[0053] In an exemplary embodiment, the wheels / tires 121 of the vehicle 100 may have various configurations. For example, the vehicle 100 may exist in the form of a unicycle, a bicycle / motorcycle, a tricycle, or a car / truck with four wheels, among other possible configurations. As such, the wheels / tires 121 may be connected to the vehicle 100 in various ways and may exist in different materials, such as metal and rubber.

[0054] The sensor system 104 may include various types of sensors, such as a global positioning system (GPS) 122, an inertial measurement unit (IMU) 124, a radar 126, a lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, among other possible sensors. In some embodiments, the sensor system 104 may also include sensors configured to monitor the internal systems of the vehicle 100 (e.g., an O2 monitor, a fuel gauge, engine oil temperature, and brake wear).

[0055] The GPS 122 may include a transceiver operable to provide information about the position of the vehicle 100 relative to the earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes, and may sense position and orientation changes of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 when the vehicle 100 is stationary or moving.

[0056] Radar 126 may represent one or more systems configured to use radio signals to sense objects within the surroundings of vehicle 100, including the speed and heading of the objects. As such, radar 126 may include an antenna configured to send and receive radio signals. In some embodiments, radar 126 may correspond to a mountable radar configured to obtain measurements of the surroundings of vehicle 100.

[0057] The laser radar 128 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components, and may operate in a coherent mode (e.g., using heterodyne detection) or in an incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of the laser radar 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes). In some examples, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). In addition, such photodetectors may be arranged (e.g., by electrical connections in series) into an array (e.g., as in a silicon photomultiplier (SiPM)). In some examples, one or more photodetectors are devices that operate in Geiger mode, and the laser radar includes subassemblies designed for such Geiger mode operation.

[0058] Camera 130 may include one or more devices configured to capture images of the surrounding environment of vehicle 100 (eg, still cameras, video cameras, thermal imaging cameras, stereo cameras, and night vision cameras).

[0059] The steering sensor 123 can sense the steering angle of the vehicle 100, which can involve measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, the steering sensor 123 can measure the angle of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the forward axis of the vehicle 100. The steering sensor 123 can also be configured to measure a combination (or subset) of the angle of the steering wheel, the electrical signal representing the angle of the steering wheel, and the angle of the wheels of the vehicle 100.

[0060] The throttle / brake sensor 125 can detect the position of the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 can measure the angle of both the throttle pedal (throttle) and the brake pedal, or can measure an electrical signal that can represent, for example, the angle of the throttle pedal (throttle) and / or the angle of the brake pedal. The throttle / brake sensor 125 can also measure the angle of the throttle body of the vehicle 100, which can include a part of the physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., a butterfly valve and a carburetor). In addition, the throttle / brake sensor 125 can measure the pressure of one or more brake pads on the rotor of the vehicle 100, or a combination (or subset) of the angles of the throttle pedal (throttle) and the brake pedal, the electrical signal representing the angles of the throttle pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied to the rotor of the vehicle 100 by at least one brake pad. In other embodiments, the throttle / brake sensor 125 can be configured to measure the pressure applied to the pedal (e.g., throttle or brake pedal) of the vehicle.

[0061] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / path system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the heading of the vehicle 100, and the throttle 134 may control the rate of operation of the engine / motor 118 to control the acceleration of the vehicle 100. The braking unit 136 may decelerate the vehicle 100, which may involve using friction to decelerate the wheels / tires 121. In some embodiments, the braking unit 136 may convert the kinetic energy of the wheels / tires 121 into electrical current for subsequent use by one or more systems of the vehicle 100.

[0062] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithm that may process data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide assessments based on the incoming sensor data, such as assessments of individual objects and / or features, assessments of particular situations, and / or assessments of potential impacts in a given situation.

[0063] The computer vision system 140 may include hardware and software (e.g., a general purpose processor such as a central processing unit (CPU), a special purpose processor such as a graphics processing unit (GPU) or a tensor processing unit (TPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) that is operable to process and analyze images in an effort to determine objects in motion (e.g., other vehicles, pedestrians, cyclists, or animals) and objects that are not in motion (e.g., traffic lights, road boundaries, roadblocks, or potholes). In this way, the computer vision system 140 may use object recognition, structure from motion (SFM), video tracking, and other algorithms used in computer vision, for example, to identify objects, map environments, track objects, estimate the velocity of objects, etc.

[0064] The navigation / routing system 142 can determine the driving path of the vehicle 100, which can involve dynamically adjusting the navigation during operation. In this way, the navigation / routing system 142 can use data from the sensor fusion algorithm 138, GPS 122 and maps, and other sources to navigate the vehicle 100. The obstacle avoidance system 144 can evaluate potential obstacles based on the sensor data and cause the system of the vehicle 100 to avoid or otherwise navigate around the potential obstacles.

[0065] like Figure 1 As shown, the vehicle 100 may also include peripherals 108, such as a wireless communication system 146, a touch screen 148, an internal microphone 150, and / or a speaker 152. The peripherals 108 may provide controls or other elements for a user to interact with the user interface 116. For example, the touch screen 148 may provide information to a user of the vehicle 100. The user interface 116 may also accept input from the user via the touch screen 148. The peripherals 108 may also enable the vehicle 100 to communicate with devices such as other vehicle devices.

[0066] The wireless communication system 146 can communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication (such as code division multiple access (CDMA), evolution data optimization (EVDO), global system for mobile communications (GSM) / general packet radio service (GPRS)) or cellular communication (such as 4G Worldwide Interoperability for Microwave Access (WiMAX) or long term evolution (LTE) or 5G). Alternatively, the wireless communication system 146 can use Or other possible connections to communicate with a wireless local area network (WLAN). The wireless communication system 146 can also communicate directly with the device using, for example, an infrared link, Bluetooth, or ZigBee. Other wireless protocols (such as various vehicle communication systems) are possible within the context of the present disclosure. For example, the wireless communication system 146 may include one or more dedicated short-range communication (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0067] The vehicle 100 may include a power source 110 for powering the components. In some embodiments, the power source 110 may include a rechargeable lithium-ion or lead-acid battery. For example, the power source 110 may include one or more batteries configured to provide power. Other types of power sources may also be used with the vehicle 100. In an exemplary embodiment, the power source 110 and the energy source 119 may be integrated into a single energy source.

[0068] The vehicle 100 may also include a computer system 112 to perform operations, such as those described herein. As such, the computer system 112 may include at least one processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored in a non-transitory computer-readable medium, such as a data storage device 114. In some embodiments, the computer system 112 may represent a plurality of computing devices that may be used to control various components or subsystems of the vehicle 100 in a distributed manner.

[0069] In some embodiments, the data storage device 114 may include data that can be executed by the processor 113 to perform various functions of the vehicle 100 (including the above combined Figure 1 The data storage device 114 may also include additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the propulsion system 102, the sensor system 104, the control system 106, and the peripheral devices 108.

[0070] In addition to instructions 115, data storage device 114 may store data such as road maps, routing information, etc. Such information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0071] The vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of the vehicle 100. The user interface 116 may control or enable control of the content and / or layout of interactive images that may be displayed on the touch screen 148. In addition, the user interface 116 may include one or more input / output devices within the set of peripherals 108, such as a wireless communication system 146, a touch screen 148, a microphone 150, and a speaker 152.

[0072] The computer system 112 may control functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, or the control system 106) and from the user interface 116. For example, the computer system 112 may utilize inputs from the sensor system 104 in order to estimate outputs produced by the propulsion system 102 and the control system 106. Depending on the embodiment, the computer system 112 may be operable to monitor many aspects of the vehicle 100 and its subsystems. In some embodiments, the computer system 112 may disable some or all functions of the vehicle 100 based on signals received from the sensor system 104.

[0073] The components of the vehicle 100 may be configured to work in a manner interconnected with other components within or outside of its corresponding system. For example, in an example embodiment, the camera 130 may capture multiple images that may represent information about the state of the surrounding environment of the vehicle 100 operating in an autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, the computer vision system 140 may be able to identify a slope (grade) or other features based on multiple images of the road. In addition, a combination of GPS 122 and features identified by the computer vision system 140 may be used together with map data stored in the data storage device 114 to determine specific road parameters. In addition, radar 126 and / or lidar 128 and / or some other environmental mapping, ranging and / or positioning sensor systems may also provide information about the vehicle's surrounding environment.

[0074] In other words, a combination of various sensors (which may be referred to as input-indicating and output-indicating sensors) and the computer system 112 may interact to provide an indication of input provided to control the vehicle or an indication of the vehicle's surroundings.

[0075] In some embodiments, the computer system 112 may make determinations about various objects based on data provided by systems other than the radio system. For example, the vehicle 100 may have a laser or other optical sensor configured to sense objects in the vehicle's field of view. The computer system 112 may use the outputs from the various sensors to determine information about objects in the vehicle's field of view, and may determine distance and direction information to the various objects. The computer system 112 may also determine whether an object is desired or undesired based on the outputs from the various sensors.

[0076] although Figure 1 The various components of the vehicle 100 (i.e., the wireless communication system 146, the computer system 112, the data storage device 114, and the user interface 116) are shown as being integrated into the vehicle 100, but one or more of these components may be installed or associated separately from the vehicle 100. For example, the data storage device 114 may exist partially or entirely separately from the vehicle 100. Therefore, the vehicle 100 may be provided in the form of device elements that may be located separately or together. The device elements that make up the vehicle 100 may be communicatively coupled together in a wired and / or wireless manner.

[0077] Figure 2A-2E An example vehicle 200 (eg, a fully autonomous vehicle or a semi-autonomous vehicle) is shown, which may include a vehicle 200 that is configured to perform operations in conjunction with reference to FIG. Figure 1 Although for illustrative purposes, vehicle 200 may be used in some or all of the functions described for vehicle 100. Figure 2A-2E 200 is shown as a van with side mirrors, but the disclosure is not limited in this regard. For example, vehicle 200 may represent a truck, a car, a semi-trailer, a motorcycle, a golf cart, an off-road vehicle, an agricultural vehicle, or any other vehicle described elsewhere herein (e.g., a bus, a boat, an airplane, a helicopter, a drone, a lawnmower, an earth-moving bulldozer, a submarine, an all-terrain vehicle, a snowmobile, an aircraft, a recreational vehicle, an amusement park vehicle, farm equipment, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, a tram, a train, a cart, a sidewalk delivery vehicle, and a robotic device).

[0078] The example vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidars, one or more radars, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding the vehicle 200. In other words, any sensor system now known or later created may be coupled to the vehicle 200 and / or may be used in conjunction with various operations of the vehicle 200. As an example, a lidar may be used for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of the vehicle 200. In addition, sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may represent a combination of sensors described herein (e.g., one or more lidars and radars; one or more lidars and cameras; one or more cameras and radars; or one or more lidars, cameras, and radars).

[0079] Notice, Figure 2A The number, location, and type of sensor systems (e.g., 202 and 204) depicted in -E are intended as non-limiting examples of the locations, number, and type of such sensor systems for an autonomous or semi-autonomous vehicle. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to conform to vehicle size, shape, aerodynamics, fuel economy, aesthetics, or other conditions, to reduce cost, or to accommodate specialized environments or applications). For example, the sensor systems (e.g., 202 and 204) can be disposed at various other locations on the vehicle (e.g., at location 216) and can have a field of view corresponding to the interior and / or surroundings of the vehicle 200.

[0080] The sensor system 202 may be mounted on top of the vehicle 200 and may include one or more sensors configured to detect information about the environment around the vehicle 200 and output an indication of the information. For example, the sensor system 202 may include any combination of cameras, radars, lidars, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). The sensor system 202 may include one or more movable mounts that may be operable to adjust the orientation of one or more sensors in the sensor system 202. In one embodiment, the movable mount may include a rotating platform that may scan the sensor to obtain information from every direction around the vehicle 200. In another embodiment, the movable mount of the sensor system 202 may move in a scanning manner within a specific range of angles and / or azimuths and / or altitudes. The sensor system 202 may be mounted on top of the roof of the car, but other mounting locations are also possible.

[0081] In addition, the sensors of sensor system 202 may be distributed at different locations and need not be collocated at a single location. Furthermore, each sensor of sensor system 202 may be configured to move or scan independently of the other sensors of sensor system 202. Additionally or alternatively, multiple sensors may be installed at one or more of sensor locations 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two lidar devices installed at a sensor location and / or there may be one lidar device and one radar installed at a sensor location.

[0082] One or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more laser radar devices. For example, a laser radar device may include a plurality of light emitter devices arranged within a certain range of angles relative to a given plane (e.g., an xy plane). For example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot around an axis perpendicular to the given plane (e.g., a z-axis) so as to illuminate the environment surrounding the vehicle 200 with light pulses. Based on detecting various aspects of the reflected light pulses (e.g., elapsed flight time, polarization, and intensity), information about the surrounding environment may be determined.

[0083] In an example embodiment, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide corresponding point cloud information that may be associated with physical objects within the surrounding environment of vehicle 200. Although vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are shown as including certain features, it should be understood that other types of sensor systems are contemplated within the scope of the present disclosure. In addition, example vehicle 200 may include a combination of Figure 1 Any components described with respect to vehicle 100 .

[0084] In an example configuration, one or more radars may be located on the vehicle 200. Similar to the above-described radar 126, one or more radars may include an antenna configured to send and receive radio waves (e.g., electromagnetic waves with a frequency between 30 Hz and 300 GHz). Such radio waves may be used to determine the distance to one or more objects in the surrounding environment of the vehicle 200 and / or the speed of one or more objects in the surrounding environment of the vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some examples, one or more radars may be located near the rear of the vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of the vehicle 200 for the presence of radio reflective objects. Similarly, one or more radars may be located near the front of the vehicle 200 (e.g., sensor systems 212 or 214) to actively scan the environment near the front of the vehicle 200. The radar may be located, for example, in a position suitable for illuminating an area including the forward path of movement of the vehicle 200 without being obscured by other features of the vehicle 200. For example, the radar may be embedded and / or mounted in or near the front bumper, headlights, fairing, and / or hood, etc. In addition, one or more additional radars may be positioned to actively scan the sides and / or rear of the vehicle 200 for the presence of radio reflective objects, such as by including such devices in or near the rear bumper, side panels, rocker panels, and / or chassis, etc.

[0085] The vehicle 200 may include one or more cameras. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may include one or more cameras. The camera may be a light-sensitive instrument, such as a static camera, a video camera, a thermal imaging camera, a stereo camera, a night vision camera, etc., which is configured to capture multiple images of the surrounding environment of the vehicle 200. To this end, the camera may be configured to detect visible light, and may additionally or alternatively be configured to detect light from other parts of the spectrum, such as infrared light or ultraviolet light. The camera may be a two-dimensional detector, and may optionally have a sensitivity of a three-dimensional spatial range. In some embodiments, the camera may include, for example, a range detector, which is configured to generate a two-dimensional image indicating the distance from the camera to multiple points in the surrounding environment. To this end, the camera may use one or more range detection techniques. For example, the camera may provide range information by using structured light technology, wherein the vehicle 200 illuminates an object in the surrounding environment with a predetermined light pattern (such as a grid or chessboard pattern), and uses the camera to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, the vehicle 200 can determine the distance to a point on the object. The predetermined light pattern may include infrared light or radiation of other suitable wavelengths for such measurement. In some examples, the camera may be mounted within the front windshield of the vehicle 200. Specifically, the camera may be positioned to capture images from a directional forward view relative to the vehicle 200. Other mounting positions and viewing angles of the camera may also be used, either inside or outside the vehicle 200. In addition, the camera may have associated optics that are operable to provide an adjustable field of view. Further, the camera may be mounted to the vehicle 200 using a movable mount to change the pointing angle of the camera, such as via a pan / tilt mechanism.

[0086] The vehicle 200 may also include one or more acoustic sensors for sensing the surrounding environment of the vehicle 200 (e.g., one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). The acoustic sensor may include a microphone (e.g., a piezoelectric microphone, a capacitive microphone, a ribbon microphone, or a micro-electromechanical system (MEMS) microphone) for sensing sound waves (i.e., pressure differences) in a fluid (e.g., air) surrounding the vehicle 200. Such an acoustic sensor may be used to identify sounds in the surrounding environment (e.g., sirens, human voices, animal sounds, or alarms) on which the control strategy of the vehicle 200 may be based. For example, if the acoustic sensor detects a siren (e.g., a patrol siren or a fire truck siren), the vehicle 200 may slow down and / or navigate to the edge of the road.

[0087] Although not in Figure 2A-2E, but the vehicle 200 may include a wireless communication system (e.g., similar to Figure 1 The wireless communication system 146 and / or in addition Figure 1 The wireless communication system may include a wireless transmitter and a receiver that may be configured to communicate with devices external or internal to the vehicle 200. Specifically, the wireless communication system may include a transceiver configured to communicate with other vehicles and / or computing devices, for example, in a vehicle communication system or a road station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other proposed communication standards for intelligent transportation systems.

[0088] Vehicle 200 may include one or more other components in addition to or in lieu of those shown. Additional components may include electrical or mechanical functionality.

[0089] The control system of the vehicle 200 can be configured to control the vehicle 200 according to a control strategy from a plurality of possible control strategies. The control system can be configured to receive information from sensors coupled to the vehicle 200 (on or off the vehicle 200), modify the control strategy (and associated driving behavior) based on the information, and control the vehicle 200 according to the modified control strategy. The control system can also be configured to monitor the information received from the sensors and continuously evaluate the driving conditions; and can also be configured to modify the control strategy and driving behavior based on changes in driving conditions. For example, the route taken by the vehicle from one destination to another can be modified based on driving conditions. Additionally or alternatively, speed, acceleration, turning angle, following distance (i.e., the distance to the vehicle in front of the current vehicle), lane selection, etc. can all be modified in response to changes in driving conditions.

[0090] Figure 3 3 is a conceptual illustration of wireless communications between various computing systems associated with an autonomous or semi-autonomous vehicle according to an example embodiment. In particular, wireless communications may occur between a remote computing system 302 and vehicle 200 via a network 304. Wireless communications may also occur between a server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0091] The vehicle 200 may correspond to various types of vehicles capable of transporting passengers or objects between locations, and may take the form of any one or more of the vehicles discussed above. In some cases, the vehicle 200 may operate in an autonomous or semi-autonomous mode that enables a control system to safely navigate the vehicle 200 between destinations using sensor measurements. When operating in an autonomous or semi-autonomous mode, the vehicle 200 may navigate with or without passengers. As a result, the vehicle 200 may pick up and drop off passengers between desired destinations.

[0092] Remote computing system 302 may represent any type of device associated with remote assistance technology, including but not limited to those described herein. In an example, remote computing system 302 may represent any type of device that is configured to (i) receive information associated with vehicle 200, (ii) provide an interface through which an operator can perceive the information and input responses associated with the information, and (iii) send responses to vehicle 200 or other devices. Remote computing system 302 may take various forms, such as a workstation, a desktop computer, a laptop computer, a tablet computer, a mobile phone (e.g., a smart phone), and / or a server. In some examples, remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0093] The remote computing system 302 may include one or more subsystems and components similar or identical to the subsystems and components of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface including input / output devices such as a touch screen and a speaker. Other examples are also possible.

[0094] Network 304 represents an infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0095] The location of remote computing system 302 may vary within examples. For example, remote computing system 302 may have a remote location from vehicle 200 with wireless communication via network 304. In another example, remote computing system 302 may correspond to a computing device within vehicle 200 that is separate from vehicle 200, but with which a human operator may interact simultaneously with a passenger or driver of vehicle 200. In some examples, remote computing system 302 may be a computing device with a touch screen that may be operated by a passenger of vehicle 200.

[0096] In some embodiments, the operations described herein performed by remote computing system 302 may additionally or alternatively be performed by vehicle 200 (i.e., by any system or subsystem of vehicle 200). In other words, vehicle 200 may be configured to provide a remote assistance mechanism with which a driver or passenger of the vehicle may interact.

[0097] The server computing system 306 may be configured to communicate wirelessly with the remote computing system 302 and the vehicle 200 via the network 304 (or possibly directly with the remote computing system 302 and / or the vehicle 200). The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information related to the vehicle 200 and its remote assistance. As such, the server computing system 306 may be configured to perform any operations or portions of such operations described herein as being performed by the remote computing system 302 and / or the vehicle 200. Some embodiments of wireless communications related to remote assistance may utilize the server computing system 306, while other embodiments may not utilize the server computing system 306.

[0098] The server computing system 306 may include one or more subsystems and components similar to or identical to the subsystems and components of the remote computing system 302 and / or the vehicle 200 , such as a processor configured to perform the various operations described herein, and a wireless communication interface for receiving information from the remote computing system 302 and the vehicle 200 and providing information to the remote computing system 302 and the vehicle 200 .

[0099] The various systems described above can perform various operations. These operations and related features will now be described.

[0100] As discussed above, a computing system (e.g., remote computing system 302, server computing system 306, or a computing system local to vehicle 200) may be operable to use a camera to capture images of the surroundings of an autonomous or semi-autonomous vehicle. Typically, at least one computing system will be able to analyze the images and possibly control the autonomous or semi-autonomous vehicle.

[0101] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data representing objects in the environment surrounding the vehicle (also referred to herein as "environmental data") in various ways. A sensor system on the vehicle may provide environmental data representing objects in the surrounding environment. For example, a vehicle may have various sensors, including cameras, radars, lidars, microphones, radio units, and other sensors. Each of these sensors may transmit environmental data about the information received by each respective sensor to a processor in the vehicle.

[0102] In one example, the camera can be configured to capture still images and / or videos. In some embodiments, the vehicle can have more than one camera positioned in different orientations. Moreover, in some embodiments, the camera can be capable of moving to capture images and / or videos in different directions. The camera can be configured to store the captured images and videos to a memory for later processing by the vehicle's processing system. The captured images and / or videos can be environmental data. In addition, the camera can include an image sensor as described herein.

[0103] In another example, the radar can be configured to transmit electromagnetic signals that will be reflected by various objects near the vehicle, and then capture the electromagnetic signals reflected from the objects. The captured reflected electromagnetic signals can enable the radar (or processing system) to make various determinations about the objects that reflected the electromagnetic signals. For example, the distances to various reflecting objects and the positions of various reflecting objects can be determined. In some embodiments, the vehicle can have more than one radar in different orientations. The radar can be configured to store the captured information to a memory for later processing by the vehicle's processing system. The information captured by the radar can be environmental data.

[0104] In another example, the lidar may be configured to emit an electromagnetic signal (e.g., infrared light, such as infrared light from a gas or diode laser or other possible light source) that will be reflected by a target object near the vehicle. The lidar may be capable of capturing the reflected electromagnetic (e.g., infrared light) signal. The captured reflected electromagnetic signal may enable a range finding system (or processing system) to determine the distance to various objects. The lidar may also be capable of determining the speed or velocity of the target object and storing it as environmental data.

[0105] In addition, in an example, the microphone may be configured to capture audio of the environment surrounding the vehicle. The sounds captured by the microphone may include emergency vehicle sirens and sounds of other vehicles. For example, the microphone may capture the sound of sirens of an ambulance, fire truck, or police car. The processing system may be able to recognize that the captured audio signal indicates an emergency vehicle. In another example, the microphone may capture the sound of the exhaust of another vehicle, such as the sound of the exhaust from a motorcycle. The processing system may be able to recognize that the captured audio signal indicates a motorcycle. The data captured by the microphone may form part of the environmental data.

[0106] In yet another example, the radio unit may be configured to send an electromagnetic signal, which may take the form of a Bluetooth signal, an 802.11 signal, and / or other radio technology signal. The first electromagnetic radiation signal may be sent via one or more antennas located in the radio unit. In addition, the first electromagnetic radiation signal may be sent using one of many different radio signaling modes. However, in some embodiments, it is desirable to send the first electromagnetic radiation signal in a signaling mode that requests a response from a device located near an autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on the response transmitted back to the radio unit, and use the transmitted information as part of the environmental data.

[0107] In some embodiments, the processing system may be able to combine information from various sensors to further determine the surroundings of the vehicle. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, the processing system may use other combinations of sensor data to make determinations about the surroundings.

[0108] When operating in autonomous mode (or semi-autonomous mode), the vehicle can control its operation with little to no human input. For example, a human operator can enter an address into the vehicle, and the vehicle can then be able to drive to the designated destination without further input from the human (e.g., the human does not have to steer or touch the brake / accelerator pedal). In addition, when the vehicle is operating autonomously or semi-autonomously, the sensor system can receive environmental data. The vehicle's processing system can change the control of the vehicle based on environmental data received from various sensors. In some examples, the vehicle can change the speed of the vehicle in response to environmental data from various sensors. The vehicle can change speed to avoid obstacles, comply with traffic regulations, etc. When the processing system in the vehicle identifies an object near the vehicle, the vehicle may be able to change speed, or change movement in another way.

[0109] When the vehicle detects an object but is not highly confident in the detection of the object, the vehicle may request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) confirming whether the object is actually present in the surrounding environment (e.g., if a stop sign is actually present or if a stop sign is not actually present), (ii) confirming whether the vehicle's identification of the object is correct, (iii) correcting the identification if the identification is incorrect, and / or (iv) providing supplemental instructions to the autonomous or semi-autonomous vehicle (or modifying current instructions). Remote assistance tasks may also include the human operator providing instructions to control the operation of the vehicle (e.g., if the human operator determines that the object is a stop sign, instructing the vehicle to stop at the stop sign), although in some scenarios, the vehicle itself may control its own operation based on the human operator's feedback related to the identification of the object.

[0110] To facilitate this, the vehicle may analyze environmental data representing objects of the surrounding environment to determine at least one object having a detection confidence below a threshold. A processor in the vehicle may be configured to detect various objects of the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important to the vehicle to identify. Such objects may include pedestrians, cyclists, street signs, other vehicles, indicator signals on other vehicles, and other various objects detected in the captured environmental data.

[0111] The detection confidence may indicate the likelihood that the determined object is correctly identified in the surrounding environment or exists in the surrounding environment. For example, the processor may perform object detection on the objects within the image data in the received environmental data, and determine that at least one object has a detection confidence below a threshold based on the inability to identify an object with a detection confidence above a threshold. If the result of object detection or object recognition of an object is uncertain, the detection confidence may be low or below a set threshold.

[0112] The vehicle can detect objects in the surrounding environment in various ways depending on the source of the environmental data. In some embodiments, the environmental data can come from a camera and be image or video data. In other embodiments, the environmental data can come from a lidar. The vehicle can analyze the captured image or video data to identify objects in the image or video data. The method and apparatus can be configured to monitor the image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data can be radar, audio, or other data. The vehicle can be configured to identify objects in the surrounding environment based on radar, audio, or other data.

[0113] In some embodiments, the technology used by the vehicle to detect objects can be based on a set of known data. For example, data related to environmental objects can be stored to a memory located in the vehicle. The vehicle can compare the received data with the stored data to determine the object. In other embodiments, the vehicle can be configured to determine the object based on the context of the data. For example, street signs associated with construction can generally have an orange color. Therefore, the vehicle can be configured to detect objects that are orange and located near the side of the road as construction-related street signs. In addition, when the vehicle's processing system detects an object in the captured data, it can also calculate a confidence level for each object.

[0114] In addition, the vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object being detected. For example, the confidence threshold may be lower for objects that may require a quick response action from the vehicle, such as brake lights on another vehicle. However, in other embodiments, the confidence threshold may be the same for all detected objects. When the confidence associated with a detected object is greater than the confidence threshold, the vehicle may assume that the object was correctly identified and responsively adjust the control of the vehicle based on that assumption.

[0115] When the confidence level associated with the detected object is less than a confidence threshold, the action taken by the vehicle may vary. In some embodiments, the vehicle may react as if the detected object is present despite the low confidence level. In other embodiments, the vehicle may react as if the detected object is not present.

[0116] When the vehicle detects an object in the surrounding environment, it can also calculate a confidence level associated with the specific detected object. The confidence level can be calculated in various ways depending on the embodiment. In one example, when an object in the surrounding environment is detected, the vehicle can compare the environmental data with predetermined data related to known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle can use mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0117] In response to determining that the object has a detection confidence level below a threshold, the vehicle may send a remote assistance request for identification of the object to the remote computing system. As described above, the remote computing system may take various forms. For example, the remote computing system may be a computing device within the vehicle that is separate from the vehicle, but with which a human operator may interact simultaneously with a passenger or driver of the vehicle, such as a touch screen interface for displaying remote assistance information. Additionally or alternatively, as another example, the remote computing system may be a remote computer terminal or other device located at a location not proximate to the vehicle.

[0118] The remote assistance request may include environmental data including objects, such as image data, audio data, etc. The vehicle may send the environmental data to the remote computing system over a network (e.g., network 304) and in some embodiments via a server (e.g., server computing system 306). A human operator of the remote computing system may in turn use the environmental data as a basis for responding to the request.

[0119] In some embodiments, when an object is detected with a confidence level below a confidence threshold, a preliminary identification of the object may be given, and the vehicle may be configured to adjust the operation of the vehicle in response to the preliminary identification. Such operational adjustments may take the form of stopping the vehicle, switching the vehicle to a human control mode, changing the speed (e.g., velocity and / or direction) of the vehicle, and other possible adjustments.

[0120] In other embodiments, the vehicle may operate in accordance with the detected object even if the vehicle detects an object with a confidence level that meets or exceeds a threshold (e.g., stop if the object is identified with a high confidence level as a stop sign), but may be configured to request remote assistance while (or later) the vehicle is operating in accordance with the detected object.

[0121] Figure 4A is a block diagram of a system according to an example embodiment. Specifically, Figure 4A System 400 is shown including system controller 402, lidar device 410, multiple sensors 412, and multiple controllable components 414. System controller 402 includes processor 404, memory 406, and instructions 408 stored on memory 406 and executable by processor 404 to perform functions.

[0122] Processor 404 may include one or more processors, such as one or more general-purpose microprocessors (e.g., with a single core or multiple cores) and / or one or more special-purpose microprocessors. The one or more processors may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein.

[0123] The memory 406 may include computer-readable media, such as non-transitory computer-readable media, which may include, but are not limited to, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), a solid-state drive (SSD), a hard disk drive (HDD), a compact disk (CD), a digital video disk (DVD), a digital tape, a read / write (R / W) CD, a R / W DVD, etc.

[0124] The lidar device 410, described further below, includes a plurality of light emitters configured to emit light (e.g., in the form of light pulses) and one or more light detectors configured to detect the light (e.g., reflected portions of the light pulses). The lidar device 410 can generate three-dimensional (3D) point cloud data from the outputs of the light detectors and provide the 3D point cloud data to the system controller 402. The system controller 402 can then perform operations on the 3D point cloud data to determine characteristics of the surrounding environment (e.g., relative positions of objects within the surrounding environment, edge detection, object detection, and proximity sensing).

[0125] Similarly, system controller 402 can use outputs from multiple sensors 412 to determine characteristics of system 400 and / or characteristics of the surrounding environment. For example, sensor 412 can include one or more of a GPS, an IMU, an image capture device (e.g., a camera), a light sensor, a thermal sensor, and other sensors that indicate parameters related to system 400 and / or the surrounding environment. For purposes of example, lidar device 410 is depicted as being separate from sensor 412, and in some examples may be considered part of or considered to be sensor 412.

[0126] Based on characteristics of system 400 and / or the surrounding environment determined by system controller 402 based on outputs from lidar device 410 and sensor 412, system controller 402 may control controllable components 414 to perform one or more actions. For example, system 400 may correspond to a vehicle, in which case controllable components 414 may include a braking system, a turning system, and / or an acceleration system of the vehicle, and system controller 402 may change aspects of these controllable components based on characteristics determined from lidar device 410 and / or sensor 412 (e.g., when system controller 402 controls the vehicle in an autonomous or semi-autonomous mode). In the example, lidar device 410 and sensor 412 may also be controlled by system controller 402.

[0127] Figure 4B is a block diagram of a laser radar device according to an example embodiment. In particular, Figure 4BA lidar device 410 is shown having a controller 416 configured to control a plurality of light emitters 424 and one or more light detectors, such as a plurality of light detectors 426, etc. The lidar device 410 also includes a firing circuit 428 configured to select and provide power to a corresponding light emitter in the plurality of light emitters 424, and may include a selector circuit 430 configured to select a corresponding light detector in the plurality of light detectors 426. The controller 416 includes a processor 418, a memory 420, and instructions 422 stored on the memory 420.

[0128] Similar to processor 404, processor 418 may include one or more processors, such as one or more general-purpose microprocessors and / or one or more special-purpose microprocessors. The one or more processors may include, for example, one or more CPUs, one or more microcontrollers, one or more GPUs, one or more TPUs, one or more ASICs, and / or one or more FPGAs. Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein.

[0129] Similar to memory 406, memory 420 may include computer-readable media, such as non-transitory computer-readable media, such as but not limited to ROM, PROM, EPROM, EEPROM, non-volatile random access memory (e.g., flash memory), SSD, HDD, CD, DVD, digital tape, R / W CD, R / W DVD, etc.

[0130] Instructions 422 are stored on memory 420 and are executable by processor 418 to perform functions associated with controlling firing circuit 428 and selector circuit 430 for generating 3D point cloud data, and for processing the 3D point cloud data (or potentially facilitating processing of the 3D point cloud data by another computing device, such as system controller 402).

[0131] The controller 416 can determine the 3D point cloud data by emitting light pulses using the light emitter 424. The emission time is established for each light emitter, and the relative position at the time of emission is also tracked. Various aspects of the surrounding environment of the laser radar device 410, such as various objects, reflect the light pulses. For example, when the laser radar device 410 is in a surrounding environment including a road, such objects may include vehicles, signs, pedestrians, road surfaces, or building cones. Some objects may be more reflective than other objects, so that the intensity of the reflected light can indicate the type of object reflecting the light pulse. In addition, the surface of the object may be in different positions relative to the laser radar device 410, and therefore take more or less time to reflect part of the light pulse back to the laser radar device 410. Therefore, the controller 416 can track the detection time when the reflected light pulse is detected by the light detector and the relative position of the light detector at the detection time. By measuring the time difference between the emission time and the detection time, the controller 416 can determine how far the light pulse traveled before being received, and thus determine the relative distance of the corresponding object. By tracking the relative position of the emission time and the detection time, the controller 416 can determine the orientation of the light pulses and the reflected light pulses relative to the lidar device 410, and thus determine the relative orientation of the object. By tracking the intensity of the received light pulses, the controller 416 can determine how reflective the object is. Therefore, the 3D point cloud data determined based on this information can indicate the relative position (e.g., within a coordinate system such as a Cartesian coordinate system) of the detected reflected light pulses and the intensity of each reflected light pulse.

[0132] Ignition circuit 428 is used to select a light emitter for emitting a light pulse. Selector circuit 430 is similarly used to sample the output from the light detector.

[0133] Figure 5 is a block diagram of a device 500 according to an example embodiment. The device 500 includes a plurality of time-of-flight sensor (ToF) pixels 510. In some embodiments, the ToF pixels 510 may be configured to detect light having a wavelength of 850 nm, 905 nm, or 940 nm. Other wavelengths are possible and contemplated.

[0134] ToF pixel 510 may include elements of a ToF sensor. A ToF sensor can measure the distance to an object by determining the time it takes for a pulse of light to travel from the sensor to the object and back. ToF sensors are used in a variety of applications, including automotive, robotics, augmented reality, and gesture recognition.

[0135] In some examples, a ToF sensor may operate as follows: The sensor may include one or more light emitters that emit light pulses. The light pulses travel to the object and are reflected back to the sensor. The sensor measures the time it takes for the light pulses to travel to the object and back. The distance to the object is calculated by dividing the speed of light by the time it takes for the light pulses to travel to the object and back. ToF sensors can be divided into two main types: active and passive. Active ToF sensors emit their own light source, while passive ToF sensors use reflected light from an external light source.

[0136] In various examples, the ToF pixel 510 may include a square shape and measure no more than 10 μm in length and width. It should be understood that other sizes and shapes are possible and contemplated. For example, a rectangular pixel with a longest side no longer than 10 μm is possible.

[0137] In some embodiments, various elements of the device 500 (including the ToF pixel 510) can be configured to operate as a direct time-of-flight (dToF) sensor and / or an indirect time-of-flight (iToF) sensor. In an example embodiment, a direct ToF sensor emits a short pulse of light and then measures the time it takes for the light to return. The distance to the object is then calculated by dividing the speed of light by the time it takes for the light to travel to the object and back. Additionally or alternatively, an indirect ToF sensor emits a continuous light beam and measures the phase shift between the emitted light and the reflected light. The phase shift is proportional to the distance to the object. In some embodiments, dToF sensors can provide higher resolution and accuracy, while iToF sensors can be cheaper and have lower power consumption.

[0138] The device 500 includes a plurality of visible image sensor (VIS) pixels 520. In some examples, the VIS pixels 520 are configured to detect wavelengths between 380 nm and 700 nm. The VIS pixels 520 can be similar or identical to imaging elements from visible camera sensors. For example, the VIS pixels 520 can include photosensitive elements for capturing images in visible light wavelengths. In this case, each pixel can include a photodetector configured to convert light into an electrical signal and a storage element that stores the electrical signal until it can be read out.

[0139] In some embodiments, the VIS pixel 520 may include a semiconductor material that is sensitive to light, such as silicon or germanium. However, other semiconductor materials (e.g., III-V materials) are possible and contemplated. When light strikes a photodetector, the photons can be converted into electrons via the photoelectric effect. These electrons are then collected by a storage element, which is typically a capacitor. The amount of charge stored in the capacitor is proportional to the amount of light that strikes the photodetector. The charge is then read out and converted into a digital signal, which can be used to create an image.

[0140] In some imaging applications, each pixel may also include a color filter. In this case, the color filter only allows specific wavelengths of light to reach the photodetector. This is how the camera is able to capture images of different colors. The most common color filter pattern used in digital cameras is the Bayer pattern. The Bayer pattern uses a grid of red, green, and blue color filters, where each pixel contains either a red, green, or blue filter. This pattern ensures that each pixel in the image contains information about all three colors.

[0141] It should be understood that the VIS pixel 520 can be sensitive to light in other wavelength ranges. As an example, the VIS pixel 520 can be configured to detect and / or image near-infrared and / or infrared light. In such a scenario, if a spatial needle grid is emitted to provide illumination for the dToF sensor, the needle grid pattern can also be sensed by the VIS pixel 520 and used as structured light illumination. By utilizing near-infrared and / or infrared structured light, the VIS pixel 520 can advantageously provide a higher resolution fused 3D finger texture image. Various examples can include VIS pixels being set with a pixel pitch of no more than 2.1 μm.

[0142] The device 500 also includes a ToF communication channel 512 configured to provide ToF image data 514 indicative of light received by the one or more ToF pixels 510 .

[0143] Apparatus 500 additionally includes a VIS communication channel 514 configured to provide VIS image data 524 indicative of one or more VIS pixels 520 .

[0144] In some embodiments, a plurality of ToF pixels 510 and a plurality of VIS pixels 520 are arranged along a plane 502 (eg, a shared substrate).

[0145] In various examples, the plurality of ToF pixels 510 are disposed in an arrangement 532 between the plurality of VIS pixels 520 . In some scenarios, the arrangement 532 includes a two-dimensional grid alternating between a 5×5 square array of VIS pixels 510 and a single ToF pixel 520 .

[0146] Additionally or alternatively, the arrangement 532 may include a lattice 534 that may define the position of each subarray of the VIS pixels 520 and each ToF pixel 510. In this case, the lattice 534 may include at least one of a diamond lattice, a square lattice, a hexagonal lattice, a rectangular lattice, a tilted lattice, or an equilateral triangle lattice.

[0147] In some scenarios, the arrangement of the ToF pixels 510 and the VIS pixels 520 can be configured to provide a composite information frame 550. In such a scenario, the composite information frame 550 can be formed based on a first partial information frame 552 captured when the ToF pixels 510 and the VIS pixels 520 are in a first position and a second partial information frame 554 captured when the ToF pixels 510 and the VIS pixels 520 are in a second position. In the context of the present disclosure, an "information frame" may include a data file having image information provided in a desired format (e.g., JPEG, TIFF, RAW, and DNG). For example, the first partial information frame 552 may include image information based on a signal received by the ToF pixel 510, and the second partial information frame 554 may include image information based on a signal received by the VIS pixel 520. Therefore, the composite information frame 550 may include ToF and VIS pixel information initially captured at a first position and subsequently captured at a second position. Such information can be arranged in a desired data format based on the corresponding spatial position of a given pixel. In some examples, the composite information frame 550 may be the result of an image convolution involving addition, subtraction, multiplication, or another operation between the first partial information frame 552 and the second partial information frame 554. In some embodiments, the VIS pixels 520 may provide a composite resolution of 8.29 megapixels. It should be understood that other composite resolutions are possible and contemplated.

[0148] In some embodiments, the device 500 may include at least one optical element 540. In this case, the at least one optical element 540 may include one or more lenses 542. In various examples, other types of optical elements 540 are possible, including mirrors, prisms, color filters, diffractive optical elements, or beam splitters. In this case, the optical element 540 may be made of materials including glass, crystals (e.g., quartz or sapphire), plastics (e.g., PETG), metals, dielectrics, and / or semiconductors.

[0149] Additionally or alternatively, at least one optical element 540 may include a color filter array (CFA) 544. In some embodiments, CFA 544 may be formed using one or more thin films that may include one or more colored dyes. In some examples, the dye may include an organic material, such as a cyanine dye or an azo dye.

[0150] As an example, CFA 544 may include a Bayer color filter. In some embodiments, the Bayer color filter may include a mosaic pattern of red, green, and blue (RGB) color filters placed on individual pixels or pixel subarrays of the image sensor. In some examples, the pattern is arranged so that the number of green color filters is twice that of red or blue color filters.

[0151] Additionally or alternatively, CFA 544 may include an RCCC color filter. An RCCC color filter is a variation of a Bayer color filter, but it has a clear color filter instead of a green or blue color filter. This allows the RCCC color filter to capture more light than a Bayer color filter, which may be beneficial in low light conditions.

[0152] Additionally, CFA 544 may include an RCCB or RCCG color filter. An RCCB or RCCG color filter is a variation of a Bayer color filter, but instead of having a green color filter in every other pixel, it has a clear color filter.

[0153] Fig. 6A FIG. 6 is a cross-sectional view 600 of a time-of-flight sensor pixel 510 according to an example embodiment. Fig. 6A As shown, ToF pixel 510 may include a photosensitive photodiode semiconductor device that can be illuminated from the back side 602 of the substrate. In some examples, ToF pixel 510 may be illuminated under bias conditions (e.g., V BB1 and / or V BB2 ) is operated under the condition of absorbing p- semiconductor layer 604 through doped p+ back side layer 606. When light is absorbed near n1 interface region 608 and converted into electrons, charge modulator 610 (e.g., G1, G2) is activated. C and G2) and drain contacts 612 (e.g., FD1 and FD2) can be biased to effectively collect electrical signals. In some embodiments, the ToF pixel 510 can include an on-chip silicon-on-insulator (SOI) circuit 614, which can utilize a buried oxide (BOX) layer 616 on the top side surface 618 of the substrate. It should be understood that other formats, designs and / or structures of the ToF pixel 510 are possible and contemplated.

[0154] Figure 6B620 is a cross-sectional view of a visible sensor pixel 520 according to an example embodiment. In some examples, the visible sensor pixel 520 may be similar or identical to a similar structure in a complementary metal oxide semiconductor (CMOS) image sensor. In an example embodiment, various elements of the visible sensor pixel 520 may be formed in a p-type substrate 622. Specifically, the n-well 624, the p-well 626, and the deep p-well 628 may form an n-type photodiode, which may include a PN junction produced by doping a region of the substrate with a high concentration of donor impurities, which produces a layer of positive charge carriers (electrons) in the region. When light shines on the photodiode, the interaction produces electron-hole pairs. Electrons are attracted to the n-type region of the photodiode, while holes are attracted to the p-type substrate. This produces charge separation, where electrons accumulate in the n-type region and holes accumulate in the p-type substrate. The readout circuit 630 may be used to amplify and read out the charge collected in the photodiode. One or more gates may be used to control the charge flow between the photodiode and the readout circuit 630.

[0155] Although Fig. 6A and 6B A cross-sectional view of a ToF sensor pixel 510 and a VIS sensor pixel 520 is illustrated, but it should be understood that other embodiments of the two types of sensor pixels are possible and contemplated within the scope of the present disclosure. As an example, the ToF sensor pixel 510 and the VIS sensor pixel 520 can be configured to share the same substrate (e.g., silicon or a III-V substrate). For example, the ToF sensor pixel 510 and the VIS sensor pixel 520 can be feasibly mounted on the same p-type substrate. However, it should be understood that heterogeneous integration of the ToF sensor pixel 510 and the VIS sensor pixel 520 on the same silicon is also possible and contemplated.

[0156] Fig. 7A 700 is a top view of an arrangement 532 of ToF sensor pixels 510 and VIS sensor pixels 520 according to an example embodiment. Fig. 7A One particular arrangement of ToF sensor pixels 510 and VIS sensor pixels 520 is described, but it should be understood that a wide range of other arrangements are possible. In an example embodiment, ToF sensor pixels 510 and VIS sensor pixels 520 may be arranged in alternating rows or columns. Other geometric arrangements of sensor pixels are possible and contemplated.

[0157] Figure 7B is an illustration 720 of a color filter array 544 overlaying a sensor array according to an example embodiment. In some embodiments, the sensor array 722 may include an arrangement of ToF pixels 510 and / or VIS pixels 520.

[0158] Figure 8800 according to an example embodiment. Apparatus 800 may include a device (e.g., device 500). As an example, the device may include a plurality of time-of-flight sensor (ToF) pixels (e.g., ToF pixels 510) configured to detect light within a predetermined wavelength range.

[0159] The device may also include a plurality of visible image sensor (VIS) pixels (e.g., VIS pixels 520). In this case, the plurality of ToF pixels and the plurality of VIS pixels may be arranged along a first plane (e.g., plane 502). In various examples, the plurality of ToF pixels may be arranged in an arrangement (e.g., arrangement 532) between the plurality of VIS pixels.

[0160] The apparatus 800 also includes one or more actuators 820 configured to displace the device along the first plane. In some examples, the actuator 820 may include a piezoelectric actuator, a voice coil, and / or a micro-electromechanical (MEMS) actuator. It should be understood that other types of actuators are possible and expected. In addition, the actuator 820 can be configured to displace the device by a predetermined distance 822. In various examples, the actuator 820 can be configured to displace the device by approximately 10 μm along the y-axis. Other predetermined distances and directions are possible and expected.

[0161] In various examples, the actuator 820 can be configured to displace the device at a predetermined frequency (e.g., 60 Hz, 120 Hz, 1 kHz, etc.). Additionally or alternatively, the actuator 820 can be disposed on a printed circuit board (e.g., substrate 502). Furthermore, while some embodiments described herein relate to moving the device along a first plane, it should be understood that other ways of moving the device relative to incident light received by the system are contemplated and possible. For example, in various embodiments, the actuator 820 can be configured to displace one or more optical elements 540, lenses 542, or color filter arrays 544.

[0162] The apparatus 800 further includes one or more light emitter devices (e.g., vertical cavity surface emitting lasers (VCSELs) 810) configured to emit light within a predetermined wavelength range. Additionally or alternatively, the VCSEL 810 may be configured to emit light pulses at a predetermined pulse rate (e.g., 60 Hz, 120 Hz, 1 kHz, etc.). In various examples, the VCSEL 810 may be configured to emit light pulses. In this case, the light pulses may include a wavelength of 850 nm or 940 nm. Other emission wavelengths, such as those in the near infrared wavelength range (e.g., 750 nm-2500 nm), are also possible and contemplated. In some example embodiments, the light emitter devices may include fiber amplifiers, laser diodes, light emitting diodes (LEDs), and the like.

[0163] In some embodiments, device 800 may include at least one optical element (e.g., optical element 540). In this case, the at least one optical element may include one or more lenses (e.g., lens 542). Additionally or alternatively, the at least one optical element may include a color filter array (CFA) (e.g., CFA 544). As some examples, the CFA may be a Bayer color filter, an RCCC color filter, or an RCCG color filter.

[0164] Apparatus 800 may additionally or alternatively include a computing device configured to control one or more elements of the apparatus.

[0165] Fig. 9 is an illustration of a system 900 according to an example embodiment. System 900 includes a sensing layer 910. Sensing layer 910 includes a plurality of time-of-flight sensor (ToF) pixels (e.g., ToF pixels 510) configured to detect light from a field of view to form ToF image data (e.g., ToF image data 514).

[0166] Sensing layer 910 also includes a plurality of visible image sensor (VIS) pixels (eg, VIS pixels 520 ) configured to detect light from the field of view to form VIS image data (eg, VIS image data 524 ).

[0167] The system 900 also includes a readout layer 920. The readout layer 920 includes a readout circuit 922 configured to receive ToF image data and VIS image data. In various embodiments, the readout layer 920 can be configured to perform functions related to at least one of: analog-to-digital conversion (ADC), high dynamic range image capture, or cache memory storage.

[0168] The system 900 also includes a processing layer 930. The processing layer 930 includes a processing circuit 932 configured to provide at least one composite image (e.g., a composite information frame 550) based on at least one of the ToF image data or the VIS image data. In some embodiments, the processing circuit 932 may be configured to manage the motion, timing, and capture of the sensor. Additionally or alternatively, the processing circuit 932 may include one or more automotive safety integrity level (ASIL) circuits 942 configured to classify the risk of a given driving scenario. Without limitation, the processing circuit 932 may include one or more of the following: a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC).

[0169] In some embodiments, the processing layer 930 may include one or more neural networks that may be configured to generate an object list based on images detected in the ToF image data or the VIS image data. The processing layer 930 may additionally include one or more convolutional neural networks (CNNs) 50 and / or recurrent neural networks (RNNs) 960 that may be configured to track an object based at least on the object list.

[0170] In various embodiments, system 900 may include a communication interface 940. In such a scenario, communication interface 940 may be configured to provide information indicative of an object identified in an image. In some examples, communication interface 940 may be configured to interface with an external device or system.

[0171] In various examples, the sensing layer 910 and the readout layer 920 are communicatively linked. In some examples, the readout layer 920 and the processing layer 930 are communicatively linked. In some embodiments, the various layers can be physically coupled. For example, the corresponding substrates of the sensing layer 910 and the readout layer 920 can be mixed (e.g., bump bonded or otherwise physically coupled together). In some embodiments, the system 900 may include a clock circuit 970, which can be used to control time synchronization between two or more layers of the system 900. Additionally or alternatively, the clock circuit 970 can advantageously synchronize the electronic shutter timing of the image sensing pixel 910. In an example embodiment, various operations of the ToF pixel 510 and the VIS pixel 520 can be controlled by the clock circuit 970. As an example, the clock circuit 970 can provide a time reference for operations such as ToF / VIS electronic shutter timing, pixel readout sorting, ToF / VIS sensor exposure adjustment, ToF / VIS sensor gain adjustment, etc.

[0172] Additionally or alternatively, the handle layer 930 can be wire-bonded to a substrate (eg, substrate 502). In other examples, the readout layer 920 can be wire-bonded to the handle layer 930. Other ways of physically coupling the various layers are contemplated and possible.

[0173] Fig. 10A 1000 is an information flow diagram according to an example embodiment. Information flow diagram 1000 includes capturing ToF and VIS images using a ToF+VIS imager (e.g., device 500). ToF and VIS images can be sent in separate image streams (e.g., VIS image stream 1004 and ToF image stream 1006). The corresponding image streams can be provided to smart sensor 1008, which can include local image processing / object recognition hardware (e.g., processing layer 930). The output of smart sensor 1008 can be provided to central computing hardware 1010.

[0174] Fig. 10B1 shows a system 1020 according to an example embodiment. The system 1020 may include the system 900 and Fig. 9 Depicted sensing layer 910, readout layer 920, and processing layer 930. System 1020 may include a compact, hybrid arrangement of various hardware elements.

[0175] Fig.11 is a method 1100 according to an example embodiment.

[0176] Block 1102 includes causing one or more vertical cavity surface emitting lasers (VCSELs) (e.g., VCSEL 810) to emit light pulses. Other types of light emitter devices are possible and contemplated. In this case, the light pulses include light within a predetermined wavelength range (e.g., near infrared and / or infrared wavelength ranges).

[0177] Block 1104 includes receiving or determining first ToF image data based on reflected light pulses received by one or more ToF pixels (e.g., ToF pixel 510) at a first position. The received light is within a predetermined wavelength range. In an example embodiment, block 1104 may include capturing time-of-flight information from a scene using ToF pixel 510. In such a scene, the time-of-flight information may represent a time difference between emitting an illumination light pulse and receiving reflected light associated with the illumination light pulse from the scene. Thereafter, the distance to the object is calculated by multiplying the speed of light by the time of flight.

[0178] Frame 1106 includes receiving or determining first VIS image data based on light received by one or more VIS pixels (e.g., VIS pixel 520) at a first position. As an example, frame 1106 may include capturing visible light information from a scene using VIS pixel 520. In such a scene, capturing visible light information may include receiving light from a scene via a camera lens. It should be understood that VIS pixel 520 may also be configured to capture near infrared and / or infrared light. The received light is converted into an electrical signal that varies based on light intensity. The electrical signal may then be amplified and converted into a digital signal or stored in a charge capacitor for later readout. The analog or digital signal may then be processed to form an image.

[0179] Block 1108 includes causing an actuator (eg, actuator 820) to shift one or more ToF pixels and one or more VIS pixels from a first position to a second position. In some alternative embodiments, the actuator may be used to shift an optical device (eg, a lens) in an optical path.

[0180] Block 1110 includes determining second ToF image data based on reflected light pulses received by one or more ToF pixels at a second location. In an example embodiment, block 1110 may be similar to block 1104, except that the ToF pixels may be at different relative locations. In this case, the second ToF image data may be based on reflected light within a predetermined wavelength range (e.g., near infrared and / or infrared wavelengths).

[0181] Block 1112 includes determining second VIS image data based on light received by one or more VIS pixels at a second location. In an example embodiment, block 1112 may be similar to block 1106, except that the VIS pixels may be at different relative locations. In this case, the second VIS image data may be based on light received within visible light wavelengths. Additionally or alternatively, the second VIS image data may be based on light within other wavelength ranges, such as near infrared and / or infrared wavelengths.

[0182] Block 1114 includes determining composite VIS image data from the first VIS image data and the second VIS image data. In some embodiments, determining the composite VIS image data may include at least one of: (i) temporal denoising, (ii) 3D noise filtering, or (iii) super-resolution processing. In some examples, temporal denoising may include reducing noise using temporal information between image frames (e.g., the first VIS image data and the second VIS image data). In such a scenario, temporal denoising may utilize various processes including frame differencing, motion compensation, and / or adaptive filtering. In an example embodiment, 3D noise filtering may reduce noise using information from multiple frames (e.g., the first VIS image data and the second VIS image data) to reduce noise using processes such as block matching, non-local means, and wavelet-based noise filtering. In addition, in some examples, super-resolution image processing may enhance the resolution of an image by increasing the number of pixels using a process that may include pixel interpolation, edge sharpening, contrast adjustment, and / or a machine learning algorithm.

[0183] Block 1116 includes determining composite ToF image data (e.g., composite information frame 550) from the first ToF image data and the second ToF image data. In various examples, determining the composite ToF image data may include determining at least one higher resolution ToF image. In various examples, determining the higher resolution ToF image may include processes such as interpolation of the ToF image data, edge sharpening, contrast adjustment, image sharpening, or application of a machine learning algorithm.

[0184] Block 1118 includes generating at least one object depth map estimate from the composite ToF image data. In various embodiments, generating the object depth map estimate may be based on triangulating the distance to each object, applying a phase-based approach, or utilizing a learning-based approach using a machine learning algorithm.

[0185] Block 1120 includes providing, by a communication interface (e.g., communication interface 940), data indicative of the composite VIS image data and at least one object depth map estimate. In some examples, the data indicative of the composite VIS image data and at least one object depth map estimate may be provided to a central server, a vehicle, or a perception system.

[0186] In some embodiments, method 1100 may include generating an object list based on a composite image by a convolutional neural network (CNN). In some examples, a CNN may generate an object list using a region proposal process. The region proposal process may include extracting features (e.g., edges and corners) from a scene. The region proposal process may additionally include using features to generate a set of candidate regions. These regions may include small rectangular regions (e.g., bounding boxes) of an image. The candidate regions may then be classified by the CNN to identify the objects they contain. In some embodiments, a CNN may use a softmax classifier to assign probabilities to each object class for each region. The region with the highest probability for a particular object class is then considered to be the best candidate for the object. A list of objects from a scene may be generated by considering regions that have been classified as containing objects. Regions may be ranked by their probability scores, and the top-ranked regions may be included in the list.

[0187] Method 1100 may additionally include tracking objects based on a list of objects and timing information by a recurrent neural network (RNN). In various examples, the RNN can provide a list of objects from a scene by using a sequence modeling process. In such a scene, sequence modeling can include predicting the next element in the sequence based on the previous element. In the case of object detection, the sequence can include a sequence of pixels in an image. In this way, sequence modeling can include extracting features (e.g., object parts and object relationships) from the image. In addition, the RNN can use features to predict the next element in the sequence. This is done by using recursive connections, which allows the RNN to consider the previous elements in the sequence when predicting the next element. A list of objects from a scene can be generated by considering a sequence of predicted elements. Elements can be ranked by their probability scores, and the top-ranked elements can be included in the list.

[0188] The present disclosure is not limited to the specific embodiments described in the present application, which are intended to be illustrations of various aspects. Without departing from the spirit and scope of the present invention, many modifications and variations may be made, which will be apparent to those skilled in the art. In addition to those listed herein, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. These modifications and variations are intended to fall within the scope of the appended claims.

[0189] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the accompanying drawings, similar symbols generally identify similar components unless the context dictates otherwise. The example embodiments described herein and in the accompanying drawings are not meant to be limiting. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It is readily understood that the various aspects of the present disclosure as generally described herein and shown in the accompanying drawings may be arranged, replaced, combined, separated, and designed in a variety of different configurations, all of which are expressly contemplated herein.

[0190] With respect to any or all of the message flow diagrams, scenarios, and flow diagrams in the accompanying drawings and as discussed herein, each step, box, operation, and / or communication may represent the processing of information and / or the transmission of information according to the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, the operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may not be performed in the order shown or discussed, including substantially simultaneously or in reverse order, depending on the functionality involved. In addition, more or fewer boxes and / or operations may be used with any of the message flow diagrams, scenarios, and flow diagrams discussed herein. Figure 1 The message flow diagrams, scenarios and flow diagrams may be used together, and these message flow diagrams, scenarios and flow diagrams may be combined with each other partially or in whole.

[0191] The steps, blocks or operations representing information processing may correspond to circuits that may be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing information processing may correspond to a portion of a module, segment or program code (including associated data). The program code may include one or more instructions executable by a processor for implementing specific logical operations or actions in a method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a RAM, a disk drive, a solid-state drive or another storage medium.

[0192] In addition, the steps, boxes or operations representing one or more information transfers may correspond to information transfers between software and / or hardware modules in the same physical device. However, other information transfers may be between software modules and / or hardware modules in different physical devices.

[0193] The specific arrangements shown in the accompanying drawings should not be considered limiting. It should be understood that other embodiments may include more or less of each element shown in a given figure. In addition, some of the elements shown may be combined or omitted. In addition, example embodiments may include elements not shown in the figures.

[0194] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope being indicated by the appended claims.

Claims

1. A device comprising: Multiple time-of-flight sensor (ToF) pixels; a plurality of visible image sensor (VIS) pixels; a ToF communication channel configured to provide ToF image data indicative of one or more ToF pixels; as well as A VIS communication channel is configured to provide VIS image data indicative of one or more VIS pixels, wherein the plurality of ToF pixels and the plurality of VIS pixels are arranged along a plane, wherein the plurality of ToF pixels are disposed in an arrangement between the plurality of VIS pixels. 2 . The device of claim 1 , wherein the ToF pixel is configured to detect a light wavelength of at least one of: 850 nm or 940 nm.

3. The device according to claim 1, wherein: The measurement length or width of the ToF pixel does not exceed 10 μm.

4. The device according to claim 1, wherein: The VIS pixels are configured to detect wavelengths between 380 nm and 700 nm.

5. The device according to claim 1, wherein: The VIS pixels are arranged with a pixel pitch of not more than 2.1 μm.

6. The device according to claim 1, wherein the arrangement of the ToF pixels and the VIS pixels is configured to provide a composite information frame, wherein the composite information frame is formed by a first partial information frame captured when the ToF pixels and the VIS pixels are in a first position and a second partial information frame captured when the ToF pixels and the VIS pixels are in a second position.

7. The device according to claim 1, wherein: The arrangement comprises a two-dimensional grid alternating between a 5x5 square array of VIS pixels and a single ToF pixel.

8. The apparatus of claim 1, wherein the arrangement comprises a lattice.

9. The device according to claim 8, wherein: The lattice includes at least one of a rhombus lattice, a square lattice, a hexagonal lattice, a rectangular lattice, a tilted lattice, or an equilateral triangle lattice.

10. The device according to claim 1, wherein: The VIS pixels provide a composite resolution of 8.29 megapixels.

11. The device of claim 1, further comprising at least one optical element.

12. The device according to claim 11, wherein The at least one optical element comprises one or more lenses.

13. The device according to claim 11, wherein: The at least one optical element includes a color filter array (CFA).

14. The apparatus of claim 13, wherein the CFA is a Bayer color filter.

15. The apparatus of claim 13, wherein the CFA is an RCCC color filter.

16. The apparatus of claim 13, wherein the CFA is a RCCB color filter or a RCCG color filter.

17. A system comprising: A sensing layer, the sensing layer comprising: a plurality of time-of-flight sensor (ToF) pixels configured to detect light from a field of view to form ToF image data; and a plurality of visible image sensor (VIS) pixels configured to detect light from the field of view to form VIS image data; a readout layer, the readout layer comprising a readout circuit, the readout circuit being configured to receive the ToF image data and the VIS image data; and A processing layer comprising processing circuitry configured to provide at least one composite image based on at least one of the ToF image data or the VIS image data, wherein the sensing layer and the readout layer are communicatively linked, wherein the readout layer and the processing layer are communicatively linked.

18. The system of claim 17, wherein the readout layer is configured to perform a function related to at least one of: analog-to-digital conversion (ADC), high dynamic range image capture, or cache memory storage.

19. The system of claim 17, further comprising a communication interface, wherein the communication interface is configured to provide information indicative of an object identified in the image.

20. The system of claim 17, wherein: The processing layer includes: One or more of: a convolutional neural network (CNN), a recurrent neural network (RNN), a transformer, or a base model configured to generate a list of objects based on images detected in the ToF image data or the VIS image data; and One or more recurrent neural networks (RNNs) configured to track objects based at least on the list of objects.

Images