Image sensor with on-sensor and application-specific object recognition.

Abstract

An exemplary embodiment relates to an image sensor having on-sensor and application-specific object identification. [Solution] An exemplary embodiment includes a method. The method includes capturing image data relating to the surrounding environment using an image sensor. The method also includes receiving the image data captured from the image sensor using an analog-to-digital converter (ADC). The method further includes providing the converted image data to an application-specific integrated circuit (ASIC) using the ADC. The method further includes applying a trained machine learning model to the converted image data using the ASIC to identify one or more objects in the surrounding environment within the converted image data. The method further includes outputting an image frame using the ASIC, wherein at least one row of the image frame includes metadata, including object classification data and object location data for one or more identified objects in the surrounding environment.

Description

Background Art

[0001] Unless otherwise specified in this specification, the descriptions in this chapter are not prior art to the claims of this application and should not be considered prior art by including them in this chapter.

[0002] Computer vision tasks, particularly those related to the operation of autonomous or semi-autonomous vehicles, are often computationally intensive. In this particular case, a computer system often processes high-frame rate data from cameras and other sensors to identify objects in the vehicle's surrounding environment, such as other vehicles, traffic signals, and signs, pedestrians, cyclists, and debris on the road.

Summary of the Invention

[0003] The present disclosure relates to an image sensor with on-sensor and application-specific object identification. In contrast to alternative methods, exemplary embodiments herein enable offloading computer vision tasks from a centralized general-purpose processor to a dedicated processor directly coupled to the sensor. This allows for greater customization and configuration of the processor for tasks associated with each sensor. For example, a dedicated processor can apply a trained machine learning model to image data received from a sensor to identify objects for navigation of an autonomous or semi-autonomous vehicle.

[0004] In one embodiment, a system is provided. The system includes a vehicle, a first device mounted on the vehicle and aligned with a first orientation relative to the vehicle, and a second device mounted on the vehicle and aligned with a second orientation relative to the vehicle. The first and second orientations are distinct. The first device includes a first image sensor configured to capture first image data relating to the surrounding environment, a first analog-to-digital converter (ADC) configured to receive first image data captured from the first image sensor and provide converted first image data, and a first application-specific integrated circuit (ASIC). The first ASIC receives the converted first image data from the first ADC and is configured to apply a first trained machine learning model to the converted first image data to identify one or more first objects in the surrounding environment within the converted first image data. The first trained machine learning model is selected from a plurality of machine learning models based on the first orientation. The second device includes a second image sensor configured to capture second image data relating to the surrounding environment, a second ADC configured to receive second image data captured from the second image sensor and provide converted second image data, and a second ASIC. The second ASIC receives the transformed second image data from the second ADC and is configured to apply a second trained machine learning model to the transformed second image data to identify one or more second objects in the surrounding environment within the transformed second image data. The second trained machine learning model is selected from multiple machine learning models based on the second orientation.

[0005] In another embodiment, a device is provided. The device includes an image sensor configured to capture image data relating to the surrounding environment, an analog-to-digital converter (ADC) configured to receive the image data captured from the image sensor and provide the converted image data, and an application-specific integrated circuit (ASIC). The ASIC is configured to receive the converted image data from the ADC, apply a trained machine learning model to the converted image data to identify one or more objects in the surrounding environment within the converted image data, and output an image frame. At least one row of the image frame includes metadata, including object classification data and object location data for one or more identified objects in the surrounding environment.

[0006] In another embodiment, a method is provided. The method includes capturing image data relating to the surrounding environment using an image sensor. The method also includes receiving the image data captured from the image sensor using an analog-to-digital converter (ADC). The method further includes providing the converted image data to an application-specific integrated circuit (ASIC) using the ADC. The method further includes applying a trained machine learning model to the converted image data using the ASIC to identify one or more objects in the surrounding environment within the converted image data. The method further includes outputting an image frame using the ASIC. At least one row of the image frame includes metadata, including object classification data and object location data for one or more identified objects in the surrounding environment.

[0007] These, as well as other embodiments, advantages, and alternatives, will become apparent to those skilled in the art by reading the following detailed description with due reference to the accompanying drawings. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a functional block diagram showing a vehicle according to an exemplary embodiment. [Figure 2A]Figure 2A is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2B] Figure 2B is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2C] Figure 2C is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2D] Figure 2D is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2E] Figure 2E is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2F] Figure 2F is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2G] Figure 2G is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2H] Figure 2H is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2I] Figure 2I is an illustrative diagram of the physical configuration of a vehicle according to an exemplary embodiment. [Figure 2J] Figure 2J is an illustrative diagram of the field of view of various sensors according to an exemplary embodiment. [Figure 2K] Figure 2K is an illustrative diagram of beam steering for a sensor according to an exemplary embodiment. [Figure 3] Figure 3 is a conceptual diagram illustrating a model embodiment of wireless communication between various computing systems related to autonomous or semi-autonomous vehicles. [Figure 4A] Figure 4A is a block diagram of a system including a LiDAR device according to an exemplary embodiment. [Figure 4B] Figure 4B is a block diagram of a LiDAR device according to an exemplary embodiment. [Figure 5A] Figure 5A is a block diagram of a device according to an exemplary embodiment. [Figure 5B] Figure 5B is an illustrative diagram of a multilayer die stack according to an exemplary embodiment. [Figure 5C]Figure 5C is an illustrative diagram of a vehicle having a device according to an exemplary embodiment. [Figure 6A] Figure 6A is a block diagram of a process according to an exemplary embodiment. [Figure 6B] Figure 6B is a block diagram of a system according to an exemplary embodiment. [Figure 7A] Figure 7A is an illustrative diagram of an image frame according to an exemplary embodiment. [Figure 7B] Figure 7B is an illustrative diagram of an image frame including metadata, according to an exemplary embodiment. [Figure 7C] Figure 7C is an illustrative diagram of an image frame according to an exemplary embodiment. [Figure 7D] Figure 7D is an illustrative diagram of an image frame including metadata, according to an exemplary embodiment. [Figure 8] Figure 8 is an illustrative flowchart of the method according to an exemplary embodiment. [Figure 9] Figure 9 is an illustrative flowchart of the method according to an exemplary embodiment. [Modes for carrying out the invention]

[0009] Exemplary methods and systems are described in the present disclosure. Any exemplary embodiment or feature described herein should not necessarily be construed as more preferable or advantageous than other embodiments or configurations. Furthermore, the exemplary embodiments described herein are not meant to be limiting. Certain aspects of the disclosed systems and methods can be arranged and combined in a variety of different configurations, and it will be readily understood that all of these configurations are contemplated in the present disclosure. Additionally, the specific arrangements shown in the figures should not be considered as limiting. It should be understood that other embodiments can include more or fewer of each element shown in a given figure. Furthermore, some of the illustrated elements can be combined or omitted. Still further, the exemplary embodiments can include elements not illustrated in the figures.

[0010] The lidar device described herein can include one or more light-emitting elements and one or more detectors used to detect light emitted by the one or more light-emitting elements and reflected by one or more objects in the environment surrounding the lidar device. As an example, the surrounding environment can include an internal or external environment such as inside or outside a building. Additionally or alternatively, the surrounding environment can include the interior of a vehicle. Still further, the surrounding environment can include the vicinity around and / or on a road. Examples of objects in the surrounding environment include, but are not limited to, other vehicles, traffic signs, pedestrians, cyclists, road surfaces, buildings, terrain, etc. Furthermore, the one or more light-emitting elements can emit light into the local environment of the lidar itself. For example, the light emitted from the one or more light-emitting elements can interact with the housing of the lidar and / or a surface or structure coupled to the lidar. In some cases, the lidar can be attached to a vehicle, and in that case, the one or more light-emitting elements can be configured to emit light that interacts with objects in the vicinity of the vehicle. Additionally, the light-emitting elements can include, among other possibilities, optical fiber amplifiers, laser diodes, light-emitting diodes (LEDs).

[0011] As described above, in some embodiments, the application-specific tasks may vary based on the positioning of the sensor device with respect to its location and / or orientation on the autonomous or semi-autonomous vehicle, and such determination may occur automatically. For example, the sensor device may determine that it is positioned at the forward-facing location on the autonomous vehicle and automatically load the appropriate machine learning model. This allows for flexibility in adapting the sensor to different tasks and enables efficient updates to the machine learning model based on the preferences of the developer and / or deployer.

[0012] Embodiments herein also provide a technical improvement over current methods because the detection performed by the sensor is typically performed by a central processing unit or system, which adds latency to the object detection data flow as the image data is transmitted to the central processing system before any analysis is performed.

[0013] Considering the problems described above, embodiments of this specification enable computer vision tasks to be performed on onboard sensor devices, offloading processing tasks from a central processing unit. Thus, the sensor devices described herein may include an image sensor, an analog-to-digital converter (ADC), and an application-specific integrated circuit (ASIC). Image data may be collected by the image sensor, converted to a digital format by the ADC, and transmitted to the ASIC, which may then use a trained machine learning model to detect objects in the surrounding environment of the autonomous vehicle. This offers a technological improvement over current methods, as such detection is often performed by a central processing unit or system located away from the image capture device, adding a delay to the object detection data flow because the image data is transmitted to the central processing system before any analysis is performed. Furthermore, the central processing unit or system may also function as multiple image capture devices, further limiting the resources that can be dedicated to each. In contrast, the devices described above perform analysis and other computer vision tasks, significantly improving performance. This also allows the central processing system to allocate more resources to autonomous driving logic. Furthermore, by co-locating the processing components within the same device as the image sensor, it becomes possible to reduce delays and communication overhead in transmissions between each component.

[0014] In some embodiments, the device may include three layers within a multilayer die stack: (i) an image sensing layer (image sensor), (ii) an ADC layer (including an ADC, HDR processor, and cache), and (iii) a processing layer including an ASIC.

[0015] The following description and accompanying drawings illustrate the features of various exemplary embodiments. The embodiments provided are illustrative and not intended to be limiting. Thus, the dimensions in the drawings are not necessarily to scale.

[0016] Herein, exemplary systems within the scope of this disclosure will be described in more detail. Exemplary systems may be implemented in automobiles or may take the form of automobiles. Furthermore, exemplary systems may also be implemented in or take the form of various vehicles, such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor-trailers), motorcycles, buses, airplanes, helicopters, drones, lawnmowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, agricultural machinery or agricultural vehicles, construction machinery or construction vehicles, warehouse equipment or warehouse vehicles, factory equipment or factory vehicles, trams, golf carts, electric trains, trolleys, pedestrian transport vehicles, and robotic devices. Other vehicles are similarly possible. Furthermore, in some embodiments, exemplary systems may not include a vehicle.

[0017] Referring here to the figure, Figure 1 is a functional block diagram illustrating an exemplary vehicle 100 that may be configured to operate fully or partially in autonomous mode. More specifically, vehicle 100 may operate in autonomous mode without human interaction by receiving control commands from a computing system. As part of its operation in autonomous mode, vehicle 100 may use sensors to detect and, in some cases, identify objects in its surrounding environment to enable safe navigation. Furthermore, exemplary vehicle 100 may operate in a partially autonomous (i.e., semi-autonomous) mode in which some functions of vehicle 100 are controlled by a human driver of vehicle 100, and some functions of vehicle 100 are controlled by a computing system. For example, vehicle 100 may also include subsystems that allow the driver to control the operation of vehicle 100, such as steering, acceleration, and braking, while the computing system implements assistance functions, such as lane departure warning / lane keeping assist or adaptive cruise control, based on other objects in the surrounding environment (e.g., other vehicles).

[0018] As described herein, in a partially autonomous driving mode, the vehicle assists with steering, braking, and / or acceleration to perform one or more driving actions (e.g., lane centering, adaptive cruise control, advanced driver-assistance systems (ADAS), and emergency braking), but the human driver is expected to situationally perceive the vehicle's surroundings and supervise the assisted driving actions. Here, the vehicle may perform all driving tasks in certain situations, but the human driver is expected to be responsible for taking control as needed.

[0019] For the sake of simplification and brevity, various systems and methods are described below in conjunction with autonomous vehicles, but these, or similar systems and methods, may be used in various driver assistance systems that do not reach the level of fully autonomous driving systems (i.e., partially autonomous driving systems). In the United States, the Society of Automotive Engineers (SAE) defines different levels of automated driving behavior to indicate how much or how little a vehicle controls the driving, although different organizations in the United States or other countries may classify the levels differently. More specifically, the systems and methods of this disclosure may be used in SAE Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver assistance. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems that enable autonomous driving under limited conditions (e.g., highways). Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 automated driving systems that operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such systems, accurate lane estimation is performed automatically without driver input or control (e.g., while the vehicle is moving), which can result in improved reliability of vehicle positioning and navigation, as well as overall safety of autonomous, semi-autonomous, and other driver assistance systems. As stated above, in addition to the way the SAE classifies levels of autonomous driving operations, other organizations in the United States or other countries may classify levels of autonomous driving operations differently. The systems and methods disclosed herein may be used in driver assistance systems defined by the levels of autonomous driving operations of these other organizations, but are not limited to these.

[0020] As shown in Figure 1, the vehicle 100 may be configured to include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112 (which may also be called a computing system) having data storage 114, and a user interface 116. In other examples, the vehicle 100 may include more or fewer subsystems, each of which may contain multiple elements. The subsystems and components of the vehicle 100 may be interconnected in various ways. In addition, the functions of the vehicle 100 described herein may be divided into additional functional or physical components, or combined into fewer functional or physical components within an embodiment. For example, the control system 106 and the computer system 112 may be combined into a single system that operates the vehicle 100 according to various operations.

[0021] The propulsion system 102 may include one or more components that are capable of providing powered motion to the vehicle 100, and among other possible components, it may include an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121. For example, the engine / motor 118 may be configured to convert the energy source 119 into mechanical energy, and among other possible options, it may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine. For example, in some embodiments, the propulsion system 102 may include multiple types of engines and / or motors, such as a gasoline engine and an electric motor.

[0022] The energy source 119 represents an energy source that can power one or more systems of the vehicle 100 (e.g., engine / motor 118), either fully or partially. For example, the energy source 119 may correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, the energy source 119 may include a combination of a fuel tank, batteries, a capacitor, and / or a flywheel.

[0023] The transmission 120 can transmit mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. Thus, the transmission 120 may include, among other possible components, a gearbox, a clutch, a differential, and a drive shaft. The drive shaft may include an axle connected to one or more wheels / tires 121.

[0024] The wheels / tires 121 of the vehicle 100 can have various configurations within the exemplary embodiment. For example, the vehicle 100 can exist in the form of a unicycle, a bicycle / motorcycle, a tricycle, or a four-wheeled automobile / truck, among other possible configurations. Thus, the wheels / tires 121 may be attached to the vehicle 100 in various ways and can exist in different materials such as metal and rubber.

[0025] The sensor system 104 may include various types of sensors, among other possible sensors, including, in particular, a Global Positioning System (GPS) 122, an Inertial Measurement Unit (IMU) 124, radar 126, lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125. In some embodiments, the sensor system 104 may also include sensors (e.g., an O2 monitor, a fuel gauge, engine oil temperature, and brake wear) configured to monitor the internal systems of the vehicle 100.

[0026] The GPS 122 may be configured to include a transceiver capable of operating to provide information regarding the position of the vehicle 100 relative to the Earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes, and may sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100, whether the vehicle 100 is stationary or in motion.

[0027] The radar 126 may represent one or more systems configured to sense objects in the surrounding environment of the vehicle 100, including their speed and orientation, using radio signals. Thus, the radar 126 may include antennas configured to transmit and receive radio signals. In some embodiments, the radar 126 may correspond to a mountable radar configured to obtain measurements of the surrounding environment of the vehicle 100.

[0028] Lidar 128 may also include, among other system components, one or more laser sources, a laser scanner, and one or more detectors, and may operate in coherent mode (e.g., using heterodyne detection) or incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of Lidar 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes). In some embodiments, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Furthermore, such photodetectors may be arranged in an array (e.g., like silicon photomultiplier tubes (SiPMs)) (e.g., through series electrical connections). In some embodiments, one or more photodetectors are devices operating in Geiger mode, and Lidar includes sub-components designed for such Geiger mode operation.

[0029] The camera 130 may include one or more devices (e.g., a still camera, a video camera, a thermal imaging camera, a stereo camera, and a night vision camera) configured to capture images of the surrounding environment of the vehicle 100.

[0030] The steering sensor 123 may sense the steering angle of the vehicle 100, and this sensing may involve measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, the steering sensor 123 may measure the angle of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the front axis of the vehicle 100. The steering sensor 123 may also be configured to measure a combination (or subset) of the steering wheel angle, an electrical signal representing the angle of the steering wheel, and the angle of the wheels of the vehicle 100.

[0031] The throttle / brake sensor 125 can detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 can measure the angles of both the accelerator pedal (throttle) and the brake pedal, or it can measure an electrical signal that can represent, for example, the angle of the accelerator pedal (throttle) and / or the angle of the brake pedal. The throttle / brake sensor 125 can also measure the angle of the throttle body of the vehicle 100, which may include part of a physical mechanism that provides the engine / motor 118 (e.g., a butterfly valve and a carburetor) with control of the energy source 119. Furthermore, the throttle / brake sensor 125 can measure the pressure of one or more brake pads on the rotor of the vehicle 100, or a combination (or subset) of the angles of the accelerator pedal (throttle) and the brake pedal, an electrical signal representing the angles of the accelerator pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied by at least one brake pad to the rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure the pressure applied to a vehicle pedal, such as the throttle or brake pedal.

[0032] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a brake unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / route search system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the orientation of the vehicle 100, and the throttle 134 may control the operating speed of the engine / motor 118 to control the acceleration of the vehicle 100. The brake unit 136 can decelerate the vehicle 100, which may involve using friction to decelerate the wheels / tires 121. In some embodiments, the brake unit 136 may convert the kinetic energy of the wheels / tires 121 into an electric current for subsequent use by the system of the vehicle 100.

[0033] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithms capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide evaluations based on the received sensor data, such as evaluations of individual objects and / or features, evaluations of specific situations, and / or evaluations of possible effects within a given situation.

[0034] The computer vision system 140 may include hardware and software (e.g., a general-purpose processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor such as a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) that can operate to process and analyze images to determine objects that are in motion (e.g., other vehicles, pedestrians, cyclists, or animals) and objects that are not in motion (e.g., traffic signals, road boundaries, speed bumps, or potholes). Thus, the computer vision system 140 may use object recognition, structure reconstruction from motion (SFM), video tracking, and other algorithms used in computer vision, such as recognizing objects, mapping the environment, tracking objects, and estimating the velocity of objects.

[0035] The navigation / route search system 142 can determine the driving path of the vehicle 100, which may include dynamically adjusting the navigation during operation. In this way, the navigation / route search system 142 can navigate the vehicle 100 using data from, among other sources, the sensor fusion algorithm 138, GPS 122, and maps. The obstacle avoidance system 144 can evaluate potential obstacles based on sensor data and cause the vehicle 100's system to avoid or otherwise navigate around potential obstacles.

[0036] As shown in Figure 1, the vehicle 100 may also include peripherals 108, such as a wireless communication system 146, a touchscreen 148, an internal microphone 150, and / or a speaker 152. The peripherals 108 may provide control or other elements for the user to interact with the user interface 116. For example, the touchscreen 148 may provide information to the user of the vehicle 100. The user interface 116 may also accept input from the user via the touchscreen 148. The peripherals 108 may also enable the vehicle 100 to communicate with devices, such as devices in other vehicles.

[0037] The wireless communication system 146 can communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication such as Code Division Multiple Access (CDMA), Evolution Data Optimization (EVDO), Global System for Mobile Communications (GSM) / General-Purpose Packet Radio Service (GPRS), or cellular communication such as 4G Worldwide Interoperability for Microwave Access (WiMAX), or Long-Term Evolution (LTE), or 5G. Alternatively, the wireless communication system 146 may communicate with a wireless local area network (WLAN) using Wi-Fi® or other possible connections. The wireless communication system 146 may also communicate directly with devices using, for example, an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are possible within the context of this disclosure. For example, the wireless communication system 146 may include one or more dedicated narrow-area communication (DSRC) devices that may include public and / or private data communications between vehicles and / or roadside stations.

[0038] Vehicle 100 may include a power supply 110 for supplying power to its components. In some embodiments, the power supply 110 may include a rechargeable lithium-ion or lead-acid battery. For example, the power supply 110 may include one or more batteries configured to provide power. Vehicle 100 may also use other types of power supplies. In exemplary embodiments, the power supply 110 and an energy source 119 may be integrated to form a single energy source.

[0039] Vehicle 100 may also include a computer system 112 for performing operations, such as those described therein. Thus, the computer system 112 may include at least one processor 113 (which may include at least one microprocessor) capable of operating to execute instructions 115 stored in a non-temporary computer-readable medium, such as data storage 114. In some embodiments, the computer system 112 may represent a plurality of computing devices that can function to control individual components or subsystems of vehicle 100 in a distributed manner.

[0040] In some embodiments, the data storage 114 may be configured to include instructions 115 (e.g., program logic) executable by the processor 113 for performing various functions of the vehicle 100, including those described above in relation to Figure 1. The data storage 114 may also include additional instructions, including instructions for transmitting, receiving, interacting with, and / or controlling data to one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0041] In addition to instruction 115, data storage 114 may store other information, among other things, such as road maps and route information. Such information may be used by the vehicle 100 and the computer system 112 during the operation of the vehicle 100 in autonomous mode, semi-autonomous mode, and / or manual mode.

[0042] Vehicle 100 may include a user interface 116 for providing information to or receiving input from the user of vehicle 100. The user interface 116 may control or enable control of the layout of content and / or interactive images that may be displayed on the touchscreen 148. Furthermore, the user interface 116 may include one or more input / output devices within a set of peripherals 108, such as a wireless communication system 146, a touchscreen 148, a microphone 150, and a speaker 152.

[0043] The computer system 112 can control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, or the control system 106) and from the user interface 116. For example, the computer system 112 may utilize inputs from the sensor system 104 to estimate the outputs generated by the propulsion system 102 and the control system 106. Depending on the embodiment, the computer system 112 may be operable to monitor many aspects of the vehicle 100 and its subsystems. In some embodiments, the computer system 112 may disable some or all of the functions of the vehicle 100 based on signals received from the sensor system 104.

[0044] The components of the vehicle 100 may be configured to function in a manner that interconnects with other components, either internally or externally, to each of their respective systems. For example, in an exemplary embodiment, the camera 130 may capture multiple images that can represent information about the state of the surrounding environment of the vehicle 100 operating in autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, the computer vision system 140 may be able to recognize the slope (gradient) or other features based on multiple images of the road. Furthermore, the combination of the GPS 122 and the features recognized by the computer vision system 140 may be used together with map data stored in the data storage 114 to determine specific road parameters. In addition, radar 126, and / or lidar 128, and / or several other environmental mapping, range, and / or positioning sensor systems may also provide information about the vehicle's surroundings.

[0045] In other words, the combination of various sensors (which can be called input indicator sensors and output indicator sensors) and the computer system 112 can interact to provide indicators of inputs or indicators of the vehicle's surroundings that are provided for controlling the vehicle.

[0046] In some embodiments, the computer system 112 may make decisions about various objects based on data provided by systems other than the wireless system. For example, the vehicle 100 may have lasers or other optical sensors configured to sense objects within the vehicle's field of view. The computer system 112 may use the outputs from the various sensors to determine information about objects within the vehicle's field of view, and may measure distance and directional information to various objects. The computer system 112 may also determine whether an object is desirable or undesirable based on the outputs from the various sensors.

[0047] Figure 1 shows various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) integrated into vehicle 100, but one or more of these components may be mounted on vehicle 100 or provided separately. For example, data storage 114 may exist partially, entirely, or completely separately from vehicle 100. Thus, vehicle 100 may be provided in the form of device elements located in separate locations, or in the form of device elements located together in the same location. The device elements constituting vehicle 100 may be coupled together in a wired and / or wireless manner so as to be able to communicate together.

[0048] Figures 2A–2E show an exemplary vehicle 200 (e.g., a fully autonomous vehicle or a semi-autonomous vehicle) which may include some or all of the functions described in relation to vehicle 100 with reference to Figure 1. For illustrative purposes, vehicle 200 is shown in Figures 2A–2E as a van with side mirrors, but the disclosure is not limited thereto. For example, vehicle 200 may represent a truck, passenger car, semi-trailer truck, motorcycle, golf cart, off-road vehicle, agricultural vehicle, or any other vehicle described elsewhere in this specification (e.g., bus, boat, airplane, helicopter, drone, lawnmower, bulldozer, submarine, all-terrain vehicle, snowmobile, aircraft, recreation vehicle, amusement park vehicle, farm equipment, construction machinery or construction vehicle, warehouse equipment or warehouse vehicle, factory equipment or factory vehicle, tram, train, trolley, pedestrian transport vehicle, and robotic device).

[0049] An exemplary vehicle 200 may include one or more sensor systems, 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidar, one or more radar, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding the vehicle 200. In other words, any sensor system currently known or to be created may be coupled to the vehicle 200 and / or used in conjunction with various operations of the vehicle 200. For example, lidar may be used for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of the vehicle 200. In addition, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent combinations of sensors described herein (e.g., one or more lidar and radar, one or more lidar and cameras, one or more cameras and radar, one or more lidar, cameras and radar).

[0050] It should be noted that the number, location, and type of sensor systems (e.g., 202 and 204) shown in Figures 2A-E are intended as non-limiting embodiments of the location, number, and type of such sensor systems in autonomous or semi-autonomous vehicles. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to reduce vehicle size, shape, aerodynamics, fuel economy, aesthetics, or cost, or to adapt to special environments or other conditions for application). For example, the sensor systems (e.g., 202 and 204) may be placed in various other locations on the vehicle (e.g., at location 216) to have a field of view corresponding to the interior of the vehicle 200 and / or the surrounding environment.

[0051] The sensor system 202 may be mounted on top of the vehicle 200 and may include one or more sensors configured to detect information about the environment surrounding the vehicle 200 and output an index of that information. For example, the sensor system 202 may include any combination of cameras, radar, lidar, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). The sensor system 202 may include one or more movable mounts that can be operated to adjust the orientation of one or more sensors within the sensor system 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to acquire information from each direction around the vehicle 200. In another embodiment, the movable mount of the sensor system 202 may be movable in a scanning manner within a range of a specific angle and / or azimuth and / or elevation. The sensor system 202 may be mounted on the roof of the vehicle, although other mounting locations are also possible.

[0052] Furthermore, the sensors of sensor system 202 may be distributed in different locations and do not need to be placed side by side in a single location. Additionally, each sensor of sensor system 202 may be configured to move or scan independently of the other sensors of sensor system 202. Additionally or alternatively, multiple sensors may be mounted on one or more of sensor locations 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two LiDAR devices mounted on sensor locations, as well as one LiDAR device and one radar mounted on sensor locations.

[0053] One or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more LiDAR devices. For example, a LiDAR device may include multiple light-emitting devices arranged over a range of angles with respect to a given plane (e.g., the xy-plane). For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot about an axis perpendicular to a given plane (e.g., the z-axis) to illuminate the environment surrounding the vehicle 200 with light pulses. Information about the surrounding environment may be determined based on the detection of various aspects of the reflected light pulses (e.g., elapsed time of flight, polarization, and intensity).

[0054] In exemplary embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide point cloud information that may relate to physical objects in the surrounding environment of vehicle 200. While vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are shown as including certain features, it will be understood that other types of sensor systems are contemplated within the scope of this disclosure. Furthermore, exemplary vehicle 200 may include any of the components described in relation to vehicle 100 in Figure 1.

[0055] In exemplary configurations, one or more radars may be located on the vehicle 200. Similar to radar 126 described above, one or more radars may include antennas configured to transmit and receive radio waves (e.g., electromagnetic waves having frequencies between 30 Hz and 300 GHz). Such radio waves may be used to determine the distance and / or speed of one or more objects in the environment surrounding the vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some embodiments, one or more radars may be located near the rear of the vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of the vehicle 200 for the presence of radio wave reflecting objects. Similarly, one or more radars may be located near the front of the vehicle 200 (e.g., sensor systems 212 and 214) to actively scan the environment near the front of the vehicle 200. The radar may be positioned in a location suitable for illuminating an area including the forward travel path of the vehicle 200 without being obstructed by other features of the vehicle 200. For example, the radar may be embedded in the front bumper, front headlights, cowl, and / or hood, and / or mounted on or near them. Furthermore, one or more additional radars may be positioned in or near the rear bumper, side panels, rocker panels, and / or lower body, for example, by including such devices, to actively scan the sides and / or rear of the vehicle 200 for the presence of radar-reflective objects.

[0056] Vehicle 200 may include one or more cameras. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more cameras. The cameras may be photosensitive devices such as still cameras, video cameras, thermal imaging cameras, stereo cameras, and night vision cameras, configured to capture multiple images of the surrounding environment of vehicle 200. For this purpose, the cameras may be configured to detect visible light and, additionally or alternatively, to detect light from other parts of the spectrum such as infrared or ultraviolet light. The cameras may be two-dimensional detectors and, optionally, may have a sensitivity range in three-dimensional space. In some embodiments, the cameras may include range detectors configured to produce a two-dimensional image showing the distance from the camera to several points in the surrounding environment. For this purpose, the cameras may use one or more range detection techniques. For example, the camera can provide range information by using structured light techniques, in which the vehicle 200 illuminates objects in the surrounding environment with a predetermined light pattern, such as a glider or checkerboard pattern, and uses the camera to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion aberration of the reflected light pattern, the vehicle 200 can determine the distance to a point on the object. The predetermined light pattern may include infrared light or radiation of other wavelengths suitable for such measurements. In some embodiments, the camera may be mounted inside the windshield of the vehicle 200. Specifically, the camera may be positioned to capture an image from a forward view relative to the orientation of the vehicle 200. Other mounting positions and field of view of the camera may be used either inside or outside the vehicle 200. Furthermore, the camera may have associated optical elements that are operable to provide an adjustable field of view. Moreover, the camera may be mounted on the vehicle 200 using a movable mount to change the camera's directional angle via a pan / tilt mechanism, etc.

[0057] Vehicle 200 may also include one or more acoustic sensors used to sense the surrounding environment of Vehicle 200 (for example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). Acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, and micro-electromechanical system (MEMS) microphones) used to sense acoustic waves (i.e., pressure differences) in the fluid (e.g., air) of the environment surrounding Vehicle 200. Such acoustic sensors may be used to identify sounds in the surrounding environment (e.g., sirens, human speech, animal sounds, and alarms) on which the control strategy of Vehicle 200 may be based. For example, if an acoustic sensor detects a siren (e.g., an ambulance siren, and / or a fire engine siren), Vehicle 200 may slow down and / or navigate to the edge of the road.

[0058] Although not shown in Figures 2A-2E, the vehicle 200 may include a wireless communication system (e.g., similar to and / or in addition to the wireless communication system 146 in Figure 1). The wireless communication system may include a radio transmitter and a radio receiver, which may be configured to communicate with devices outside or inside the vehicle 200. Specifically, the wireless communication system may include, for example, a vehicle communication system or a transceiver configured to communicate with other vehicles and / or computing devices at a road station. Examples of such vehicle communication systems include DSRC, radio frequency detection (RFID), and other communication standards proposed for intelligent transport systems.

[0059] Vehicle 200 may include, in addition to or instead of, these indicated components. These additional components may include electrical or mechanical functions.

[0060] The control system of the vehicle 200 may be configured to control the vehicle 200 according to a control strategy selected from a plurality of possible control strategies. The control system may be configured to receive information from sensors coupled to the vehicle 200 (on or outside the vehicle 200), modify the control strategy (and associated driving behavior) based on that information, and control the vehicle 200 according to the modified control strategy. The control system may further be configured to monitor the information received from the sensors and continuously evaluate the driving conditions, and may also be configured to modify the control strategy and driving behavior based on changes in the driving conditions. For example, the route taken by the vehicle from one destination to another may be modified based on the driving conditions. Additionally or alternatively, speed, acceleration, turning angle, inter-vehicle distance (i.e., distance to the vehicle in front of the current vehicle), lane selection, etc., may all be modified in response to changes in driving conditions.

[0061] As described above, in some embodiments, the vehicle 200 may take the form of a van, but alternative forms are also possible and intended herein. Thus, Figures 2F to 2I show embodiments in which the vehicle 250 takes the form of a semi-track. For example, Figure 2F shows a front view of the vehicle 250, and Figure 2G shows an isometric view of the vehicle 250. In embodiments in which the vehicle 250 is a semi-track, the vehicle 250 may include a tractor portion 260 and a trailer portion 270 (shown in Figure 2G). Figures 2H and 2I provide a side view and a top view of the tractor portion 260, respectively. Similar to the vehicle 200 shown above, the vehicle 250 shown in Figures 2F to 2I may also include various sensor systems (for example, similar to sensor systems 202, 206, 208, 210, 212, and 214 shown and described with reference to Figures 2A to 2E). In some embodiments, the vehicle 200 in Figures 2A-2E may include only a single copy of several sensor systems (e.g., sensor system 204), while the vehicle 250 shown in Figures 2F-2I may include multiple copies of its sensor system (e.g., sensor systems 204A and 204B, as shown).

[0062] While the drawings and overall description may refer to a given vehicle configuration (e.g., a semi-truck vehicle 250, or a van vehicle 200), it should be understood that the embodiments described herein are equally applicable in the context of various vehicles (e.g., with modifications adopted to take into account the vehicle's form factor). For example, sensors and / or other components described or shown as part of a van vehicle 200 may also be used in a semi-truck vehicle 250 (e.g., for navigation, and / or obstacle detection and avoidance).

[0063] Figure 2J shows various sensor fields of view (e.g., associated with the vehicle 250 described above). As described above, the vehicle 250 may include multiple sensors / sensor units. The locations of the various sensors may correspond to the sensor locations disclosed in Figures 2F to 2I, for example. However, in some cases, sensors may be located elsewhere. For the sake of simplicity in the drawings, sensor location reference numbers are omitted from Figure 2J. For each sensor unit of the vehicle 250, Figure 2J shows typical fields of view (e.g., fields of view labeled as 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The sensor field of view may include angular regions (e.g., azimuth region and / or elevation region) in which the sensor can detect objects.

[0064] Figure 2K shows beam steering with respect to sensors of a vehicle (e.g., vehicle 250 shown and described with reference to Figures 2F-2J) according to an exemplary embodiment. In various embodiments, the sensor unit of vehicle 250 may be radar, lidar, sonar, etc. Furthermore, in some embodiments, while the sensor is operating, the sensor may scan within the sensor's field of view. Various different scan angles for the exemplary sensor are shown as regions 272, each indicating the angular region in which the sensor is operating. The sensor may periodically or iteratively change the region in which it is operating. In some embodiments, multiple sensors may be used by vehicle 250 to measure the regions 272. In addition, other regions may be included in other embodiments. For example, one or more sensors may measure the aspect of the trailer 270 of vehicle 250 and / or the region in front of vehicle 250.

[0065] At certain angles, the sensor's operating area 275 may include the rear wheels 276A and 276B of the trailer 270. Therefore, the sensor may measure the rear wheels 276A and / or 276B during operation. For example, the rear wheels 276A and 276B may reflect lidar or radar signals transmitted by the sensor. The sensor may receive signals reflected from the rear wheels 276A and 276. Therefore, the data collected by the sensor may include data from reflections from the wheels.

[0066] In some cases, such as when the sensor is radar, reflections from rear wheels 276A and 276B can appear as noise in the received radar signal. As a result, the radar may operate with an enhanced signal-to-noise ratio in cases where rear wheels 276A and 276B direct the radar signal away from the sensor.

[0067] Figure 3 is a conceptual diagram illustrating a representative embodiment of wireless communication between various computing systems related to an autonomous or semi-autonomous vehicle. In particular, wireless communication can be performed between a remote computing system 302 and a vehicle 200 via a network 304. Wireless communication can also be performed between a server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0068] Vehicle 200 can accommodate various types of vehicles capable of transporting passengers or objects between locations, and can take any one or more of the above-described forms. In some cases, vehicle 200 may operate in autonomous or semi-autonomous mode, where a control system uses sensor measurements to enable safe navigation of vehicle 200 between destinations. When operating in autonomous or semi-autonomous mode, vehicle 200 can navigate with or without passengers. As a result, vehicle 200 can pick up and drop off passengers between desired destinations.

[0069] The remote computing system 302 may represent any type of device relating to remote assistance technology, including but not limited to those described herein. In examples, the remote computing system 302 may represent any type of device configured to (i) receive information relating to the vehicle 200, (ii) provide an interface through which a human operator can then become aware of the information and input a response relating to the information, and (iii) transmit the response to the vehicle 200 or to another device. The remote computing system 302 may take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some embodiments, the remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0070] The remote computing system 302 may include one or more subsystems and components that are similar to or identical to the subsystems and components of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface, such as a touchscreen and input / output devices, such as speakers. Other embodiments are equally possible.

[0071] Network 304 represents the infrastructure that enables wireless communication between the remote computing system 302 and the vehicle 200. Network 304 also enables wireless communication between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0072] The location of the remote computing system 302 can vary within the scope of the embodiment. For example, the remote computing system 302 may be located remotely from the vehicle 200, having wireless communication via the network 304. In another embodiment, the remote computing system 302 may correspond to a computing device within the vehicle 200, separate from the vehicle 200, where a human operator interacts with the passengers or driver of the vehicle 200. In some embodiments, the remote computing system 302 may be a computing device with a touchscreen that can be operated by the passengers of the vehicle 200.

[0073] In some embodiments, the operations described herein, which are performed by the remote computing system 302, may be additionally or alternatively performed by the vehicle 200 (i.e., by any system or subsystem of the vehicle 200). In other words, the vehicle 200 may be configured to provide a remote assistance mechanism with which the vehicle's driver or passengers can interact.

[0074] The server computing system 306 may be configured to communicate wirelessly with the remote computing system 302 and the vehicle 200 (or, optionally, directly with the remote computing system 302 and / or the vehicle 200) via the network 304. The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information about the vehicle 200 and its remote assistance. Thus, the server computing system 306 may be configured to perform any operation, or a portion of such operation, described herein, as performed by the remote computing system 302 and / or the vehicle 200. The server computing system 306 may be available in some embodiments of the wireless communication related to remote assistance, but not in other embodiments.

[0075] The server computing system 306 may include a processor configured to perform the various operations described herein, as well as one or more subsystems and components similar to or identical to the subsystems and components of the remote computing system 302 and / or the vehicle 200, such as a wireless communication interface for receiving information from and providing information to the vehicle 200.

[0076] The various systems described above can perform a variety of operations. These operations and their associated characteristics are described below.

[0077] In line with the above considerations, a computing system (e.g., a remote computing system 302, a server computing system 306, or a computing system local to the vehicle 200) may operate to capture images of the surrounding environment of the autonomous or semi-autonomous vehicle using a camera. Generally, at least one computing system can analyze the images and, if possible, control the autonomous or semi-autonomous vehicle.

[0078] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data (also referred to herein as “environmental data”) representing objects in the environment surrounding the vehicle in various ways. The vehicle’s sensor system may provide environmental data representing objects in the surrounding environment. For example, a vehicle may have various sensors such as cameras, radar, lidar, microphones, radio units, and other sensors. Each of these sensors may communicate environmental data about the information it has received to a processor in the vehicle.

[0079] In one embodiment, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have two or more cameras positioned in different orientations. Also in some embodiments, the cameras may be movable to capture images and / or video in different directions. The cameras may be configured to store the captured images and video in memory for subsequent processing by the vehicle's processing system. The captured images and / or video may be environmental data. Furthermore, the cameras may include image sensors, such as those described herein.

[0080] In another embodiment, the radar may be configured to transmit electromagnetic signals reflected by various objects near the vehicle and then capture the electromagnetic signals reflected from the objects. The captured reflected electromagnetic signals may enable the radar (or processing system) to make various decisions about the objects that reflected the electromagnetic signals. For example, the distance and position to various reflective objects may be determined. In some embodiments, the vehicle may have two or more radars in different orientations. The radar may be configured to store the captured information in memory for subsequent processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0081] In another embodiment, the lidar may be configured to transmit electromagnetic signals (e.g., infrared light, such as from a gas, a diode laser, or other possible light source) reflected by target objects near the vehicle. The lidar may be capable of capturing the reflected electromagnetic (e.g., infrared) signals. The captured reflected electromagnetic signals may enable a ranging system (or processing system) to determine the distance to various objects. The lidar may also enable the determination of the velocity or speed of target objects and storage of this as environmental data.

[0082] Furthermore, in one embodiment, the microphone may be configured to capture sounds from the vehicle's surrounding environment. The sounds captured by the microphone may include the sirens of emergency vehicles and the sounds of other vehicles. For example, the microphone may capture the sounds of sirens from ambulances, fire engines, or police vehicles. The processing system may be able to identify that the captured audio signal indicates an emergency vehicle. In another embodiment, the microphone may capture the sounds of exhaust from another vehicle, such as the exhaust from a motorcycle. The processing system may be able to identify that the captured audio signal indicates a motorcycle. The data captured by the microphone may form part of the environmental data.

[0083] In yet another embodiment, the radio unit may be configured to transmit an electromagnetic signal, which may take the form of a Bluetooth signal, an 802.11 signal, and / or other radio technology signal. The first electromagnetic radiation signal may be transmitted via one or more antennas located on the radio unit. Furthermore, the first electromagnetic radiation signal may be transmitted in one of many different radio signal modes. However, in some embodiments, it is desirable to transmit the first electromagnetic radiation signal in a signal mode that requests a response from a device located near an autonomous or semi-autonomous vehicle. The processing system may detect nearby devices based on the response returned to the radio unit and use this communicated information as part of the environmental data.

[0084] In some embodiments, the processing system may combine information from various sensors to further determine the vehicle's surrounding environment. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, other combinations of sensor data may be used by the processing system to determine the surrounding environment.

[0085] While operating in autonomous (or semi-autonomous) mode, a vehicle can control its movement with little to no human input. For example, if a human operator enters an address into the vehicle, the vehicle may then be able to drive to the designated destination without further human input (e.g., without the human needing to operate or touch the brake / accelerator pedals). Furthermore, while the vehicle is operating autonomously or semi-autonomously, sensor systems may receive environmental data. The vehicle's processing system may modify the vehicle's control based on the environmental data received from various sensors. In some embodiments, the vehicle may change its speed in response to environmental data from various sensors. The vehicle may change its speed to avoid obstacles, comply with traffic laws, etc. If the vehicle's processing system identifies an object near the vehicle, the vehicle may change its speed or otherwise alter its movement.

[0086] If a vehicle detects an object but lacks sufficient confidence in its detection, it may request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object is actually present in the surrounding environment (e.g., whether there is actually a stop signal or not), (ii) verifying whether the vehicle's object detection is correct, (iii) correcting the detection if it was incorrect, and / or (iv) providing supplementary instructions for the autonomous or semi-autonomous vehicle (or modifying the current instructions). Remote assistance tasks may also include providing instructions for a human operator to control the vehicle's actions (e.g., if the human operator determines that the object is a stop signal, instructing the vehicle to stop at the stop signal), although in some scenarios the vehicle itself may control its actions based on human operator feedback related to object detection.

[0087] To facilitate this, the vehicle may analyze environmental data representing objects in the surrounding environment to determine at least one object with a detection confidence level below a threshold. The vehicle's processor may be configured to detect various objects in the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, cyclists, street signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0088] Detection confidence can indicate the likelihood that a determined object is correctly identified or present in its surrounding environment. For example, a processor may perform object detection on the image data in the received environmental data and determine that an object has a detection confidence below a threshold if it cannot identify at least one object with a detection confidence above a threshold. If the result of object detection or recognition is inconclusive, the detection confidence may be low or below a set threshold.

[0089] Depending on the source of the environmental data, a vehicle may detect objects in its surroundings in various ways. In some embodiments, the environmental data may be image or video data from a camera. In other embodiments, the environmental data may come from LiDAR. The vehicle may analyze the captured image or video data to identify objects within the image or video data. The method and apparatus may be configured to monitor the image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in its surroundings based on radar, audio, or other data.

[0090] In some embodiments, the technique a vehicle uses to detect objects may be based on a set of known data. For example, data related to environmental objects may be stored in memory located in the vehicle. The vehicle may compare the received data with the stored data to make a decision about an object. In other embodiments, the vehicle may be configured to make a decision about an object based on the context of the data. For example, road signs related to construction may generally be orange in color. Therefore, the vehicle may be configured to detect orange objects located near the side of the road as road signs related to construction. Furthermore, when the vehicle's processing system detects an object in the captured data, it may also calculate the confidence level of each object.

[0091] Furthermore, a vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object being detected. For example, for objects that may require a quick response from the vehicle, such as the brake lights of another vehicle, the confidence threshold may be low. However, in other embodiments, the confidence threshold may be the same for all detected objects. If the confidence associated with a detected object is higher than the confidence threshold, the vehicle may assume that the object has been correctly recognized and, based on that assumption, responsively adjust the vehicle's controls.

[0092] If the confidence level associated with a detected object is lower than the confidence threshold, the action taken by the vehicle may change. In some embodiments, the vehicle may react as if the detected object exists, despite the low confidence level. In other embodiments, the vehicle may react as if the detected object does not exist.

[0093] When a vehicle detects an object in its surrounding environment, it can also calculate a confidence level to associate it with a specific detected object. The confidence level can be calculated in various ways depending on the embodiment. In one embodiment, when an object is detected in the surrounding environment, the vehicle may compare environmental data with predetermined data associated with known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle may use a mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0094] In response to a determination that an object has a detection confidence level below a threshold, the vehicle may send a request for remote assistance to a remote computing system along with the detection of the object. As described above, the remote computing system can take various forms. For example, the remote computing system may be a computing device located within the vehicle, separate from the vehicle itself, but with a touchscreen interface for displaying remote assistance information, thereby allowing a human operator to interact with the vehicle's passengers or driver. Additionally or alternatively, as another example, the remote computing system may be a remote computer terminal or other device located not near the vehicle.

[0095] Remote assistance requests may include environmental data, such as image data and audio data, including objects. The vehicle may transmit the environmental data to a remote computing system via a network (e.g., network 304) and, in some embodiments, a server (e.g., server computing system 306) over a network. A human operator in the remote computing system may then use the environmental data as a basis for responding to the request.

[0096] In some embodiments, if an object is detected as having a confidence level below a confidence threshold, the object may be given a preliminary detection, and the vehicle may be configured to adjust its operation in response to the preliminary detection. Such adjustments to operation may take the form of stopping the vehicle, switching the vehicle to a human-controlled mode, or changing the vehicle's speed (e.g., speed and / or direction), among other possible adjustments.

[0097] In other embodiments, even if the vehicle detects an object with a confidence level that meets or exceeds a threshold, the vehicle may act according to the detected object (for example, stop if the object is identified with high confidence as a stop signal), but the vehicle may be configured to request remote assistance at the same time as (or after) acting according to the detected object.

[0098] Figure 4A is a block diagram of a system according to an exemplary embodiment. In particular, Figure 4A shows a system 400 including a system controller 402, a lidar device 410, a plurality of sensors 412, and a plurality of controllable components 414. The system controller 402 includes a processor 404, memory 406, and instructions 408 stored on memory 406 and executable by the processor 404 to perform functions.

[0099] The processor 404 may include one or more processors, such as one or more general-purpose microprocessors (e.g., having single-core or multi-core) and / or one or more dedicated microprocessors. One or more processors may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also construed herein.

[0100] Memory 406 may include, but is not limited to, computer-readable media such as non-temporary computer-readable media, including read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), solid-state drives (SSDs), hard disk drives (HDDs), compact discs (CDs), digital video discs (DVDs), digital tapes, read / write (R / W) CDs, R / W DVDs, etc.

[0101] The LiDAR device 410, further described below, includes a plurality of light-emitting elements configured to emit light (e.g., in light pulses), and one or more photodetectors configured to detect light (e.g., the reflected portion of the light pulses). The LiDAR device 410 may generate three-dimensional (3D) point cloud data from the output of the photodetectors and provide the 3D point cloud data to the system controller 402. The system controller 402 may then perform operations on the 3D point cloud data to determine the characteristics of the surrounding environment (e.g., relative positions of objects in the surrounding environment, edge detection, object detection, and / or proximity sensing).

[0102] Similarly, the system controller 402 may use the outputs from multiple sensors 412 to determine the characteristics of the system 400 and / or the surrounding environment. For example, the sensors 412 may include one or more of the following: GPS, IMU, image capture device (e.g., camera), light sensor, thermal sensor, and other sensors that indicate parameters related to the system 400 and / or the surrounding environment. The lidar device 410 is depicted separately from the sensors 412 for illustrative purposes and may be considered as part of or as part of the sensors 412 in some embodiments.

[0103] Based on the characteristics of the surrounding environment determined by the system controller 402 based on the outputs from system 400 and / or the lidar device 410 and sensor 412, the system controller 402 may control controllable components 414 to perform one or more actions. For example, system 400 may correspond to a vehicle, in which case the controllable components 414 may include the vehicle's braking system, turning system, and / or acceleration system, and the system controller 402 may change the configuration of these controllable components based on the characteristics determined from the lidar device 410 and / or sensor 412 (for example, when the system controller 402 controls the vehicle in autonomous or semi-autonomous mode). In this example, the lidar device 410 and sensor 412 are also controllable by the system controller 402.

[0104] Figure 4B is a block diagram of a lidar device according to an exemplary embodiment. In particular, Figure 4B shows a lidar device 410 having a controller 416 configured to control a plurality of light-emitting elements 424 and one or more photodetectors, for example, a plurality of photodetectors 426. The lidar device 410 may further include a firing circuit 428 configured to select and supply power to each of the plurality of light-emitting elements 424, and a selector circuit 430 configured to select each of the plurality of photodetectors 426. The controller 416 includes a processor 418, a memory 420, and instructions 422 stored on the memory 420.

[0105] Similar to processor 404, processor 418 may include one or more processors, such as one or more general-purpose microprocessors and / or one or more dedicated microprocessors. One or more processors may include, for example, one or more CPUs, one or more microcontrollers, one or more GPUs, one or more TPUs, one or more ASICs, and / or one or more FPGAs. Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein.

[0106] Similar to memory 406, memory 420 may include, but is not limited to, computer-readable media such as ROM, PROM, EPROM, EEPROM, non-volatile random access memory (e.g., flash memory), SSD, HDD, CD, DVD, digital tape, R / W CD, R / W DVD, and other non-temporary computer-readable media.

[0107] Instruction 422 is stored in memory 420 and is executable by processor 418, and performs functions related to controlling firing circuit 428 and selector circuit 430 for generating 3D point cloud data and for processing the 3D point cloud data (or possibly to facilitate processing of the 3D point cloud data by another computing device, such as system controller 402).

[0108] The controller 416 can determine 3D point cloud data by using the light-emitting elements 424 to emit pulses of light. The emission time is established for each light-emitting element, and the relative position during the emission time is also tracked. Various aspects of the environment surrounding the LiDAR device 410, such as different objects, reflect the pulses of light. For example, if the LiDAR device 410 is in an environment that includes a road, such objects may include vehicles, signs, pedestrians, road surfaces, construction cones, etc. Some objects may be more reflective than others, such that the intensity of the reflected light may indicate the type of object reflecting the pulses of light. Furthermore, the surfaces of objects may be in different positions relative to the LiDAR device 410, and therefore it may take some time for them to reflect a portion of the pulses of light back to the LiDAR device 410. Thus, the controller 416 can track the detection time at which the reflected light pulses are detected by the photodetector, and the relative position of the photodetector during the detection time. By measuring the time difference between the emission time and the detection time, the controller 416 can determine how far the light pulse travels before being received, and therefore the relative distance to the corresponding object. By tracking the relative positions at the emission and detection times, the controller 416 can determine the orientation of the light pulse and the reflected light pulse relative to the LiDAR device 410, and therefore the relative orientation of the object. By tracking the intensity of the received light pulse, the controller 416 can determine how reflective the object is. The 3D point cloud data determined based on this information can therefore show the relative position of the detected reflected light pulse (e.g., in a coordinate system such as a Cartesian coordinate system) and the intensity of each reflected light pulse.

[0109] The firing circuit 428 is used to select a light-emitting element to emit a light pulse. Similarly, the selector circuit 430 is used to sample the output from the photodetector.

[0110] Figure 5A illustrates device 500. In some embodiments, device 500 may correspond to a portion of sensor system 102 described in relation to Figure 1, sensor systems 202, 204, 206, 208, 210, 212, and 214 described in relation to Figures 2A-2E, and / or system 400 described in relation to Figure 4A. Such devices may also be positioned on the vehicle 200 in the same location and / or orientation as sensor systems 202, 204, 206, 208, 210, 212, and 214. Furthermore, in some embodiments, multiple instances of device 500 may be mounted on the same vehicle.

[0111] In some embodiments, device 500 may include an image sensor 502, an analog-to-digital converter (ADC) 504, and / or an application-specific integrated circuit (ASIC) 506. As described above, the image sensor 502 may be configured to capture image data relating to the surrounding environment of device 500. The ADC 504 may be configured to receive the image data captured from the image sensor 502 and / or to provide the transformed image data. The ASIC 506 may be configured to receive the transformed image data and / or to apply a trained machine learning model to the transformed image data. The machine learning model may be configured to identify one or more objects in the surrounding environment within the transformed image data.

[0112] In some embodiments, the trained machine learning model may be selected from a plurality of machine learning models based on the orientation and / or position of the device 500 relative to the vehicle 200. For example, a first trained machine learning model may be trained to recognize vehicles, pedestrians, traffic signals, or signs, while a second trained machine learning model may be trained to recognize passing vehicles, passing cyclists, or pedestrians. Thus, the appropriate machine learning model may be selected for the task at hand based on the orientation and / or position of the device 500 relative to the vehicle 200.

[0113] Figure 5B shows one embodiment of device 500 shown in Figure 5A. As shown, in some embodiments, the device (shown as device 550 in Figure 5B) may include a multilayer die stack, with certain components deposited on each layer. Each layer is physically bonded to the layer immediately above and immediately below it. For example, each layer may be built on a substrate made of a semiconductor material such as silicon. These layers can then be interconnected using wire bonding, controlled foldable chip connection (C4) methods (also known as “flip-chip”), and / or through-silicon vias (TSVs). The layers can be bonded using die attach films (DAFs).

[0114] In some embodiments, the device 550 may include a multilayer die stack having at least three layers, namely a first layer 552, a second layer 554, and a third layer 556. The second layer 554 may be located above the first layer 552, and the third layer 556 may be located above the second layer 554. Each layer may have one or more components of the sensor device 500 placed on it. For example, an application-specific integrated circuit (ASIC) 506 may be located on the first layer 552, an analog-to-digital converter (ADC) 504 may be located on the second layer 554, and an image sensor 502 may be located on the third layer 556.

[0115] As described above in relation to Figures 2A-2E, the sensor system may be located at different positions and orientations on the vehicle 200. This also applies to the device 500. For example, the vehicle may have several devices 500 located at different positions and / or orientations around its exterior.

[0116] An example of this is shown in Figure 5C, which depicts a vehicle 570, having two devices 500 positioned around its exterior, a front device 572, and a side device 574. The dashed lines represent the direction of travel of the vehicle 570. As described above, a vehicle may have several devices 500 positioned at different locations and / or orientations around its exterior. In some situations, the orientation of the devices 500 around the vehicle 570 may be relative to the direction of travel of the vehicle 570.

[0117] For example, the first orientation of a device may be within 15° parallel to the direction of travel of the vehicle (e.g., forward-facing device 572), and the second orientation of another device may be within 15° perpendicular to the direction of travel of the vehicle (e.g., side-facing device 574). Based on this difference in position and / or orientation, different sensor devices may generate different image frames, as will be discussed later with respect to Figures 7A-7D.

[0118] Figure 6A shows a process 600 that may be performed by device 602, which may correspond to device 500 or device 550 described above. Similar to devices 500 and 550, device 602 may include an image sensor 604, which may be configured to capture image data 606 about the environment surrounding device 602. This image data 606 may then be supplied to an analog-to-digital converter (ADC) 608. The ADC may then convert the image data 606 from an analog signal to digital information, generating converted image data 610. The converted image data 610 may then be supplied to an application-specific integrated circuit (ASIC) 612. The ASIC 612 may be configured to perform processing and / or computer vision tasks on the converted image data 610. For example, the ASIC 612 may be configured to apply a trained machine learning model 614 to the converted image data 610 to identify one or more objects in the surrounding environment within the converted image data 610. In some embodiments, the trained machine learning model may include a convolutional neural network (CNN).

[0119] In the context of device 602 operating in relation to vehicle 200, a trained machine learning model 614 may be configured to identify objects related to the navigation of vehicle 200. In some embodiments, ASIC 612 may receive lidar data from a lidar device (e.g., lidar device 410, as described with respect to Figures 4A and 4B) indicating the distance between vehicle 200 and one or more objects in the surrounding environment, and / or the distance to one or more objects in the surrounding environment. In some embodiments, ASIC 612 may receive lidar data from a source other than a lidar device. For example, ASIC 612 may receive lidar data from a central processing unit or system, or the lidar data may be stored in memory within device 602. Thus, lidar data collected by other lidar devices (e.g., on other vehicles other than vehicle 200) may also be processed by device 602. ASIC 612 may be configured to apply a trained machine learning model 614 to a subset of transformed image data 610 based on the lidar data. For example, the lidar data may indicate that an object is likely to be located within a particular region of the transformed image data 610, and a trained machine learning model 614 may be applied to this particular region based on such indications.

[0120] After processing, the ASIC 612 may output an image frame 616. The image frame 616 may include the converted image data 610, as well as additional metadata, as will be described later in relation to Figure 6B.

[0121] Figure 6B shows a block diagram of an image frame 650, such as an image frame 616 generated by the process 600 shown in Figure 6A. The output image frame 650 may include converted image data 652, which may correspond to converted image data 610 generated by the ADC 608. In some embodiments, the image frame 650 may be provided to a central computing device (e.g., a system controller 402, as described above in relation to Figure 4A). The image frame 650 shown in Figure 6B includes both converted image data 652 and metadata 654, but in some embodiments, the image frame 650 may include only one or the other (e.g., only converted image data 652).

[0122] The image frame 650 may also include metadata 654, which may include additional information about the image frame 650, the converted image data 652, and / or the device 602 that generated it. For example, the metadata 654 may include object classification data 656 and / or object location data 658 related to an object identified in the converted image data 652. Such an object may be identified and / or located through the application of a trained machine learning model 614. For example, the object classification data 656 may include a determination of whether an object present in the converted image data 652 is a vehicle, a person, a traffic signal, or a traffic sign. In some embodiments, the object classification data 656 may also include information about the orientation of such an object relative to a vehicle (e.g., passing, forward, backward, etc.).

[0123] The object location data 658 may include a representation of the location of an identified object in the transformed image data 652. For example, the object location data 658 may include a region of the transformed image data 652. As another example, the object location data 658 may include coordinates that specify the location of an object in the transformed image data 652. In the embodiments described above, when the trained machine learning model 614 is applied based on the lidar data, the object location data 658 may include spatial coordinates (e.g., x, y, and / or z coordinates) in the lidar data and / or the transformed image data.

[0124] Metadata 654 may be contained within the image frame 650 as one or more rows and / or columns of the image frame 650, for example, as binary or text data. Examples are provided below with reference to Figures 7A to 7D.

[0125] Figure 7A shows image data 700 captured by an image sensor of a front-mounted device (for example, located at or near the location of the sensor system 206 shown in Figures 2A-2E and / or the device 572 shown in Figure 5C). During the operation of the vehicle 200, such a front-mounted device may capture objects such as vehicles, pedestrians, traffic signals, or signs. In Figure 7A, the image data 700 captured a vehicle 702 and a stop signal 704. Such image data may then be processed by other components of the device described herein, for example, an ADC and / or ASIC in process 600 shown in Figure 6A.

[0126] Figure 7B represents the same image data 700 after being processed by the components of the device described herein, for example, through the process 600 shown in Figure 6. In particular, Figure 7B shows an image frame 720 containing both the converted image data 722 (converted from image data 700) and metadata 724. Such metadata 724 may be attached as an additional row (or more rows) of the image frame 720. In the particular example of Figure 7B, the metadata 724 includes object classification data (object_type) and object location data (object_location), the latter in the form of three-dimensional coordinates. In some embodiments, the object location data may refer to LiDAR data, as described above. The metadata 724 is attached to the top row of the image frame 720 in Figure 7B, but this is merely an example. In some embodiments, for example, the metadata 724 may be attached to the bottom row of the image frame 720 and / or one or more columns of the image frame 720.

[0127] Figure 7C shows another example of image data captured by a device, positioned and / or oriented differently from that in Figure 7A. Specifically, it shows image data 740 captured by an image sensor of a side-mounted device (e.g., located at or near the location of the sensor system 216 shown in Figures 2A-2E and / or the device 574 shown in Figure 5C). In the course of the vehicle 200's operation, such a side-mounted device may capture objects such as passing vehicles, passing cyclists, or pedestrians. In Figure 7C, image data 740 captured a passing cyclist 742 and a passing vehicle 744. Such image data may then be processed by other components of the device described herein, e.g., an ADC and / or ASIC in process 600 shown in Figure 6A.

[0128] Figure 7D represents the same image data 740 after being processed by the components of the device described herein, for example, through the process 600 shown in Figure 6. In particular, Figure 7D shows an image frame 760 containing both the transformed image data 762 (transformed from image data 740) and metadata 764. Such metadata 764 may be attached as an additional row (or more rows) of the image frame 760. In the particular example of Figure 7D, the metadata 726 contains object classification data (object_type) and object location data (object_location) of the passing cyclist 742, the latter in the form of two-dimensional coordinates (e.g., x and y coordinates in the transformed image data 762).

[0129] As described above, image frames such as image frame 720 and image frame 760 may be provided to a central computing device (for example, the system controller 402 mentioned above in relation to Figure 4A).

[0130] As shown in Figures 7A–7D, different devices may generate different image frames. For specialization and greater efficiency, different devices may be provided with trained machine learning models that are specifically designed and / or trained for the types of image frames in which they are likely to collaborate. For example, the device that generated image frame 720 in Figure 7B may use a machine learning model trained, among other things, to recognize stop signals. However, in some embodiments, the trained machine learning model for each of the different devices may be the same model. For example, one model may be trained to recognize multiple different types of objects (for example, based on the collection of training data used to train the model).

[0131] As another example, a trained machine learning model may be trained according to a supervised learning process using image data collected from other vehicles using a camera or other sensor having the same or similar orientation as the device on which the trained machine learning model may be deployed. For example, a front device (e.g., a front device 572 as shown in Figure 5C) may be trained on image data collected from other vehicles using a camera also mounted on the front of such a vehicle. In this way, the trained machine learning model may improve the accuracy of object recognition performed using the model.

[0132] In other words, the trained machine learning model may be trained in a supervised learning process using image data collected by a camera coupled to a first auxiliary vehicle, operating in an autonomous or semi-autonomous mode along the auxiliary orientation with respect to the direction of travel of the auxiliary vehicle, such that the orientation with respect to the direction of travel of the vehicle and the auxiliary orientation with respect to the direction of travel of the auxiliary vehicle are substantially the same.

[0133] As described above, the trained machine learning model may be selected from a plurality of machine learning models. In some embodiments, the device (e.g., device 500) may include memory, and the trained machine learning model may be stored in the first memory during vehicle manufacturing. In some embodiments, the trained machine learning model may be selected from a plurality of machine learning models based on the location and / or orientation of the device.

[0134] In other words, the device may include memory, and the ASIC may be configured to receive trained machine learning models from remote computing devices (for example, remote computing system 302 and / or server computing system 306, as described with respect to Figure 3) and store the trained machine learning models in memory.

[0135] Figure 8 is a flowchart of Method 800 according to an exemplary embodiment. Method 800 may be carried out by a system (e.g., System 400 shown and described with reference to Figure 4A), a system controller (e.g., System Controller 402 shown and described with reference to Figure 4A), and / or a device (e.g., Device 500 shown in Figure 5A).

[0136] In block 802, method 800 may include capturing image data relating to the surrounding environment using an image sensor.

[0137] In block 804, method 800 may include receiving image data captured from an image sensor by an analog-to-digital converter (ADC).

[0138] In block 806, method 800 may include providing the converted image data to an application-specific integrated circuit (ASIC) via an ADC.

[0139] In block 808, method 800 may include applying a trained machine learning model to the transformed image data using an ASIC to identify one or more objects in the surrounding environment within the transformed image data.

[0140] In block 810, method 800 may include outputting an image frame by an ASIC, wherein at least one row of the image frame includes metadata, which includes object classification data and object location data for one or more identified objects in the surrounding environment.

[0141] In some embodiments, the method 800 may further include: receiving an updated trained machine learning model from a remote computing device by an ASIC; storing the updated trained machine learning model in memory by an ASIC; capturing additional image data of the surrounding environment by an image sensor; receiving the additional captured image data from the image sensor by an ADC; providing the additional transformed image data to the ASIC by an ADC; applying the updated trained machine learning model to the additional transformed image data by an ASIC to identify one or more additional objects in the surrounding environment within the additional transformed image data; and outputting additional image frames by an ASIC, wherein at least one row of the additional image frames includes metadata, including object classification data and object location data for one or more additional identified objects in the surrounding environment.

[0142] In some embodiments, method 800 may further include providing image frames to a central computing device by an ASIC.

[0143] Figure 9 is a flowchart of Method 900 according to an exemplary embodiment. Method 900 may be carried out by a system (e.g., System 400 shown and described with reference to Figure 4A), a system controller (e.g., System Controller 402 shown and described with reference to Figure 4A), and / or a device (e.g., Device 500 shown in Figure 5A).

[0144] In block 902, method 900 may include capturing first image data relating to the surrounding environment by a first image sensor of a first device, the first device being mounted on the vehicle in a first orientation relative to the vehicle.

[0145] In block 904, method 900 may include receiving first image data captured from a first image sensor by a first ADC of a first device.

[0146] In block 906, method 900 may involve providing a first converted image data by a first ADC.

[0147] In block 908, method 900 may involve receiving converted first image data from a first ADC by a first ASIC of a first device.

[0148] In block 910, method 900 may also involve applying a first trained machine learning model to the transformed first image data using a first ASIC to identify one or more first objects in the surrounding environment within the transformed first image data, the first trained machine learning model being selected from a plurality of machine learning models based on a first orientation.

[0149] In block 912, method 900 may also include capturing second image data of the surrounding environment by a second image sensor of a second device, the second device being mounted on the vehicle along a second orientation relative to the vehicle, the first orientation and the second orientation being different.

[0150] In block 914, method 900 may include receiving a second image data captured from a second image sensor by a second ADC of a second device.

[0151] In block 916, method 900 may involve providing a second converted image data by a second ADC.

[0152] In block 918, method 900 may involve receiving the converted second image data from the second ADC by a second ASIC of the second device.

[0153] In block 920, method 900 may also involve applying a second trained machine learning model to the transformed second image data using a second ASIC to identify one or more second objects in the surrounding environment within the transformed second image data, the second trained machine learning model being selected from a plurality of machine learning models based on the second orientation.

[0154] This disclosure is not limited to the specific embodiments described herein, and the specific embodiments are intended to be illustrative of various aspects. As will be apparent to those skilled in the art, many changes and modifications can be made without departing from the spirit and scope of this disclosure. In addition to the methods and devices enumerated herein, functionally equivalent methods and devices within the scope of this disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and modifications are intended to fall within the scope of the appended claims.

[0155] The above detailed description, with reference to the accompanying drawings, describes various features and functions of the disclosed systems, devices, and methods. In the drawings, unless the context otherwise indicates, similar symbols typically represent similar components identically. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized and other modifications may be made without departing from the scope of the subject matter presented herein. It will be readily apparent that the aspects of the disclosure described herein and illustrated in the drawings can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly assumed.

[0156] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and those considered herein, each step, block, action, and / or communication may represent the processing of information and / or the transmission of information according to the exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, actions described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed in an order different from those shown or discussed, such as substantially simultaneously or in reverse order, depending on the relevant functions. Furthermore, more or fewer blocks and / or actions may be used in any of the message flow diagrams, scenarios, and flowcharts considered herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other in part or as a whole.

[0157] Steps, blocks, or operations corresponding to the processing of information may correspond to a network of circuits configured to perform a specific logical function of the method or technique described herein. Alternatively or additionally, steps or blocks corresponding to the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions that are executable by a processor to perform a specific logical operation or action in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including RAM, a disk drive, a solid-state drive, or another storage medium.

[0158] Furthermore, one or more steps, blocks, or operations corresponding to information transmission may correspond to information transmission between software modules and / or hardware modules on the same physical device. However, other information transmissions may be between software modules and / or hardware modules on different physical devices.

[0159] The specific arrangement shown in the figure should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in the given figure. Also, some of the illustrated elements may be combined or omitted. Furthermore, exemplary embodiments may include elements not illustrated in the figure.

[0160] Various aspects and embodiments are disclosed herein, but other aspects and embodiments will be obvious to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes only and are not intended to limit, and the true scope is indicated by the following claims.

Claims

1. It is a system, Vehicles and A first device attached to the vehicle and oriented in a first orientation relative to the vehicle, wherein the first device is A first image sensor configured to capture first image data about the surrounding environment, A first analog-to-digital converter (ADC) is configured to receive the first image data captured from the first image sensor and to provide the converted first image data. The first application-specific integrated circuit (ASIC), Receiving the first converted image data from the first ADC, Applying a first trained machine learning model to the transformed first image data to identify one or more first objects in the surrounding environment within the transformed first image data, wherein the first trained machine learning model is selected from a plurality of machine learning models based on the first orientation. A first application-specific integrated circuit (ASIC) configured to perform the following: A first device equipped with, A second device attached to the vehicle and aligned with a second orientation relative to the vehicle, wherein the first orientation and the second orientation are different, and the second device is A second image sensor configured to capture second image data relating to the surrounding environment, A second ADC is configured to receive the second image data captured from the second image sensor and provide the converted second image data. It is the second ASIC, Receiving the converted second image data from the second ADC, Applying a second trained machine learning model to the transformed second image data to identify one or more second objects in the surrounding environment within the transformed second image data, wherein the second trained machine learning model is selected from a plurality of machine learning models based on the second orientation. A second ASIC configured to perform the following, A second device equipped with, A system equipped with these features.

2. The system according to claim 1, wherein the first orientation is within 15° parallel to the direction of travel of the vehicle, and the second orientation is within 15° perpendicular to the direction of travel of the vehicle.

3. The system according to claim 2, wherein the first trained machine learning model is trained in a supervised learning process using image data collected by a camera coupled to the first auxiliary vehicle, which operates in autonomous or semi-autonomous mode along a first auxiliary orientation with respect to the direction of travel of the first auxiliary vehicle, and the first orientation of the vehicle with respect to the direction of travel and the first auxiliary orientation of the first auxiliary vehicle with respect to the direction of travel are substantially the same.

4. The system according to claim 2, wherein the first trained machine learning model is trained to recognize vehicles, pedestrians, traffic signals, or signs.

5. The system according to claim 2, wherein the second trained machine learning model is trained to recognize passing vehicles, passing cyclists, or pedestrians.

6. The system according to claim 1, wherein the first device further comprises a first memory, and the first trained machine learning model is stored in the first memory during the manufacture of the vehicle.

7. The system according to claim 1, wherein the first device further comprises a first memory, the second device further comprises a second memory, the first ASIC is further configured to receive the first trained machine learning model from a remote computing device and store the first trained machine learning model in the first memory, and the second ASIC is further configured to receive the second trained machine learning model from the remote computing device and store the second trained machine learning model in the second memory.

8. The system according to claim 1, wherein the first and second trained machine learning models are different from each other.

9. The system according to claim 1, further comprising a lidar device configured to generate lidar data indicating the distance between the vehicle and one or more objects in the surrounding environment, wherein the first ASIC is configured to apply the first trained machine learning model based on the lidar data to a subset of the transformed first image data.

10. The first device is A first layer of a multilayer die stack, wherein the ASIC is arranged on the first layer, The second layer of the multilayer die stack is positioned above the first layer and coupled to the first layer, and the ADC is placed on the second layer. The system according to claim 1, further comprising: a third layer of the multilayer die stack, the third layer being positioned above the second layer and coupled to the second layer, and the first image sensor being positioned on the third layer.

11. It is a device, An image sensor configured to capture image data related to the surrounding environment, An analog-to-digital converter (ADC) is configured to receive the captured image data from the image sensor and provide the converted image data. Application-specific integrated circuits (ASICs), The ADC receives the converted image data, Applying a trained machine learning model to the transformed image data to identify one or more objects in the surrounding environment within the transformed image data, Outputting an image frame, wherein at least one row of the image frame includes metadata, including object classification data and object position data for one or more identified objects in the surrounding environment. An application-specific integrated circuit (ASIC) configured to perform the following: A device that includes this.

12. The device according to claim 11, wherein the trained machine learning model includes a convolutional neural network (CNN).

13. The device according to claim 11, wherein the ASIC is configured to apply the trained machine learning model to a subset of the transformed image data based on lidar data indicating the distance to one or more objects in the surrounding environment.

14. The device according to claim 13, wherein the object position data includes spatial coordinates within the lidar data.

15. The device according to claim 11, wherein the object position data includes spatial coordinates in the converted image data.

16. The device according to claim 11, wherein the object classification data includes determining whether the object is a vehicle, a person, a traffic signal, or a traffic sign.

17. A first layer of a multilayer die stack, wherein the ASIC is arranged on the first layer, The second layer of the multilayer die stack is positioned above the first layer and coupled to the first layer, and the ADC is placed on the second layer. The device according to claim 11, further comprising: a third layer of the multilayer die stack, the third layer being positioned above the second layer and coupled to the second layer, and the image sensor being disposed on the third layer.

18. It is a method, Image sensors capture image data about the surrounding environment, The analog-to-digital converter (ADC) receives the captured image data from the image sensor, The ADC provides the converted image data to an application-specific integrated circuit (ASIC), The ASIC applies the trained machine learning model to the transformed image data to identify one or more objects within the transformed image data and the surrounding environment. The ASIC outputs an image frame, wherein at least one row of the image frame includes metadata, including object classification data and object position data of one or more identified objects in the surrounding environment. Methods that include...

19. The aforementioned ASIC receives updated, trained machine learning models from a remote computing device. The ASIC stores the updated, trained machine learning model in memory. The image sensor captures additional image data relating to the surrounding environment, The ADC receives the additional captured image data from the image sensor, The ADC provides the additional converted image data to the ASIC, The ASIC applies the updated, trained machine learning model to the additional transformed image data to identify one or more additional objects in the surrounding environment within the additional transformed image data. The ASIC outputs an additional image frame, wherein at least one row of the additional image frame includes metadata, including object classification data and object position data, of the one or more additional identified objects in the surrounding environment. The method according to claim 18, further comprising:

20. The method according to claim 18, further comprising providing the image frame to a central computing device using the ASIC.

Images