Thermal imaging for self-driving cars

Abstract

To utilize machine learning techniques to improve object classification in thermal imaging systems.SOLUTION: In an example embodiment, a method is provided. The method includes receiving, at a computing device, one or more infrared images of an environment. The method additionally includes, applying, using the computing device, a trained machine learning system to the one or more infrared images to determine an identified object type in the environment, where the determination is performed by at least: determining one or more prior thermal maps associated with the environment; using the one or more prior thermal maps and the one or more infrared images, determining a current thermal map associated with the environment; and determining the identified object type based on the current thermal map. The method also includes providing the identified object type using the computing device.SELECTED DRAWING: Figure 7

Description

[Background technology]

[0001] Thermal (or infrared) imaging devices can be used to classify objects in a given environment. For example, a thermal imaging device can obtain radiometric information about a given object, and if the emissivity value of the given object can be estimated, this information can be utilized to determine the temperature of the given object. When multiple objects with estimated emissivity values ​​are involved, the thermal imaging device can determine the temperature variation between the objects, and then make predictions about the composition or type of object. Summary of the Invention

[0002] The present disclosure advantageously utilizes machine learning techniques to improve object classification in thermal imaging systems.

[0003] In a first aspect, a method is provided. The method includes receiving, at a computing device, one or more infrared images of an environment. The method further includes applying, using the computing device, a trained machine learning system to the one or more infrared images to determine identified object types in the environment. Applying the trained machine learning system includes determining one or more previous thermal maps associated with the environment. Applying the trained machine learning system also includes determining a current thermal map associated with the environment using the one or more previous thermal maps and the one or more infrared images, and determining the identified object types based on the current thermal maps. The method also includes providing, using the computing device, the identified object types.

[0004] In a second aspect, a method is provided. The method includes receiving, at a computing device, one or more infrared images of an environment. The method further includes using the computing device to train a machine learning system on the one or more infrared images to determine identified object types in the environment. Training the machine learning system includes training the machine learning system to determine one or more previous heat maps associated with the environment. Training the machine learning system also includes training the machine learning system to determine a current heat map associated with the environment using the one or more previous heat maps and the one or more infrared images, and training the machine learning system to determine identified object types using the current heat maps. The method also includes using the computing device to provide the trained machine learning system.

[0005] In a third aspect, a computing device is provided. The computing device includes one or more processors and a data storage area. The data storage area has stored therein computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform operations. The operations include receiving one or more infrared images taken of an environment. The operations also include applying a trained machine learning system to the one or more infrared images to determine an identified object type in the environment. Applying the trained machine learning system includes determining one or more previous heat maps associated with the environment. Applying the trained machine learning system also includes using the one or more previous heat maps and the one or more infrared images to determine a current heat map associated with the environment, and determining the identified object type based on the current heat map. The operations further include providing the identified object type.

[0006] Other aspects, embodiments, and implementations will become apparent to those skilled in the art from a reading of the following detailed description, where appropriate with reference to the accompanying drawings. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates a block diagram of a vehicle, according to an example embodiment. [Figure 2A] 1 illustrates the physical configuration of a vehicle according to an example embodiment. [Figure 2B] 1 illustrates the physical configuration of a vehicle according to an example embodiment. [Figure 2C] 1 illustrates the physical configuration of a vehicle according to an example embodiment. [Figure 2D] 1 illustrates the physical configuration of a vehicle according to an example embodiment. [Figure 2E] 1 illustrates the physical configuration of a vehicle according to an example embodiment. [Figure 3] 1 illustrates wireless communication between various computing systems associated with a vehicle, according to an example embodiment. [Figure 4] 1 illustrates the training and inference phases of a machine learning model, according to an example embodiment. [Figure 5] 1 illustrates a system, according to an example embodiment. [Figure 6A] 1 illustrates aspects of a thermal machine learning model, according to an example embodiment. [Figure 6B] 1 illustrates aspects of an emissivity machine learning model, according to an example embodiment. [Figure 7] 1 illustrates a method according to an example embodiment. [Figure 8] 1 illustrates a method according to an example embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Example methods, devices, and systems are described herein. It should be understood that the words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or features. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein.

[0009] As such, the example embodiments described herein are not meant to be limiting. The aspects of the disclosure as generally described herein and illustrated in the Figures can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

[0010] Furthermore, unless the context dictates otherwise, the features shown in each of these figures can be used in combination with one another. As such, the figures should generally be considered as component aspects of one or more overall embodiments, with the understanding that not every illustrated feature is required for every embodiment.

[0011] I. Overview Conventional sensor systems tend to misidentify and / or misclassify objects in a given environment. This degradation can be particularly true for identification techniques that utilize thermal imaging. In such cases, large temperature variations in the environment can make it difficult for thermal imaging systems to detect objects. For example, daily and / or seasonal variations in the environment can change the appearance of objects of interest, introduce thermal noise, and so on. Such variations limit the ability to reliably train machine learning models to classify objects based on thermal images (or "thermal maps" as referred to herein) obtained from thermal imaging systems. Some solutions attempt to normalize the variations with radiometrically calibrated cameras that estimate the temperature in the environment, but the accuracy of these estimates is limited by one or more assumptions about, or approximations of, the emissivity of objects in a given environment.

[0012] The embodiments described herein can enhance the performance of thermal imaging systems to correctly identify and / or classify objects within a given environment. Specifically, the disclosure provides a machine learning system that can accurately predict the emissivity and / or temperature of objects within an environment, thereby improving the classification of objects within such environments.

[0013] In some embodiments, the machine learning system may include models trained on visible and near-infrared reflectance, two-color infrared radiation, LIDAR data, and / or ambient temperature / sun position / weather data to infer the emissivity of objects in the environment to more accurately estimate the physical temperature of such objects.

[0014] To further improve classification, the machine learning system can compare the current thermal image with previous thermal images that have been processed to remove the organism. The systems and methods herein can also combine LIDAR data with thermal images to detect and track moving organisms by correlating object points across various thermal image frames. Other embodiments, aspects, and enhancements may be possible.

[0015] II. System Example Example systems within the scope of the present disclosure will now be described in more detail. Some example systems may be implemented in or take the form of automobiles. However, some example systems may also be implemented in or take the form of other vehicles, such as cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, bulldozers, boats, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment, construction equipment, trams, golf carts, trains, trolleys, robotic devices, etc. Other vehicles are also contemplated. Furthermore, in some embodiments, example systems may not include a vehicle.

[0016] Referring now to the figures, Figure 1 is a functional block diagram illustrating an example vehicle 100, according to an example embodiment, that may be configured to operate fully or partially in an autonomous mode. More specifically, vehicle 100 may operate in the autonomous mode without human interaction through receiving control instructions from a computing system. As part of operating in the autonomous mode, vehicle 100 may use sensors to detect and possibly identify objects in the surrounding environment to enable safe navigation. In some embodiments, vehicle 100 may also include subsystems that enable a driver to control the operation of vehicle 100.

[0017] 1, vehicle 100 may include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripherals 108, a power source 110, a computer system 112 (sometimes referred to as a computing system), data storage 114, and a user interface 116. In other examples, vehicle 100 may include more or fewer subsystems, each of which may include multiple elements. The subsystems and components of vehicle 100 may be interconnected in various manners.

[0018] Propulsion system 102 may include one or more components operable to provide powered motion to vehicle 100, and may include, among other possible components, engine / motor 118, energy source 119, transmission 120, and wheels / tires 121. For example, engine / motor 118 may be configured to convert energy source 119 into mechanical energy and may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine, among other possible options. For example, in some embodiments, propulsion system 102 may include multiple types of engines and / or motors, such as gasoline engines and electric motors.

[0019] Energy source 119 represents an energy source that may fully or partially power one or more systems (e.g., engine / motor 118) of vehicle 100. For example, energy source 119 may correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, energy source 119 may include a combination of a fuel tank, batteries, capacitors, and / or flywheels.

[0020] The transmission 120 may transfer mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. As such, the transmission 120 may include, among other possible components, a gearbox, a clutch, a differential, and a drive shaft. The drive shaft may include an axle that connects to one or more wheels / tires 121.

[0021] The wheels / tires 121 of the vehicle 100 may have a variety of configurations within example embodiments. For example, the vehicle 100 may exist in the form of a unicycle, a bicycle / motorcycle, a tricycle, or a four-wheeled car / truck, among other possible configurations. Thus, the wheels / tires 121 may be connected to the vehicle 100 in a variety of ways and may exist in different materials, such as metal and rubber.

[0022] The sensor system 104 may include various types of sensors, such as a global positioning system (GPS) 122, an inertial measurement unit (IMU) 124, radar 126, a laser range finder / LIDAR 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, among other possible sensors. In some embodiments, the sensor system 104 may include sensors configured to monitor internal systems of the vehicle 100 (e.g., an O monitor, a fuel gauge, engine oil temperature, brake wear).

[0023] The GPS 122 may include a transceiver operable to provide information regarding the position of the vehicle 100 relative to the Earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes and may sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 while the vehicle 100 is stationary or moving.

[0024] Radar 126 may represent one or more systems configured to sense objects in the local environment of vehicle 100 using radio signals, including the objects' speed and direction of travel. Thus, radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, radar 126 may correspond to a wearable radar system configured to obtain measurements of the environment surrounding vehicle 100.

[0025] The laser rangefinder / LIDAR 128 may include one or more laser sources, a laser scanner, and one or more detectors, among other system components, and may operate in a coherent mode (e.g., using heterodyne detection) or a non-coherent detection mode. In some embodiments, the one or more detectors of the laser rangefinder / LIDAR 128 may include one or more light-receiving elements. Such light-receiving elements may be particularly sensitive detectors (e.g., avalanche photodiodes (APDs)). In some examples, such light-receiving elements may even be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Such light-receiving elements may also be arranged in an array (e.g., as found in silicon photomultipliers (SiPMs)).

[0026] Camera 130 may include one or more devices (e.g., still cameras or video cameras) configured to capture images of the environment of vehicle 100.

[0027] Steering sensor 123 may sense the steering angle of vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representative of the angle of the steering wheel. In some embodiments, steering sensor 123 may measure the angle of the wheels of vehicle 100, such as detecting the angle of the wheels relative to the forward axle of vehicle 100. Steering sensor 123 may also be configured to measure a combination (or subset) of the steering wheel angle, the electrical signal representative of the steering wheel angle, and the angle of the wheels of vehicle 100.

[0028] The throttle / brake sensor 125 may detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angle of both the accelerator pedal (throttle) and the brake pedal, or may measure an electrical signal that may represent, for example, the accelerator pedal (throttle) angle and / or the brake pedal angle. The throttle / brake sensor 125 may also measure the angle of a throttle body of the vehicle 100, which may include part of the physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., a butterfly valve or a carburetor). Additionally, the throttle / brake sensor 125 may measure the pressure of one or more brake pads on a rotor of the vehicle 100, or a combination (or subset) of the angles of the accelerator pedal (throttle) and the brake pedal, an electrical signal representing the accelerator pedal (throttle) and the brake pedal angle, the throttle body angle, and the pressure that at least one brake pad applies to a rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure pressure applied to a vehicle pedal, such as a throttle or brake pedal.

[0029] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / pathfinding system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the heading of the vehicle 100, and the throttle 134 may control the operating speed of the engine / motor 118 to control the acceleration of the vehicle 100. The braking unit 136 may decelerate the vehicle 100, which may include slowing the wheels / tires 121 using friction. In some embodiments, the braking unit 136 may convert the kinetic energy of the wheels / tires 121 into electrical current for subsequent use by one or more systems of the vehicle 100.

[0030] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithm capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide an assessment based on the incoming sensor data, such as an assessment of individual objects and / or features, an assessment of a particular situation, and / or an assessment of possible effects within a given situation.

[0031] Computer vision system 140 may include hardware and software operable to process and analyze images in an attempt to determine objects, environmental objects (e.g., traffic signals, roadway boundaries, etc.), and obstacles. Thus, computer vision system 140 may employ object recognition, Structure From Motion (SFM), video tracking, and other algorithms used in computer vision, for example, to recognize objects, map the environment, track objects, estimate object speed, etc.

[0032] Navigation / routing system 142 may determine a driving route for vehicle 100, which may include dynamically adjusting navigation during operation. Thus, navigation / routing system 142 may use data from sensor fusion algorithms 138, GPS 122, and maps, among other sources, to navigate vehicle 100. Obstacle avoidance system 144 may evaluate potential obstacles based on sensor data and cause systems of vehicle 100 to avoid or otherwise navigate the potential obstacles.

[0033] 1 , vehicle 100 may also include peripherals 108, such as a wireless communication system 146, a touchscreen 148, a microphone 150, and / or a speaker 152. Peripherals 108 may provide controls or other elements for a user to interact with a user interface 116. For example, touchscreen 148 may provide information to a user of vehicle 100. User interface 116 may also accept input from a user via touchscreen 148. Peripherals 108 may also enable vehicle 100 to communicate with devices, such as devices in other vehicles.

[0034] The wireless communication system 146 may communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communications such as Code Division Multiple Access (CDMA), Evolutionary Data Optimization (EVDO), Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or 4G cellular communications such as Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE). Alternatively, the wireless communication system 146 may communicate with a wireless local area network (WLAN) using WiFi or other possible connections. The wireless communication system 146 may also communicate directly with devices using, for example, an infrared link, BLUETOOTH®, or ZigBee®. Other wireless protocols, such as various vehicle communication systems, are possible within the context of this disclosure. For example, the wireless communication system 146 may include one or more dedicated short-range communication (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside gas stations.

[0035] Vehicle 100 may include a power source 110 for powering its components. Power source 110, in some embodiments, may include a rechargeable lithium-ion or lead-acid battery. For example, power source 110 may include one or more batteries configured to provide power. Vehicle 100 may also use other types of power sources. In some example embodiments, power source 110 and energy source 119 may be integrated into a single energy source.

[0036] Vehicle 100 may also include a computer system 112 for performing operations such as those described herein. Accordingly, computer system 112 may include at least one processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored on a non-transitory computer-readable medium, such as data storage 114. In some embodiments, computer system 112 may represent multiple computing devices that may function to provide distributed control of individual components or subsystems of vehicle 100.

[0037] In some embodiments, data storage 114 may include instructions 115 (e.g., program logic) executable by processor 113 for performing various functions of vehicle 100, including those described above in connection with Figure 1. Data storage 114 may also include additional instructions, including instructions for transmitting data to, receiving data from, interacting with, and / or controlling one or more of propulsion system 102, sensor system 104, control system 106, and peripherals 108.

[0038] In addition to instructions 115, data storage 114 may store data such as road maps, route information, among other information, that may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0039] Vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of vehicle 100. User interface 116 may control or allow control of content and / or the layout of interactive images that may be displayed on touchscreen 148. Additionally, user interface 116 may include one or more input / output devices in the set of peripherals 108, such as wireless communication system 146, touchscreen 148, microphone 150, and speaker 152.

[0040] Computer system 112 may control functions of vehicle 100 based on inputs received from various subsystems (e.g., propulsion system 102, sensor system 104, and control system 106) and from user interface 116. For example, computer system 112 may utilize inputs from sensor system 104 to estimate outputs generated by propulsion system 102 and control system 106. In some embodiments, computer system 112 may be operable to monitor many aspects of vehicle 100 and its subsystems. In some embodiments, computer system 112 may disable some or all functions of vehicle 100 based on signals received from sensor system 104.

[0041] Components of vehicle 100 may be configured to work in interconnection with other components within or outside their respective systems. For example, in an example embodiment, camera 130 may capture multiple images that may represent information about the state of the environment of vehicle 100 operating in autonomous mode. The state of the environment may include parameters of the road on which the vehicle is operating. For example, computer vision system 140 may be able to recognize slopes (gradients) or other features based on multiple images of the road. Additionally, the combination of GPS 122 and features recognized by computer vision system 140 may be used with map data stored in data storage 114 to determine certain road parameters. Furthermore, radar 126 may also provide information about the vehicle's surroundings.

[0042] In other words, a combination of various sensors (which may be referred to as input and output indication sensors) and the computer system 112 may interact to provide an indication of the inputs provided to control the vehicle or an indication of the vehicle's surroundings.

[0043] In some embodiments, computer system 112 may make decisions regarding various objects based on data provided by other systems. For example, vehicle 100 may have laser or other optical sensors configured to sense objects within the vehicle's field of view. Computer system 112 may use output from the various sensors to determine information regarding objects within the vehicle's field of view and may determine distance and direction information to the various objects. Computer system 112 may also determine whether an object is desirable or undesirable based on output from the various sensors.

[0044] 1 depicts various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) as being integrated into vehicle 100, one or more of these components may be separately mounted or associated with vehicle 100. For example, data storage 114 may exist partially or completely separate from vehicle 100. Thus, vehicle 100 may be provided in the form of device elements that may be located separately or together. The device elements that make up vehicle 100 may be communicatively coupled together in a wired and / or wireless manner.

[0045] 2A-2E illustrate an example vehicle 200 that may include some or all of the functionality described in connection with vehicle 100 with reference to FIG. 1, according to an example embodiment. Vehicle 200 is shown in FIGS. 2A-2E as a van for purposes of illustration, but the present disclosure is not so limited. For example, vehicle 200 may represent a truck, a car, a semi-trailer truck, a motorcycle, a golf cart, an off-road vehicle, an agricultural vehicle, or the like.

[0046] Example vehicle 200 includes a sensor unit 202, a first LIDAR unit 204, a second LIDAR unit 206, a first radar unit 208, a second radar unit 210, a first LIDAR / radar unit 212, a second LIDAR / radar unit 214, and also includes two additional locations 216, 218 on vehicle 200 where one or more radar units, LIDAR units, laser rangefinder units, and / or other types of sensors may be located. Each of first LIDAR / radar unit 212 and second LIDAR / radar unit 214 may take the form of a LIDAR unit, a radar unit, or both.

[0047] Additionally, the example vehicle 200 may include any of the components described in connection with the vehicle 100 of Figure 1. The first and second radar units 208, 210 and / or the first and second LIDAR units 204, 206 may actively scan their surrounding environment for potential obstructions and may be similar to the radar 126 and / or laser rangefinder / LIDAR 128 of the vehicle 100.

[0048] The sensor unit 202 is mounted on top of the vehicle 200 and includes one or more sensors configured to detect information about the environment surrounding the vehicle 200 and output an indication of the information. For example, the sensor unit 202 may include any combination of cameras, radar, LIDAR, range finders, and acoustic sensors. The sensor unit 202 may include one or more movable mounts that may be operable to adjust the orientation of one or more sensors within the sensor unit 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to obtain information from each direction around the vehicle 200. In another embodiment, the movable mount of the sensor unit 202 may be movable in a scanning manner within a specified range of angles and / or orientations. The sensor unit 202 may be mounted on the roof of the vehicle 200, although other mounting locations are also possible.

[0049] Additionally, the sensors of the sensor unit 202 may be distributed in various locations and need not be co-located in one location. Some possible sensor types and mounting locations include two additional locations 216, 218. Furthermore, each sensor of the sensor unit 202 may be configured to be moved or scanned independently of the other sensors of the sensor unit 202.

[0050] In one example configuration, one or more radar scanners (e.g., first and second radar units 208, 210) may be located near the rear of vehicle 200 to actively scan the environment near the rear of vehicle 200 for radio wave reflective objects. Similarly, first LIDAR / radar unit 212 and second LIDAR / radar unit 214 may be mounted near the front of vehicle 200 to actively scan the environment near the front of vehicle 200. The radar scanners may be positioned in a location suitable for illuminating an area including the forward path of vehicle 200, for example, without being obstructed by other features of vehicle 200. For example, the radar scanners may be embedded in and / or mounted near the front bumper, front headlights, cowl, and / or hood, etc. Also, one or more additional radar scanning devices may be positioned to actively scan the sides and / or rear of vehicle 200 for radio wave reflective objects, such as by including such devices on or near the rear bumper, side panels, rocker panels, and / or chassis, etc.

[0051] 2A-2E, vehicle 200 may include a wireless communication system. The wireless communication system may include a wireless transmitter and a wireless receiver that may be configured to communicate with devices external or internal to vehicle 200. Specifically, the wireless communication system may include a transceiver configured to communicate with other vehicles and / or computing devices, for example, in a vehicle communication system or road station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transport systems.

[0052] Vehicle 200 may optionally include a camera at a location inside sensor unit 202. The camera may be a light-sensitive device, such as a still camera, a video camera, or the like, configured to capture multiple images of the environment of vehicle 200. To this end, the camera may be configured to detect visible light and may additionally or alternatively be configured to detect light from other parts of the electromagnetic spectrum, such as infrared or ultraviolet light. The camera may be a two-dimensional detector and, optionally, may have a three-dimensional spatial range of sensitivity.

[0053] In some embodiments, the camera may include a range detector configured to generate a two-dimensional image showing, for example, the distance from the camera to several points in the environment. To this end, the camera may use one or more range detection techniques. For example, the camera may provide range information by using structured light techniques, in which the vehicle 200 illuminates objects in the environment with a predetermined light pattern, such as a grid or checkerboard pattern, and uses the camera to detect reflections of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, the vehicle 200 may determine the distance to points on the objects.

[0054] The predetermined light pattern may be composed of infrared light or other wavelengths of radiation suitable for such measurements. In some examples, the camera may be mounted inside the windshield of vehicle 200. Specifically, the camera may be positioned to capture images from a forward-looking perspective relative to the orientation of vehicle 200. Other mounting locations and viewing angles for the camera may also be used, either inside or outside vehicle 200. The camera may also have associated optics operable to provide an adjustable field of view. Furthermore, the camera may be mounted to vehicle 200 using a movable mount to change the pointing angle of the camera, such as via a pan / tilt mechanism.

[0055] A control system of vehicle 200 may be configured to control vehicle 200 according to a control strategy from among multiple possible control strategies. The control system may be configured to receive information (vehicle 200 on or off) from sensors coupled to vehicle 200, modify the control strategy (and associated driving behavior) based on the information, and control vehicle 200 according to the modified control strategy. The control system may be further configured to monitor the information received from the sensors and constantly evaluate driving conditions, and may be configured to modify the control strategy and driving behavior based on changes in driving conditions.

[0056] 3 illustrates wireless communication between various computing systems associated with a vehicle, according to an example embodiment. Specifically, wireless communication may occur between a remote computing system 302 and the vehicle 200 over a network 304. Wireless communication may also occur between a server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0057] Remote computing system 302 may represent any type of device involved in remote assistance techniques, including but not limited to those described herein. In examples, remote computing system 302 may represent any type of device configured to (i) receive information related to vehicle 200, (ii) provide an interface through which a human operator can then observe the information and enter a response related to the information, and (iii) transmit the response to vehicle 200 or to another device. Remote computing system 302 may take various forms, such as a workstation, a desktop computer, a laptop, a tablet, a mobile phone (e.g., a smartphone), and / or a server. In some examples, remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0058] Remote computing system 302 may include one or more subsystems and components that are similar to or identical to those of vehicle 100 or vehicle 200. At a minimum, remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, remote computing system 302 may also include a user interface that includes input / output devices such as a touchscreen and speakers. Other examples are also possible.

[0059] Network 304 represents an infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0060] The location of remote computing system 302 can vary within the examples. For example, remote computing system 302 can have a location remote from vehicle 200 with wireless communication over network 304. In another example, remote computing system 302 can correspond to a computing device within vehicle 200 that is separate from vehicle 200 but that allows a human operator to interact with a passenger or driver of vehicle 200. In some examples, remote computing system 302 can be a computing device with a touchscreen that can be operated by a passenger of vehicle 200.

[0061] In some embodiments, the operations described herein performed by remote computing system 302 may additionally or alternatively be performed by vehicle 200 (i.e., by any system or subsystem of vehicle 200). In other words, vehicle 200 may be configured to provide remote assistance mechanisms with which a driver or passengers of the vehicle can interact.

[0062] Server computing system 306 may be configured to wirelessly communicate with remote computing system 302 and vehicle 200 over network 304 (or, in some cases, directly with remote computing system 302 and / or vehicle 200). Server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information related to vehicle 200 and its remote assistance. As such, server computing system 306 may be configured to perform any operation or portion of such operation described herein as being performed by remote computing system 302 and / or vehicle 200. Some embodiments of wireless communication related to remote assistance may utilize server computing system 306, while other embodiments may not.

[0063] The server computing system 306 may include one or more subsystems and components similar to or identical to the subsystems and components of the remote computing system 302 and / or the vehicle 200, such as a processor configured to perform the various operations described herein, and a wireless communication interface for receiving information from and providing information to the remote computing system 302 and the vehicle 200.

[0064] In line with the above description, computing systems (e.g., remote computing system 302, server computing system 306, or a computing system local to vehicle 200) may operate to capture images of the autonomous vehicle's environment using cameras. Generally, at least one computing system may analyze the images and, in some cases, control the autonomous vehicle.

[0065] In some embodiments, to facilitate autonomous operation, a vehicle (e.g., vehicle 200) may receive data representing objects in the environment in which the vehicle operates (also referred to herein as "environmental data") in various ways. A sensor system on the vehicle may provide the environmental data representing objects in the environment. For example, the vehicle may have various sensors including cameras, radar units, laser range finders, microphones, radio units, and other sensors. Each of these sensors may communicate environmental data regarding the information each respective sensor receives to a processor within the vehicle.

[0066] While operating in autonomous mode, the vehicle may control its operation with little or no human input. For example, a human operator may input an address into the vehicle, and the vehicle may be able to drive to the specified destination without further input from the human (e.g., without the human having to steer or touch the brake / accelerator pedals). Additionally, while the vehicle is operating autonomously, the sensor system may receive environmental data. The vehicle's processing system may alter the control of the vehicle based on the environmental data received from the various sensors. In some examples, the vehicle may vary the vehicle's speed in response to the environmental data from the various sensors. The vehicle may vary its speed to avoid obstacles, obey traffic laws, etc. If the processing system in the vehicle identifies an object near the vehicle, the vehicle may be able to change its speed or otherwise alter its movement.

[0067] If the vehicle detects an object but is not very confident in detecting such an object, the vehicle can request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object is, in fact, in the environment (e.g., whether there is actually a stop sign or whether there is actually no stop sign at all), (ii) verifying whether the vehicle's identification of the object is correct, (iii) correcting the identification if it is incorrect, and / or (iv) providing supplemental instructions (or amending current instructions) to the autonomous vehicle. Remote assistance tasks can also include the human operator providing instructions to control the vehicle's behavior (e.g., if the human operator determines that the object is a stop sign, commanding the vehicle to stop at the stop sign), although in some cases the vehicle itself may control its own behavior based on the human operator's feedback regarding the object's identification.

[0068] To facilitate this, the vehicle may analyze environmental data representing objects in the environment to determine at least one object having a detection confidence below a threshold. A processor of the vehicle may be configured to detect various objects in the environment based on the environmental data from various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, road signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0069] The detection confidence may indicate the likelihood that a determined object is correctly identified or present in the environment. For example, the processor may perform object detection of objects in image data in the received environmental data and determine that at least one object has a detection confidence below a threshold based on being unable to identify the object with a detection confidence above a threshold. If the object detection or object recognition results for an object are inconclusive, the detection confidence may be low or below a set threshold.

[0070] The vehicle may detect environmental objects in a variety of ways, depending on the source of the environmental data. In some embodiments, the environmental data comes from a camera and may be image or video data. In other embodiments, the environmental data may come from a LIDAR unit. The vehicle may analyze the captured image or video data to identify objects in the image or video data. Methods and apparatus may be configured to monitor the image and / or video data for environmental objects. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify environmental objects based on the radar, audio, or other data.

[0071] III. Machine Learning System Examples FIG. 4 illustrates a system 400 illustrating a learning phase 402 and an inference phase 404 of a trained machine learning model 432, according to an example embodiment. Some machine learning techniques involve training one or more machine learning systems on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions regarding (the patterns in) the training data. The resulting trained machine learning models may be referred to as trained machine learning systems or trained machine learning models. For example, FIG. 4 illustrates a learning phase 402 in which one or more machine learning systems 420 are trained on training data 410 to become one or more trained machine learning models 432. Then, during the inference phase 404, the trained machine learning models 432 receive input data 430 and one or more inference / prediction requests 440 (perhaps as part of the input data 430) and, in response, can provide one or more inferences and / or predictions 450 as output.

[0072] The machine learning system 420 may include, but is not limited to, an artificial neural network (e.g., a convolutional neural network described herein, a recurrent neural network using the segmentation techniques described herein), a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system. During the learning phase 402, the machine learning system 420 may be trained by providing at least the training data 410 as a training input using a training technique such as, but not limited to, unsupervised, supervised, semi-supervised, reinforcement learning, transfer learning, incremental learning, and / or curriculum learning techniques.

[0073] Unsupervised learning may involve providing a portion (or all) of the training data 410 to the machine learning system 420, which may determine one or more output inferences based on the provided portion (or all) of the training data 410. Supervised learning may involve providing a portion of the training data 410 to the machine learning system 420, which may determine one or more output inferences based on the provided portion (or all) of the training data 410, and which may be accepted or corrected based on the correct results associated with the training data 410. In some examples, the supervised learning of the machine learning system 420 may be governed by a set of rules and / or a set of labels on the training inputs, and the set of rules and / or the set of labels may be used to correct the inferences of the machine learning system 420.

[0074] Semi-supervised learning may involve obtaining correct results (e.g., partially labeled data) for some, but not necessarily all, of the training data 410. During semi-supervised learning, supervised learning is used for the portions of the training data 410 that have correct results, and unsupervised learning is used for the portions of the training data 410 that do not produce correct results.

[0075] Reinforcement learning involves the machine learning system 420 receiving a reward signal for a previous inference, where the reward signal can be a numerical value. During reinforcement learning, the machine learning system 420 can output an inference and receive a reward signal in response, where the machine learning system 420 is configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected sum of the numerical values ​​provided by the reward signal over time.

[0076] Transfer learning techniques may include pre-training the trained machine learning model 432 on one dataset and further training it using the training data 410. More specifically, the machine learning system 420 may be pre-trained on data from one or more computing devices, and the resulting trained machine learning model may be provided to the computing device CD1, which is intended to execute the trained machine learning model during the inference phase 404. Then, during the learning phase 402, the pre-trained machine learning model may be further trained using the training data 410, where the training data 410 may be derived from kernel data and non-kernel data of the computing device CD1. This further training of the machine learning system 420 and / or the pre-trained machine learning model using the training data 410 of the data of CD1 may be performed using either supervised learning or unsupervised learning. Once the machine learning system 420 and / or the pre-trained machine learning model have been trained on at least the training data 410, the learning phase 402 may be completed. The resulting trained machine learning model may be utilized as at least one of the trained machine learning models 432.

[0077] Incremental learning techniques may involve providing the trained machine learning model 432 (and possibly the machine learning system 420) with input data that is used to continually expand the knowledge of the trained machine learning model 432. Curriculum learning techniques may involve providing the machine learning system 420 with training data arranged in a particular order, such as providing relatively easy training examples first and gradually progressing to more difficult training examples, similar to a curriculum or course of study in a school. Other techniques are also possible for training the machine learning system 420 and / or the trained machine learning model 432.

[0078] In some examples, after the learning phase 402 is complete but before the inference phase 404 begins, the trained machine learning model 432 may be provided to a computing device CD1 on which the trained machine learning model 432 does not yet reside; for example, after the learning phase 402 is complete, the trained machine learning model 432 may be downloaded to the computing device CD1.

[0079] For example, computing device CD2 storing trained machine learning model 432 may provide the trained machine learning model 432 to computing device CD1 by one or more of communicating a copy of the trained machine learning model 432 to computing device CD1, making a copy of the trained machine learning model 432 for computing device CD1, providing access to the trained machine learning model 432 to computing device CD1, and / or otherwise providing a trained machine learning system to computing device CD1. In some examples, the trained machine learning model 432 may be used by computing device CD1 immediately after being provided by computing device CD2. In some examples, after the trained machine learning model 432 is provided to computing device CD1, the trained machine learning model 432 may be installed and / or otherwise prepared for use before the trained machine learning model 432 can be used by computing device CD1.

[0080] During the inference phase 404, the trained machine learning model 432 can receive input data 430 and generate and output corresponding inferences and / or predictions 450 for the input data 430. In this manner, the input data 430 can be used as input to the trained machine learning model 432 to provide the corresponding inferences and / or predictions 450 to kernel and non-kernel components. For example, the trained machine learning model 432 can generate the inferences and / or predictions 450 in response to an inference / prediction request 440. In some examples, the trained machine learning model 432 can be executed by another piece of software. For example, the trained machine learning model 432 can be executed by an inference or prediction daemon so that it is readily available to provide inferences and / or predictions upon request. The input data 430 can include data from the computing device CD1 executing the trained machine learning model 432 and / or input data from one or more computing devices other than CD1.

[0081] In some examples, input data 430 may include data from the environment in which vehicle 100 / 200 operates. As noted above, vehicle 100 / 200 may have a variety of sensors, including cameras, radar units, laser range finders, microphones, radio units, and other sensors. Each of these sensors may convey this environmental data as an input to trained machine learning model 432. However, other types of input data are also possible.

[0082] The inferences and / or predictions 450 may include output images, heat maps, depth maps, numerical values, and / or other output data produced by the trained machine learning model 432 operating on the input data 430 (and training data 410). In some examples, the trained machine learning model 432 can use the output inferences and / or predictions 450 as input feedback 460. The trained machine learning model 432 can also utilize past inferences as input to generate new inferences.

[0083] In some examples, the machine learning system 420 and / or the trained machine learning model 432 may be executed and / or accelerated using one or more computer processors and / or on-device coprocessors. On-device coprocessors may include, but are not limited to, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more digital signal processors (DSPs), and / or one or more application-specific integrated circuits (ASICs). Such on-device coprocessors may speed up the training of the machine learning system 420 and / or the generation of inferences and / or predictions 450 by the trained machine learning model 432. In some examples, the trained machine learning model 432 may be trained, reside, execute, and / or otherwise perform inferences for a particular computing device to provide inferences and / or predictions 450.

[0084] In some examples, one computing device CD_SOLO may include a trained learning model 432, perhaps after training a machine learning system 420 on the computing device CD_SOLO. The computing device CD_SOLO may thereby receive an inference / prediction request 440 to provide an inference and / or prediction 450 and may provide the inference and / or prediction 450 accordingly using the trained machine learning model 432 operating on input data 430, where the inference and / or prediction 450 may be provided using a user interface and / or display, such as one or more electronic communications, one or more printed documents, etc.

[0085] In some examples, two or more computing devices CD_CLI and CD_SRV may be used to provide inferences and / or predictions 450. For example, a first computing device CD_CLI may generate and send an inference / prediction request 440 to a second computing device CD_SRV. Upon receiving the inference / prediction request 440 from CD_CLI, CD_SRV may operate on the input data 430 using a trained machine learning model 432, perhaps after training a machine learning system 420, and determine an inference and / or prediction 450 accordingly. After determining the inference and / or prediction 450, CD_SRV fulfills the request from CD_CLI by providing the inference and / or prediction 450 to CD_CLI.

[0086] It should be noted that in the above example, CD1, CD2, CD_SOLO, CD_CLI, and CD_SRV may take the form of vehicle 100 or another similar vehicle.

[0087] As discussed above, vehicle 100 can detect objects in its surrounding environment in a variety of ways. In some embodiments, vehicle 100 can use a thermal imaging system to actively transmit electromagnetic signals (e.g., near-infrared light) that are reflected by target objects near vehicle 100. The thermal imaging system can capture the reflected electromagnetic signals using detector elements. In other embodiments, the thermal imaging system can passively capture infrared radiation (e.g., mid-infrared or far-infrared light) emitted as heat from target objects near vehicle 100. The captured reflected electromagnetic signals and / or captured infrared radiation can enable vehicle 100 to classify objects in its surrounding environment using thermography techniques (e.g., classifying objects as animate or inanimate based on temperature variations between the objects).

[0088] Nevertheless, the vehicle 100's ability to accurately classify objects may be limited by daily / seasonal variations in its surrounding environment. Factors such as the amount of sunlight and the ambient temperature in the environment may introduce noise into the captured reflected electromagnetic signals or captured infrared emissions. Also, because thermography typically uses the thermal contrast between an object and its background environment, changes in the surrounding environment may make it difficult to apply thermography to correctly distinguish objects. As an example, living organisms may have different thermal contrasts from their surrounding environment at night than during the day.

[0089] Some solutions attempt to normalize these variations via a radiometrically calibrated camera that estimates the temperature within the surrounding environment. That is, the radiometrically calibrated camera may be configured to use a standard emissivity value across the surrounding environment. However, using a standard emissivity can reduce the accuracy of thermographic techniques. For example, the emissivity of asphalt is significantly different from that of grass, so setting a standard emissivity for both of these materials can result in inaccurate object classification.

[0090] To address these and other issues, the present embodiments provide a machine learning system that can be trained to generate one or more prior thermal maps based on environmental attributes surrounding a vehicle. These prior thermal maps can be used to normalize thermal variations in the environment surrounding the vehicle. Additionally, the machine learning system described herein can utilize visible-near-infrared reflectance, two-color infrared radiation, and / or LIDAR data to predict the intrinsic emissivity of various elements in an environment, thereby improving classification of objects within such an environment. Other advantages may also be considered.

[0091] 5 illustrates an example system 500 that may be configured to classify objects in an environment through which a vehicle 100 is passing, according to an example embodiment. System 500 is shown to include an environmental sensor 510, an infrared module 520, a LIDAR / visible module 530, a thermal machine learning (ML) model 540, an emissivity ML model 550, a normalization module 560, a projection module 570, and a classification machine learning ML model 580.

[0092] Environmental sensors 510 may be configured to ascertain physical properties associated with the current environment surrounding vehicle 100. The physical properties may include properties representative of ambient / background characteristics of the surrounding environment. Environmental sensors 510 may responsively generate environmental attributes 512, which are signals representative of the current environment surrounding vehicle 100.

[0093] Infrared module 520 may be configured to generate infrared image 522 corresponding to the environment through which vehicle 100 is passing. In some embodiments, infrared module 520 actively emits near-infrared light pulses that reflect off respective points in the environment. The intensity of these reflected pulses (i.e., return pulses) may be measured by infrared module 520 and captured as infrared image 522. In some embodiments, infrared module 520 may passively measure infrared radiation (e.g., mid-infrared radiation or far-infrared radiation) emitted as heat from target objects near vehicle 100. The intensity of the measured infrared radiation may be captured as infrared image 522. In some embodiments, infrared image 522 represents multiple different images captured from the environment at multiple different times.

[0094] The LIDAR / visible module 530 may be configured to generate LIDAR data / visible light data from the environment through which the vehicle 100 is traversing. In some embodiments, the LIDAR / visible module 530 may utilize the laser rangefinder / LIDAR unit 128 to transmit electromagnetic signals in the near-infrared spectrum (e.g., 905 nm or 1550 nm) that are reflected by various objects near the vehicle 100. In some embodiments, the LIDAR / visible module 530 may utilize the camera 130 to transmit electromagnetic signals in the visible light spectrum (e.g., 380 nm to 740 nm) that are reflected by various objects near the vehicle 100. The LIDAR / visible module 530 may capture the reflected electromagnetic signals and, in response, may make various determinations about the object that reflected the electromagnetic signals. For example, the LIDAR / visible module 530 may determine the reflectivity of the object that reflected the electromagnetic signals based on the radiant energy of the reflected electromagnetic signals. Thus, the LIDAR / visual data 532 may represent the radiant energy of reflected electromagnetic signals captured by the LIDAR / visual module 530 from the environment.

[0095] The thermal ML model 540 may be implemented as software instructions executable by a processor (e.g., processor 113), as programmable circuitry (e.g., a field programmable gate array (FPGA)), as dedicated circuitry (e.g., an application specific integrated circuit (ASIC)), or as a combination thereof. The thermal ML model 540 may be communicatively connected to the environmental sensor 510. In operation, the thermal ML model 540 may receive the environmental attributes 512 from the environmental sensor 510 and process the environmental attributes 512 to generate a previous thermal map 542. The previous thermal map 542 may correspond to a thermal map (e.g., 3-D or 2-D) captured under similar physical properties as the environmental attributes 512.

[0096] Vehicle 100 can use one or more previous thermal maps 542 to normalize thermal variations in infrared image 522. For example, when infrared module 520 generates infrared image 522 representing the environment surrounding vehicle 100, thermal ML model 540 can also responsively generate previous thermal map 542 using environmental attributes 512 received from environmental sensor 510. Both infrared image 522 and previous thermal map 542 can then be transmitted to normalization module 560, which can be implemented as software instructions executable by a processor (e.g., processor 113). Normalization module 560 can subtract previous thermal map 542 from infrared image 522 to produce normalized map 562. The idea here is that because normalization module 560 removes thermal noise / background in infrared image 522, the resulting normalized map 562 allows objects to exhibit higher contrast from their surrounding environment, which improves the ability to classify these objects. In some embodiments, if multiple prior heat maps 542 are used, the normalization module 560 may average or otherwise weight the values ​​of each of the multiple heat maps before performing normalization. In some embodiments, the normalization module 560 may also act to edit or otherwise remove areas in the prior heat maps 542 that include predefined object types (e.g., living things).

[0097] The emissivity ML model 550 may be implemented as software instructions executable by a processor (e.g., processor 113). The emissivity ML model 550 may be communicatively coupled to the LIDAR / visible module 530. In operation, the emissivity ML model 550 may receive LIDAR / visible data 532 from the LIDAR / visible module 530 and process the LIDAR / visible data 532 to generate an emissivity map 552. In an example, each point (e.g., pixel) in the emissivity map 552 may be labeled with a corresponding emissivity value. Thus, the emissivity map 552 may be used to predict the intrinsic emissivity of various objects in the environment surrounding the vehicle 100.

[0098] Projection module 570 may be implemented as software instructions executable by a processor (e.g., processor 113). Projection module 570 may take as input both normalization map 562 and emissivity map 552. Projection module 570 may then match coordinates of emissivity map 552 with coordinates of normalization map 562 and project emissivity values ​​from emissivity map 552 onto normalization map 562. The resulting projection map 572 may then include the emissivity value for each point in normalization map 562.

[0099] In some embodiments, projection module 570 may combine emissivity map 552 and normalization map 562 to generate a temperature map. In other words, projection map 572 may take the form of a temperature map of the environment surrounding vehicle 100. To do this, projection module 570 may use emissivity map 552 to determine emissivity values ​​of various objects in the environment. Projection module 570 may then calculate the contact temperature of each object by using the determined emissivity values ​​of the objects and the infrared radiation of the objects, as captured by normalization map 562.

[0100] In some embodiments, system 500 can directly determine a temperature map via two-color ratio thermometry. For example, infrared module 520 can include infrared light emitter devices and / or light receivers / cameras operating in various spectral wavelength bands (e.g., mid-wavelength infrared (3-5 μm) and long-wavelength infrared (7-14 μm)). As a result, a temperature map can be computed by calculating the ratio between the respective spectral intensity information of the various wavelength bands.

[0101] Classification ML model 580 may be implemented as software instructions executable by a processor (e.g., processor 113). Classification ML model 580 may be communicatively coupled to projection module 570. In operation, classification ML model 580 may receive projection map 572 from projection module 570 and process projection map 572 to classify objects in projection map 572. In examples, classification ML model 580 may be configured to identify living things, traffic lights and signs, roads, vegetation, and other environmental features, including mailboxes, benches, trash cans, sidewalks, and / or any other objects in the environment that may be of interest to the operation of vehicle 100.

[0102] In example embodiments, classification ML model 580 can perform spatiotemporal association of objects to improve classification accuracy. For example, classification ML model 580 can operate to classify objects at timestamps T-1 and T-2. In performing this classification, classification ML model 580 can be configured to assign an identifier to each of the objects classified at timestamps T-1 and T-2. Thus, upon classifying an object at timestamp T, classification ML model 580 can assign a high level of confidence to the classification of the object if the object's identifier was present at timestamps T-1 and T-2. For example, if "Pedestrian A" was present at timestamps T-1 and T-2, classification ML model 580 can assign a high level of confidence to the "Pedestrian A" classification at timestamp T. In some embodiments, classification ML model 580 can perform spatiotemporal association of objects using Mahalanobis distance between the timestamps and uncertainty information from the Kalman filter, taking into account the object's location.

[0103] In example embodiments, the classification ML model 580 may take the form of a convolutional neural network (CNN). In some cases, the CNN may follow a semantic segmentation architecture using an encoder network coupled to a decoder network. For example, the classification ML model 580 may take the form of a U-Net. The classification ML model 580 may take the form of a trained machine learning model 432.

[0104] In example embodiments, the classification ML model 580 may be configured to detect objects with a confidence threshold. The confidence threshold may vary depending on the object type being detected. For example, an object that may require quick response action, such as the brake lights of another vehicle, may have a lower confidence threshold. However, in other embodiments, the confidence threshold may be the same for all detected objects.

[0105] If the confidence associated with the detected object is higher than the confidence threshold, the classification ML model 580 may consider the object to have been correctly recognized and adjust the control of the vehicle 100 accordingly based on that assumption.

[0106] If the confidence associated with the detected object is lower than a confidence threshold, the action taken by classification ML model 580 may vary. In some embodiments, classification ML model 580 may cause vehicle 100 to respond as if the detected object were present, despite the low confidence level. In other embodiments, classification ML model 580 may cause vehicle 100 to respond as if the detected object were not present. In some embodiments, if an object is detected with a confidence below the confidence threshold, the object may be given a preliminary identification, and classification ML model 580 may cause vehicle 100 to adjust its operation in response to the preliminary identification. Such adjustments in operation may take the form of stopping, switching to a human-controlled (e.g., manual) operation mode, changing speed (e.g., speed and / or direction), among other adjustments.

[0107] In some embodiments, in response to determining that the object has a detection confidence below a threshold, classification ML model 580 may send a request for remote assistance along with an identification of the object to a remote computing system, such as remote computing system 302. Additionally and / or alternatively, if vehicle 100 has an obstructed view of its surrounding environment and / or if environmental sensors 510, infrared module 520, or LIDAR / visible module 530 are malfunctioning, vehicle 100 may request remote assistance from a remote computing system.

[0108] As mentioned above, the remote computing system may take various forms. For example, the remote computing system may be a second computing device in a second vehicle that is separate from the vehicle 100. The remote computing system may be within a threshold distance (e.g., within 100 m or 1000 m) from the vehicle 100 and may communicate via the wireless communication system 146 or the network 304. The remote computing system may be configured to provide data to the vehicle 100, such as the environmental attributes 512, the infrared image 522, the LIDAR / visible data 532, the previous thermal map 542, the emissivity map 552, the normalized map 562, and / or the projection map 572. In some embodiments, the data transmitted by the remote computing system to the vehicle 100 may be determined by the remote computing system within a threshold time limit (e.g., 1 millisecond, 1 second) from when the infrared module 520 generated the infrared image 522.

[0109] In other embodiments, if classification ML model 580 detects an object with a confidence that meets or exceeds a threshold, classification ML model 580 may still cause vehicle 100 to act on the detected object (e.g., stop if the object is identified with high confidence as a stop sign), but may be configured to request remote assistance at the same time (or after) vehicle 100 acts on the detected object.

[0110] 6A illustrates aspects of a thermal ML model 540, according to an example embodiment. In keeping with the above description, the environmental attributes 512 may take the form of input data 430, the thermal ML model 540 may take the form of a trained machine learning model 432, and the prior heat map 542 may take the form of inferences and / or predictions 450.

[0111] Environmental attributes 512 may represent physical properties associated with the current environment surrounding vehicle 100. These physical properties may include weather conditions 602 of the current environment, GPS 604 (which may take the form of GPS 122) information regarding the current position of vehicle 100 relative to the Earth, current ambient temperature 606 of the current environment, current solar position 608 of the sun relative to vehicle 100, and current road surface temperature 610 of the road beneath vehicle 100. In some embodiments, GPS 604 and the time of day may be used to estimate solar position 608 (e.g., vehicle 100 may be operable to determine the position of the sun based on the date and location). However, other environmental attributes 512 may also be considered.

[0112] The thermal ML model 540 may be configured to receive the environmental attributes 512 as an N-dimensional vector. For example, the environmental attributes 512 may be passed to the thermal ML model 540 as multiple values ​​X1 through XN (i.e., X1, X2, X3, X4, X5, X6, X7, X8, X9, and X10 through XN) in an N-dimensional vector space. The thermal ML model 540 may be trained to predict one or more previous heat maps captured under similar physical conditions as the environmental attributes 512. The previous heat maps may be captured by the vehicle 100 on the previous day or by another vehicle operating in the same environment as the vehicle 100 on the previous day. The thermal ML model 540 may be trained on several (e.g., 1,000, 10,000) pairs of (environmental attributes, heat maps), where the environmental attributes represent training features and the heat maps represent labels. In some cases, the thermal machine learning model 514 may be trained only on (environment attribute, thermal map) pairs collected during the night (e.g., as detailed by sun position 608).

[0113] In example embodiments, each previous heat map in previous heat maps 542 may be associated with (i) a probability value indicating the correlation of the previous heat map with environmental attribute 512, and (ii) a timestamp indicating the date the previous heat map was captured. In some examples, vehicle 100 may determine to utilize the most highly correlated previous heat map from previous heat maps 542 (e.g., any previous heat map with greater than 90% or 80% correlation). In other examples, vehicle 100 may determine to utilize all sufficiently correlated (e.g., greater than 70% correlation) previous heat maps captured within the past N days (e.g., 7 days, 14 days). Other manners for selecting a previous heat map may also be contemplated. It will be understood that other correlation values ​​and / or correlation thresholds are possible and contemplated within the scope of the present disclosure.

[0114] 6B illustrates aspects of an emissivity ML model 550, according to an example embodiment. Consistent with the above description, the LIDAR / visible data 532 may take the form of input data 430, the emissivity ML model 550 may take the form of a trained machine learning model 432, and the emissivity map 552 may take the form of inference and / or prediction 450. As shown in FIG. 6B, the LIDAR / visible data 532 may include LIDAR data 612 and visible light data 614.

[0115] As described above, LIDAR data 612 may represent electromagnetic signal radiant energy captured in the infrared spectrum from the environment surrounding vehicle 100. For example, laser rangefinder / LIDAR unit 128 may be configured to (i) transmit electromagnetic signals in the near-infrared spectrum that are reflected by various objects near vehicle 100 and (ii) capture the electromagnetic signals that reflect from the objects. Visible light data 614 may represent electromagnetic signal radiant energy captured in the visible light spectrum from the environment surrounding vehicle 100. For example, camera 130 may be configured to (i) transmit electromagnetic signals in the visible light spectrum that are reflected by various objects near vehicle 100 and (ii) capture the electromagnetic signals that reflect from the objects.

[0116] In some embodiments, the LIDAR data 612 / visible light data 614 may further include, as a parameter within the data, the pointing direction of the device used to capture the LIDAR data 612 / visible light data 614. That is, the device (e.g., laser rangefinder / LIDAR unit 128, camera 130, or another device) may be operable to discretize the environment surrounding vehicle 100 into a finite set of pointing directions that cover the 360° range of the device. Because the emissivity of an object may be angular, including the pointing direction may improve the predictions made by emissivity ML model 550.

[0117] Similar to the thermal ML model 540, the emissivity ML model 550 may be configured to receive the LIDAR / visible data 532 as an N-dimensional vector. For example, the LIDAR / visible data 532 may be passed to the emissivity ML model 550 as multiple values ​​X1 through XN (i.e., X1, X2, X3, X4, X5, X6, X7, X8, X9, and X10 through XN) in an N-dimensional vector space. After receiving the LIDAR / visible data 532, the emissivity ML model 550 can predict an emissivity map 552 corresponding to the scene captured by the LIDAR / visible data 532. The emissivity ML model 550 may be trained on several (e.g., 1,000, 10,000) pairs of (LIDAR / visible data, emissivity map), where the LIDAR / visible data represent training features and the emissivity map represent labels.

[0118] The emissivity ML model 550 can work in two phases. In the first phase, the emissivity ML model 550 can use the inverse correlation between reflectance and emissivity to convert the radiant energy values ​​in the LIDAR / visible data 532 into a predicted spectral distribution of emissivity values ​​for each object around the vehicle 100. In the second phase, the emissivity ML model 550 can predict the true emissivity spectrum for each object that best correlates with the predicted spectral distribution of emissivity values. Using the true emissivity spectrum, the emissivity ML model 550 can infer the emissivity values ​​for each object regardless of its wavelength. As a result, the emissivity ML model 550 can accordingly generate an emissivity map 552 having emissivity values ​​for each object that correspond to the same wavelengths at which the infrared module 520 operates.

[0119] IV. Method example 7 illustrates a method 700 according to an example embodiment. The operations of method 700 may be used in either vehicle 100 or 200, server computing system 306, remote computing system 302, or system 500. The operations may be performed, for example, by control system 106, computer system 112, or circuitry configured to perform the operations.

[0120] Block 710 includes receiving, at the computing device, one or more infrared images of the environment. In some embodiments, receiving one or more infrared images of the environment may include receiving multiple infrared images from an infrared camera, such as laser rangefinder / LIDAR 128, camera 130, or infrared module 520, via a wired or wireless communications link.

[0121] Block 720 includes applying, using a computing device, a trained machine learning system to the one or more infrared images to determine identified object types within the environment. Applying the trained machine learning system may involve determining one or more previous heat maps associated with the environment. Applying the trained machine learning system may further involve determining a current heat map associated with the environment using the one or more previous heat maps and the one or more infrared images. Applying the trained machine learning system may also involve determining identified object types based on the current heat map.

[0122] Block 730 includes providing the identified object type using a computing device. In some embodiments, providing the identified object type may include transmitting the identified object type to the vehicle 100 / 200 via a wired or wireless communications link. Other methods of providing the identified object type for use are possible and contemplated.

[0123] In some embodiments, the identified object type is a living thing. For example, the living thing may take the form of a pedestrian, a motorcyclist, a bicyclist, a dog, or a horse.

[0124] In some embodiments, determining the one or more previous thermal maps involves receiving a current thermal condition of the environment from a plurality of sensors communicatively coupled to the computing device, and determining the one or more previous thermal maps from a plurality of previous thermal maps stored on the computing device based on the current thermal condition.

[0125] In some embodiments, the plurality of sensors comprises at least one of an ambient temperature sensor, a GPS sensor, and a sun sensor operable to determine the current position of the sun.

[0126] In some embodiments, the computing device is integrated into the scanning laser system, and in such embodiments, each of the one or more previous thermal maps corresponds to an angle of the scanning laser system, and determining the one or more previous thermal maps is further based on a current angle of the scanning laser system.

[0127] In some embodiments, one or more previous heat maps include one or more regions that are modified, edited, or deleted based on the object type that was removed.

[0128] In some embodiments, determining a current thermal map associated with the environment involves calibrating the thermal contrast of one or more infrared images based on one or more previous thermal maps.

[0129] Some embodiments further involve determining a current spatial point cloud of the environment based on data received from the LIDAR system, identifying objects in the environment using the current spatial point cloud, and calculating emissivity values ​​for the identified objects, wherein determining the identified object type is further based on the calculated emissivity values.

[0130] In some embodiments, the current thermal map comprises a current spatial point cloud projected onto one or more infrared images.

[0131] Some embodiments further include determining a current visible light image based on data from the camera system, identifying objects in the environment using the current visible light image, and calculating emissivity values ​​for the identified objects, wherein determining the identified object type is further based on the calculated emissivity values.

[0132]

[0006] In some embodiments, the one or more thermal images include a set of temporally consecutive thermal images of the environment. In such embodiments, applying the trained machine learning system to the one or more thermal images involves spatiotemporal association of identified object types among the one or more thermal images.

[0133] In some embodiments, the spatiotemporal association includes assigning a unique identifier to each object of the identified object type in the environment.

[0134]

[0006] In some embodiments, the trained machine learning system comprises a convolutional neural network. In some embodiments, the convolutional neural network is a segmented network having an encoding pass and a decoding pass.

[0135] In some embodiments, the one or more previous thermal maps include a thermal map determined by a second computing device positioned at a different location in the environment than the computing device.

[0136] In some embodiments, the thermal map determined by the second computing device is determined within a threshold time limit from when the computing device receives the one or more infrared images.

[0137] 8 illustrates a method 800 according to an example embodiment. The operations of method 800 may be used in either vehicle 100 or 200, server computing system 306, remote computing system 302, or system 500. The operations may be performed, for example, by control system 106, computer system 112, or circuitry configured to perform the operations.

[0138] Block 810 involves receiving, at the computing device, one or more infrared images of the environment. In some embodiments, receiving one or more infrared images of the environment may include receiving multiple infrared images from an infrared camera, such as laser rangefinder / LIDAR 128, camera 130, or infrared module 520, via a wired or wireless communications link.

[0139] Block 820 involves using a computing device to train a machine learning system on one or more infrared images to determine identified object types in the environment. Training the machine learning system may involve training the machine learning system to determine one or more previous heat maps associated with the environment. Training the machine learning system may further involve training the machine learning system to determine a current heat map associated with the environment using the one or more previous heat maps and the one or more infrared images. Training the machine learning system may also involve training the machine learning system to determine identified object types using the current heat map.

[0140] Block 830 involves providing, using a computing device, the trained machine learning system. In some embodiments, providing the trained machine learning system may include transmitting the trained machine learning system to the vehicle 100 / 200 via a wired or wireless communications link. Other methods of providing a trained machine learning system for use are possible and contemplated.

[0141]

[0008] In some embodiments, the one or more thermal images include thermal images with spectral intensity information corresponding to a plurality of respective spectral wavelength bands. In these embodiments, training the machine learning system to determine a current thermal map associated with the environment includes calculating a ratio between the respective spectral intensity information to calculate the emissivity of objects in the environment.

[0142] The particular arrangement shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Also, some of the illustrated elements may be combined or omitted. Still further, example embodiments may include elements not shown in the figures.

[0143] Steps or blocks representing the processing of information may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, steps or blocks representing the processing of information may correspond to modules, segments, physical computers (e.g., field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs)), or portions of program code (including associated data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or operations in the methods or techniques. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device, including a disk, hard drive, or other storage medium.

[0144] Computer-readable media may also include non-transitory computer-readable media, such as register memory, processor cache, and random access memory (RAM), which store data for a short period of time. Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for a long period of time. Thus, computer-readable media may include, for example, secondary or long-term permanent storage, such as read-only memory (ROM), optical or magnetic disks, and compact disk read-only memory (CD-ROM). Computer-readable media may also be any other volatile or non-volatile storage system. Computer-readable media may be considered, for example, as computer-readable storage media or tangible storage devices.

[0145] While various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for purposes of illustration and are not intended to be limiting, the true scope of which is indicated by the following claims.

Claims

1. 1. A computer-implemented method comprising: receiving, at a computing device, one or more infrared images of an environment; using the computing device to apply a machine learning system trained to determine identified object types within the environment to the one or more thermal images, wherein determining includes at least: determining one or more previous thermal maps associated with the environment; determining a current thermal map associated with the environment using the one or more previous thermal maps and the one or more infrared images; and applying, performed by determining the identified object type based on the current thermal map; and providing, using the computing device, the identified object type.

2. The computer-implemented method of claim 1 , wherein the identified object type is a living thing.

3. determining the one or more previous heat maps receiving a current thermal condition of the environment from a plurality of sensors communicatively coupled to the computing device; and determining the one or more previous thermal maps from a plurality of previous thermal maps stored on the computing device based on the current thermal conditions.

4. 4. The computer-implemented method of claim 3, wherein the plurality of sensors comprises at least one of an ambient temperature sensor, a global positioning system (GPS) sensor, and a sun sensor operable to determine a current position of the sun.

5. 4. The computer-implemented method of claim 3, wherein the computing device is incorporated into a scanning laser system, and wherein each of the one or more previous thermal maps is associated with an angle of the scanning laser system, and determining the one or more previous thermal maps is further based on the current angle of the scanning laser system.

6. The computer-implemented method of claim 1 , wherein the one or more previous thermal maps include one or more regions that are modified, edited, or deleted based on the removed object type.

7. 2. The computer-implemented method of claim 1, wherein determining the current thermal map associated with the environment comprises calibrating thermal contrast of the one or more infrared images based on the one or more previous thermal maps.

8. determining a current spatial point cloud of the environment based on data received from a LIDAR system; using the current spatial point cloud to identify objects within the environment; 10. The computer-implemented method of claim 1, further comprising: calculating an emissivity value of the identified object, wherein determining the identified object type is further based on the calculated emissivity value.

9. The computer-implemented method of claim 8 , wherein a current thermal map comprises the current spatial point cloud projected onto the one or more infrared images.

10. determining a current visible light image based on data from the camera system; using the current visible light image to identify objects in the environment; 10. The computer-implemented method of claim 1, further comprising: calculating an emissivity value of the identified object, wherein determining the identified object type is further based on the calculated emissivity value.

11. 2. The computer-implemented method of claim 1, wherein the one or more infrared images comprise a set of temporally consecutive infrared images of the environment, and applying the trained machine learning system to the one or more infrared images comprises spatiotemporal association of the identified object types across the one or more infrared images.

12. The computer-implemented method of claim 11 , wherein the spatiotemporal association comprises assigning a unique identifier to each object of the identified object type in the environment.

13. 10. The computer-implemented method of claim 1, wherein the trained machine learning system comprises a convolutional neural network.

14. 14. The computer-implemented method of claim 13, wherein the convolutional neural network is a segmented network having an encoding pass and a decoding pass.

15. The computer-implemented method of claim 1 , wherein the one or more previous thermal maps include a thermal map determined by a second computing device positioned in the environment at a different location than the computing device.

16. 16. The computer-implemented method of claim 15, wherein the thermal map determined by the second computing device is determined within a threshold time limit from when the computing device received the one or more infrared images.

17. The computer-implemented method of claim 1 , wherein the computing device is part of an autonomous vehicle system.

18. 1. A computer-implemented method comprising: receiving, at a computing device, one or more infrared images of an environment; and training a machine learning system on the one or more infrared images using the computing device to determine identified object types within the environment, the machine learning system comprising at least: training the machine learning system to determine one or more prior heat maps associated with the environment; training the machine learning system using the one or more previous thermal maps and the one or more thermal images to determine a current thermal map associated with the environment; and training the machine learning system to determine the identified object type using the current thermal map; and providing the trained machine learning system using the computing device.

19. 20. The computer-implemented method of claim 18, wherein the one or more infrared images include infrared images with spectral intensity information corresponding to a plurality of respective spectral wavelength bands, and wherein training the machine learning system to determine the current thermal map associated with the environment includes calculating a ratio between the respective spectral intensity information to calculate the emissivity of objects in the environment.

20. 1. A computing device comprising: one or more processors; and a data storage area storing computer-executable instructions that, when executed by the one or more processors, cause the computing device to: receiving one or more infrared images taken of the environment; applying a machine learning system trained to determine identified object types within the environment to the one or more thermal images, wherein said determining includes at least: determining one or more previous thermal maps associated with the environment; determining a current thermal map associated with the environment using the one or more previous thermal maps and the one or more infrared images; and determining the identified object type using the current thermal map; and and providing the identified object type.

Images