Thermal Imaging for Autonomous Vehicles

Abstract

To utilize machine learning techniques to improve object classification in thermal imaging systems.SOLUTION: In an example embodiment, a method is provided. The method includes receiving, at a computing device, one or more infrared images of an environment. The method additionally includes, applying, using the computing device, a trained machine learning system to the one or more infrared images to determine an identified object type in the environment, where the determination is performed by at least: determining one or more prior thermal maps associated with the environment; using the one or more prior thermal maps and the one or more infrared images, determining a current thermal map associated with the environment; and determining the identified object type based on the current thermal map. The method also includes providing the identified object type using the computing device.SELECTED DRAWING: Figure 7

Description

Background Art

[0001] A thermal (or infrared) imaging device can be used to classify objects within a given environment. For example, a thermal imaging device can obtain radiometric information regarding a given object, and if the emissivity value of the given object can be estimated, this information can be utilized to determine the temperature of the given object. When multiple objects with estimated emissivity values are involved, the thermal imaging device can determine the temperature variation between the objects and then make predictions regarding the composition or type of the objects.

Summary of the Invention

[0002] This disclosure advantageously utilizes machine learning techniques to improve object classification in a thermal imaging system.

[0003] In a first aspect, a method is provided. The method includes receiving, at a computing device, one or more infrared images of an environment. The method further includes applying, using the computing device, a machine learning system trained to determine identified object types within the environment to the one or more infrared images. Applying the trained machine learning system includes determining one or more previous heat maps associated with the environment. Applying the trained machine learning system also includes using the one or more previous heat maps and the one or more infrared images to determine a current heat map associated with the environment and determining the identified object types based on the current heat map. The method also includes providing, using the computing device, the identified object types.

[0004] In a second aspect, a method is provided. The method includes receiving, at a computing device, one or more infrared images of an environment. The method further includes using the computing device to train a machine learning system in the one or more infrared images to determine an identified object type in the environment. Training the machine learning system includes training the machine learning system to determine one or more previous heat maps associated with the environment. Training the machine learning system includes training the machine learning system to determine a current heat map associated with the environment using the one or more previous heat maps and the one or more infrared images, and training the machine learning system to determine the identified object type using the current heat map. The method also includes using the computing device to provide a trained machine learning system.

[0005] In a third aspect, a computing device is provided. The computing device includes one or more processors and a data storage. The data storage stores computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform operations. The operations include receiving one or more infrared images captured from an environment. The operations also include applying a trained machine learning system to the one or more infrared images to determine an identified object type in the environment. Applying the trained machine learning system includes determining one or more previous heat maps associated with the environment. Applying the trained machine learning system includes determining a current heat map associated with the environment using the one or more previous heat maps and the one or more infrared images, and determining the identified object type based on the current heat map. The operations further include providing the identified object type.

[0006] Other aspects, embodiments, and implementations will be apparent to those of ordinary skill in the art upon reading the following detailed description with reference to the accompanying drawings as appropriate.

Brief Description of Drawings

[0007]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 3

Figure 4

Figure 5

Figure 6A

Figure 6B

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0008] Examples of methods, devices, and systems are described herein. It should be understood that the terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or feature described herein as "example" or "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or features. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0009] Thus, the example embodiments described herein are not meant to be limiting. The aspects of the disclosure generally described herein and illustrated in the figures can be arranged, replaced, combined, separated, and designed in a variety of different configurations, and all of these configurations are contemplated herein.

[0010] Furthermore, unless the context indicates otherwise, the features shown in each of these figures can be used in combination with one another. Thus, the figures should generally be considered as aspects of the components of one or more embodiments, with the understanding that not all of the features shown are necessarily required for each embodiment.

[0011] I. Overview Conventional sensor systems tend to misidentify and / or misclassify objects within a given environment. Such performance degradation can particularly apply to identification techniques that utilize thermal imaging. In such cases, large temperature variations within the environment can make it difficult for a thermal imaging system to detect an object. For example, due to daily and / or seasonal variations in the environment, the appearance of the object of interest can change, thermal noise can be introduced, etc. Such variations limit the ability to reliably train a machine learning model to classify an object based on a thermal image (or, as referred to herein, a "thermal map") obtained from a thermal imaging system. Some solutions attempt to normalize the variations with a radiometric calibration camera that estimates the temperature within the environment, but the accuracy of these estimates is limited by one or more assumptions or approximations regarding the emissivity of the objects within the given environment.

[0012] The embodiments described herein can enhance the performance of a thermal imaging system to correctly identify and / or classify an object within a given environment. Specifically, the disclosure herein provides a machine learning system that can accurately predict the emissivity and / or temperature of an object within the environment. Accordingly, the classification of such objects within the environment can be improved.

[0013] Depending on the embodiment, the machine learning system can include a model trained in visible / near-infrared reflectivity, two-color infrared radiation, LIDAR data, and / or ambient temperature / sun position / weather data to infer the emissivity of an object within the environment in order to more accurately estimate the physical temperature of such an object.

[0014] To further improve classification, the machine learning system can compare the thermal image at that time with the previous thermal image processed to remove organisms. Also, the systems and methods herein can detect and track moving organisms by combining LIDAR data with thermal images and associating points of objects between various thermal image frames. Other embodiments, aspects, and improvements may be possible.

[0015] II. System Example Here, a system example within the scope of the present disclosure will be described in more detail. One system example can be implemented in or take the form of an automobile. However, one system example can also be implemented in or take the form of other vehicles such as cars, trucks, motorcycles, buses, boats, airplanes, helicopters, lawn mowers, bulldozers, boats, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, agricultural implements, construction machinery, trams, golf carts, trains, trolleys, robotic devices, etc. Other vehicles are also conceivable. Further, depending on the embodiment, the system example may not include a vehicle.

[0016] Referring now to the figures, FIG. 1 is a functional block diagram showing a vehicle example 100 according to an example embodiment that can be configured to operate fully or partially in autonomous mode. More specifically, vehicle 100 can operate in autonomous mode without human interaction by receiving control instructions from a computing system. As part of the operation in autonomous mode, vehicle 100 can use sensors to detect and possibly identify objects in the surrounding environment to enable safe navigation. Depending on the embodiment, vehicle 100 may also include a subsystem that allows a driver to control the operation of vehicle 100.

[0017] As shown in FIG. 1, vehicle 100 may include various subsystems such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power source 110, a computer system 112 (which may also be referred to as a computing system), a data storage area 114, and a user interface 116. In other examples, vehicle 100 may include more or fewer subsystems, each of which may include a number of elements. The subsystems and components of vehicle 100 may be interconnected in various ways.

[0018] Propulsion system 102 may include one or more components operable to provide motive movement to vehicle 100 and may include, among other possible components, engine / motor 118, energy source 119, transmission 120, and wheels / tires 121. For example, engine / motor 118 may be configured to convert energy source 119 into mechanical energy and may correspond to, among other possible options, one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine. For example, in some embodiments, propulsion system 102 may include multiple types of engines and / or motors such as a gasoline engine and an electric motor.

[0019] Energy source 119 represents an energy source that can fully or partially power one or more systems of vehicle 100 (e.g., engine / motor 118). For example, energy source 119 can correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, energy source 119 may include a combination of a fuel tank, a battery, a capacitor, and / or a flywheel.

[0020] The transmission 120 can transmit the mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. Thus, the transmission 120 can include, among other possible components, a gearbox, a clutch, a differential, and a drive shaft. The drive shaft can include an axle that connects to one or more wheels / tires 121.

[0021] The wheels / tires 121 of the vehicle 100 can have various configurations within the exemplary embodiments. For example, the vehicle 100 can exist in, among other possible configurations, a unicycle, a bicycle / motorcycle, a tricycle, or a car / truck four-wheel format. Thus, the wheels / tires 121 can be connected to the vehicle 100 in various ways and can exist in different materials such as metal and rubber.

[0022] The sensor system 104 can include various types of sensors, among other possible sensors, such as a global positioning system (GPS) 122, an inertial measurement unit (IMU) 124, a radar 126, a laser range finder / LIDAR 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125. Depending on the embodiment, the sensor system 104 can include sensors configured to monitor the internal systems of the vehicle 100 (e.g., an O2 monitor, a fuel gauge, an engine oil temperature, a brake wear).

[0023] The GPS 122 can include a transceiver operable to provide information regarding the position of the vehicle 100 relative to the earth. The IMU 124 can have a configuration that uses one or more accelerometers and / or gyroscopes and can sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 can detect the pitch and roll of the vehicle 100 while the vehicle 100 is stationary or moving.

[0024] Radar 126 may represent one or more systems configured to use radio wave signals to sense objects in the local environment of vehicle 100, including the speed and direction of travel of the objects. Thus, radar 126 may include an antenna configured to transmit and receive radio wave signals. In some embodiments, radar 126 may correspond to a wearable radar system configured to obtain measurements of the surrounding environment of vehicle 100.

[0025] Laser rangefinder / LIDAR 128 may include, among other system components, one or more laser sources, a laser scanner, and one or more detectors, and may operate in a coherent mode (e.g., using heterodyne detection) or a non-coherent detection mode. In some embodiments, one or more detectors of laser rangefinder / LIDAR 128 may include one or more light-receiving elements. Such light-receiving elements may be, in particular, highly sensitive detectors (e.g., avalanche photodiodes (APDs)). By way of example, such light-receiving elements may even be capable of detecting single photons (e.g., single photon avalanche diodes (SPADs)). Also, such light-receiving elements may be arranged in an array (e.g., as seen in silicon photomultipliers (SiPMs)).

[0026] Camera 130 may include one or more devices (e.g., a still camera or a video camera) configured to capture an image of the environment of vehicle 100.

[0027] Steering sensor 123 may sense the steering angle of vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representative of the angle of the steering wheel. In some embodiments, steering sensor 123 may measure the angle of the wheels of vehicle 100, such as by detecting the angle of the wheels relative to the front axle of vehicle 100. Steering sensor 123 may also be configured to measure a combination (or subset) of the angle of the steering wheel, an electrical signal representative of the angle of the steering wheel, and the angle of the wheels of vehicle 100.

[0028] The throttle / brake sensor 125 can detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angles of both the accelerator pedal (throttle) and the brake pedal, or, for example, measure an electrical signal that can represent the angle of the accelerator pedal (throttle) and / or the angle of the brake pedal. The throttle / brake sensor 125 may also measure the angle of the throttle body of the vehicle 100, which may include a part of the physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., a butterfly valve or a carburetor). In addition, the throttle / brake sensor 125 may measure the pressure of one or more brake pads applied to the rotor of the vehicle 100, or a combination (or subset) of the angles of the accelerator pedal (throttle) and the brake pedal, the electrical signals representing the angles of the accelerator pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied by at least one brake pad to the rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure the pressure applied to a vehicle pedal such as the throttle or brake pedal.

[0029] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a brake unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / route finding system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the direction of travel of the vehicle 100, the throttle 134 may control the operating speed of the engine / motor 118 to control the acceleration of the vehicle 100. The brake unit 136 can decelerate the vehicle 100, which may include using friction to decelerate the wheels / tires 121. In some embodiments, the brake unit 136 may convert the kinetic energy of the wheels / tires 121 into an electric current for subsequent use by one or more systems of the vehicle 100.

[0030] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithms capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide an assessment based on incoming sensor data, such as an assessment of individual objects and / or features, an assessment of a particular situation, and / or an assessment of possible effects within a given situation.

[0031] The computer vision system 140 may include hardware and software operable to process and analyze images when attempting to determine objects, environmental objects (e.g., traffic signals, lane boundaries, etc.), and obstacles. Thus, the computer vision system 140 may use object recognition, Structure From Motion (SFM), video tracking, and other algorithms used in computer vision, for example, to recognize objects, map the environment, track objects, and estimate the speed of objects.

[0032] The navigation / route finding system 142 can determine the driving route of the vehicle 100, which may include dynamically adjusting the navigation during operation. Thus, the navigation / route finding system 142 can use data from, among other information sources, the sensor fusion algorithm 138, the GPS 122, and a map to navigate the vehicle 100. The obstacle avoidance system 144 can evaluate what may be an obstacle based on sensor data and cause the vehicle 100's system to avoid or otherwise navigate around what may be an obstacle.

[0033] As shown in FIG. 1, vehicle 100 may also include peripheral devices 108 such as a wireless communication system 146, a touch screen 148, a microphone 150, and / or a speaker 152. The peripheral devices 108 may provide a control device or other elements for a user to interact with the user interface 116. For example, the touch screen 148 may provide information to a user of the vehicle 100. The user interface 116 may also receive input from the user via the touch screen 148. The peripheral devices 108 may also enable the vehicle 100 to communicate with devices such as devices of other vehicles.

[0034] The wireless communication system 146 may communicate wirelessly with one or more devices directly or via a communication network. For example, in the wireless communication system 146, 3G cellular communication such as code division multiple access (CDMA), evolution data optimized (EVDO), global system for mobile communications (GSM) / general packet radio service (GPRS), or 4G cellular communication such as worldwide interoperability for microwave access (WiMAX) or long term evolution (LTE) may be used. Alternatively, the wireless communication system 146 may communicate with a wireless local area network (WLAN) using WiFi or other possible connections. The wireless communication system 146 may also communicate directly with devices using, for example, an infrared link, BLUETOOTH (registered trademark), or ZigBee (registered trademark). Other wireless protocols such as various vehicle communication systems are possible within the context of the present disclosure. For example, the wireless communication system 146 may include one or more dedicated short range communication (DSRC) devices that may include public data communication and / or private data communication between the vehicle and / or a fueling station along the road.

[0035] Vehicle 100 may include a power source 110 for supplying power to components. In some embodiments, the power source 110 may include a rechargeable lithium-ion or lead battery. For example, the power source 110 may include one or more batteries configured to provide power. Vehicle 100 may also use other types of power sources. In an exemplary embodiment, the power source 110 and the energy source 119 may be integrated into a single energy source.

[0036] Vehicle 100 may also include a computer system 112 for performing operations such as those described herein. Thus, the computer system 112 may include at least one processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored on a non-transitory computer-readable medium such as the data storage 114. In some embodiments, the computer system 112 may represent a plurality of computing devices capable of functioning to distributively control individual components or subsystems of Vehicle 100.

[0037] In some embodiments, the data storage 114 may include instructions 115 (e.g., program logic) executable by the processor 113 for performing various functions of Vehicle 100, including those described above in connection with FIG. 1. The data storage 114 may also include additional instructions for transmitting data to, receiving data from, interacting with, and / or controlling one or more of the propulsion system 102, the sensor system 104, the control system 106, and the peripheral devices 108.

[0038] In addition to the instructions 115, the data storage 114 may store data such as, among other things, road maps, route information, etc. Such information may be used by Vehicle 100 and the computer system 112 during operation of Vehicle 100 in autonomous mode, semi-autonomous mode, and / or manual mode.

[0039] Vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of vehicle 100. The user interface 116 may be able to control or enable the layout of content and / or interactive images that may be displayed on the touch screen 148. Further, the user interface 116 may include one or more input / output devices within a set of peripheral devices 108 such as a wireless communication system 146, a touch screen 148, a microphone 150, and a speaker 152.

[0040] Computer system 112 may control the functions of vehicle 100 based on inputs received from various subsystems (e.g., propulsion system 102, sensor system 104, and control system 106) and also from the user interface 116. For example, computer system 112 may utilize inputs from sensor system 104 to estimate the outputs generated by propulsion system 102 and control system 106. In some embodiments, computer system 112 may be operable to monitor many aspects of vehicle 100 and its subsystems. In some embodiments, computer system 112 may disable some or all of the functions of vehicle 100 based on signals received from sensor system 104.

[0041] The components of vehicle 100 can be configured to operate in an interconnected state with other components within or outside of their respective systems. For example, in one embodiment, camera 130 can capture a plurality of images that can represent information about the state of the environment of vehicle 100 operating in autonomous mode. The state of the environment can include parameters of the road on which the vehicle is operating. For example, computer vision system 140 may be able to recognize inclinations (gradients) or other features based on a plurality of images of the road. Additionally, the combination of GPS 122 and the features recognized by computer vision system 140 can be used together with the map data stored in data storage 114 to determine specific road parameters. Further, radar 126 can also provide information about the surroundings of the vehicle.

[0042] In other words, a combination of various sensors (which may be referred to as input display sensors and output display sensors) and computer system 112 can interact with each other to provide a display of the input provided to control the vehicle or a display of the surroundings of the vehicle.

[0043] In some embodiments, computer system 112 can make determinations regarding various objects based on data provided by other systems. For example, vehicle 100 may have a laser or other optical sensor configured to sense objects within the field of view of the vehicle. Computer system 112 can use the outputs from the various sensors to determine information about the objects within the field of view of the vehicle and can determine the distance and direction information to the various objects. Computer system 112 can also determine whether an object is desirable or undesirable based on the outputs from the various sensors.

[0044] FIG. 1 shows various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) as being integrated into vehicle 100, although one or more of these components can be mounted or associated separately from vehicle 100. For example, data storage 114 may exist partially or completely separately from vehicle 100. Thus, vehicle 100 can be provided in the form of device elements that can be arranged separately or together. The device elements that make up vehicle 100 can be communicatively coupled together in wired and / or wireless manners.

[0045] FIGS. 2A-2E show vehicle example 200, which may include some or all of the functions described in connection with vehicle 100 with reference to FIG. 1 according to an example embodiment. Vehicle 200 is shown in FIGS. 2A-2E as a van for purposes of illustration, but the present disclosure is not so limited. For example, vehicle 200 may represent a truck, passenger car, semi-trailer truck, motorcycle, golf cart, off-road vehicle, or agricultural vehicle, among others.

[0046] Vehicle example 200 includes sensor unit 202, first LIDAR unit 204, second LIDAR unit 206, first radar unit 208, second radar unit 210, first LIDAR / radar unit 212, second LIDAR / radar unit 214, and also includes two additional locations 216, 218 on vehicle 200 where one or more sensors of a radar unit, LIDAR unit, laser distance measurement unit, and / or other types may be located. Each of the first LIDAR / radar unit 212 and the second LIDAR / radar unit 214 can take the form of a LIDAR unit, a radar unit, or both.

[0047] Furthermore, vehicle example 200 may include any of the components described in relation to vehicle 100 of FIG. 1. The first and second radar units 208, 210 and / or the first and second LIDAR units 204, 206 can actively scan their surrounding environment to check for any potential obstacles and can be similar to radar 126 and / or laser rangefinder / LIDAR 128 of vehicle 100.

[0048] Sensor unit 202 is mounted on top of vehicle 200 and includes one or more sensors configured to detect information about the environment surrounding vehicle 200 and output a display of the information. For example, sensor unit 202 may include any combination of cameras, radars, LIDARs, rangefinders, and acoustic sensors. Sensor unit 202 may include one or more movable mounts operable to adjust the orientation of one or more sensors within sensor unit 202. In one embodiment, the movable mount may include a rotating platform capable of scanning sensors to obtain information from each direction around vehicle 200. In another embodiment, the movable mount of sensor unit 202 may be scan-movably within a specific range of angles and / or azimuths. Although other mounting locations are possible, sensor unit 202 may be mounted on the roof of vehicle 200.

[0049] In addition, the sensors of sensor unit 202 may be dispersed in various locations and do not need to be co-located in one place. Two additional locations 216, 218 are included in some possible sensor types and mounting locations. Furthermore, each sensor of sensor unit 202 may be configured to move or scan independently of the other sensors of sensor unit 202.

[0050] In one configuration example, one or more radar scanners (e.g., first and second radar units 208, 210) may be located near the rear of vehicle 200 to actively scan the environment near the rear of vehicle 200 for the presence of radio wave reflecting objects. Similarly, a first LIDAR / radar unit 212 and a second LIDAR / radar unit 214 may be attached near the front of vehicle 200 to actively scan the environment near the front of vehicle 200. The radar scanner may be positioned in a location suitable for illuminating an area including the forward path of vehicle 200, for example, without being blocked by other features of vehicle 200. For example, the radar scanner may be embedded in and / or attached to or near a front bumper, front headlight, cowl, and / or hood. Also, one or more additional radar scanning devices may be positioned to actively scan the sides and / or rear of vehicle 200 for the presence of radio wave reflecting objects, such as by including such devices in or near a rear bumper, side panel, rocker panel, and / or underbody of vehicle 200.

[0051] Although not shown in FIGS. 2A - 2E, vehicle 200 may include a wireless communication system. The wireless communication system may include a wireless transmitter and a wireless receiver configured to communicate with devices external or internal to vehicle 200. Specifically, the wireless communication system may include, for example, a transceiver configured to communicate with other vehicles and / or computing devices in a vehicle communication system or a road station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transport systems.

[0052] Vehicle 200 may optionally include a camera in a location inside sensor unit 202. The camera can be a photosensitive device such as a still camera or a video camera configured to capture a plurality of images of the environment of vehicle 200. For this purpose, the camera can be configured to detect visible light and additionally or alternatively can be configured to detect light from other parts of the electromagnetic spectrum such as infrared or ultraviolet light. The camera can be a two-dimensional detector and can optionally have a three-dimensional spatial range of sensitivity.

[0053] In some embodiments, the camera can include a range detector configured to generate a two-dimensional image indicating, for example, the distance from the camera to several points in the environment. For this purpose, the camera can use one or more range detection techniques. For example, the camera can provide range information by using a structured light technique in which vehicle 200 illuminates an object in the environment with a predetermined light pattern such as a grid or checkerboard pattern and also uses the camera to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, vehicle 200 can determine the distance to points on the object.

[0054] The predetermined light pattern can be composed of infrared light or radiation of other wavelengths suitable for such measurements. In some examples, the camera can be mounted inside the front windshield of vehicle 200. Specifically, the camera can be positioned to capture images from a forward view with respect to the orientation of vehicle 200. Other mounting locations and field of view angles of the camera can also be used either inside or outside vehicle 200. Also, the camera can have associated optical elements operable to provide an adjustable field of view. Furthermore, the camera can be attached to vehicle 200 using a movable mount to vary the pointing angle of the camera, such as via a pan / tilt mechanism.

[0055] The control system of vehicle 200 can be configured to control vehicle 200 according to a certain control strategy selected from a number of possible control strategies. The control system can receive information (whether vehicle 200 is on or off) from sensors coupled to vehicle 200, modify the control strategy (and related driving behavior) based on that information, and be configured to control vehicle 200 according to the modified control strategy. The control system can further be configured to monitor the information received from the sensors and continuously evaluate the driving state, and can also be configured to modify the control strategy and driving behavior based on changes in the driving state.

[0056] FIG. 3 shows wireless communication between various computing systems related to a vehicle according to an exemplary embodiment. Specifically, the wireless communication can occur between remote computing system 302 and vehicle 200 via network 304. The wireless communication can also occur between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0057] Remote computing system 302 can represent any type of device related to remote assistance techniques, including but not limited to those described herein. Among the examples, remote computing system 302 can be configured to (i) receive information related to vehicle 200, (ii) provide an interface through which a human operator can then become aware of the information and input a response related to the information, and (iii) transmit the response to vehicle 200 or to other devices. Remote computing system 302 can take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some examples, remote computing system 302 can include a number of computing devices operating together in a network configuration.

[0058] The remote computing system 302 may include one or more subsystems and components that are similar or identical to the subsystems and components of vehicle 100 or vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface that includes input / output devices such as a touch screen and speakers. Other examples are possible.

[0059] Network 304 represents an infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0060] The location of remote computing system 302 can vary within the scope of the example. For example, remote computing system 302 can have a remote location from vehicle 200 having wireless communication via network 304. In another example, remote computing system 302 can correspond to a computing device within vehicle 200 that is separate from vehicle 200 but where a human operator can interact with a passenger or driver of vehicle 200. In some examples, remote computing system 302 can be a computing device having a touch screen operable by a passenger of vehicle 200.

[0061] In some embodiments, the operations described herein performed by remote computing system 302 can additionally or alternatively be performed by vehicle 200 (i.e., by any system or subsystem of vehicle 200). In other words, vehicle 200 can be configured to provide a remote assistance mechanism with which a driver or passenger of the vehicle can interact.

[0062] The server computing system 306 may be configured to wirelessly communicate with the remote computing system 302 and the vehicle 200 via the network 304 (or, in some cases, directly with the remote computing system 302 and / or the vehicle 200). The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information regarding the vehicle 200 and its remote assistance. Thus, the server computing system 306 may be configured to perform any operation or portion of such operations described herein as being performed by the remote computing system 302 and / or the vehicle 200. In some embodiments of the wireless communication related to remote assistance, the server computing system 306 can be utilized, while in other embodiments it cannot be utilized.

[0063] The server computing system 306 may include one or more subsystems and components similar or identical to those of the remote computing system 302 and / or the vehicle 200, such as a processor configured to perform the various operations described herein, and a wireless communication interface for receiving information from and providing information to the remote computing system 302 and the vehicle 200.

[0064] In accordance with the above description, a computing system (e.g., the remote computing system 302, the server computing system 306, or a computing system local to the vehicle 200) can be operative to use a camera to capture an image of the environment of the autonomous vehicle. Generally, at least one computing system can analyze the image and, in some cases, control the autonomous vehicle.

[0065] Depending on the embodiment, in order to facilitate autonomous operation, a vehicle (e.g., vehicle 200) may receive data representing objects in the environment in which the vehicle operates (also referred to herein as "environmental data") in various ways. The vehicle's sensor system may provide environmental data representing objects in the environment. For example, the vehicle may have various sensors including cameras, radar units, lidar, microphones, radio units, and other sensors. Each of these sensors may convey environmental data regarding the information received by each respective sensor to a processor within the vehicle.

[0066] While operating in autonomous mode, the vehicle may control its operation with little or no human input. For example, when a human operator inputs an address into the vehicle, the vehicle may be able to drive to the specified destination without further input from the human (e.g., without the need for the human to operate or touch the brake / accelerator pedal). Further, while the vehicle is operating autonomously, the sensor system may be receiving environmental data. The vehicle's processing system may change the control of the vehicle based on the environmental data received from various sensors. By way of example, the vehicle may change its speed in response to environmental data from various sensors. The vehicle may change its speed to avoid obstacles, comply with traffic laws, etc. When the processing system within the vehicle identifies an object near the vehicle, the vehicle may be able to change its speed or move in another way.

[0067] If the vehicle detects an object but is not very confident in such object detection, the vehicle can request a human operator (or a more powerful computer) to perform one or more remote assistance tasks such as (i) verifying the fact whether the object is in the environment (e.g., whether there is actually a stop sign or there is actually no stop sign), (ii) verifying whether the vehicle's identification of the object is correct, (iii) correcting the identification if it is incorrect, and / or (iv) giving supplementary instructions to the autonomous vehicle (or modifying the instructions at that time). The remote assistance tasks may also include providing instructions for a human operator to control the operation of the vehicle (e.g., when the human operator determines that the object is a stop sign, the human operator instructs the vehicle to stop at the stop sign). However, in some cases, the vehicle itself may control its own operation based on the feedback of the human operator regarding the identification of the object.

[0068] To facilitate this, the vehicle can analyze environmental data representing the objects in the environment and determine at least one object having a detection reliability below a threshold. The processor of the vehicle can be configured to detect various objects in the environment based on environmental data from various sensors. For example, in one embodiment, the processor can be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, road signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0069] The detection reliability can indicate the possibility that the determined object is correctly identified or exists in the environment. For example, the processor can perform object detection of an object in the image data in the received environmental data and, based on the inability to identify an object having a detection reliability exceeding a threshold, determine that at least one object has a detection reliability below the threshold. If the result of object detection or object recognition of an object is not conclusive, the detection reliability may be low or may be below a set threshold.

[0070] The vehicle can detect environmental objects in various ways depending on the source of the environmental data. In some embodiments, the environmental data may come from a camera and be image or video data. In other embodiments, the environmental data may come from a LIDAR unit. The vehicle can analyze the captured image or video data to identify the objects within the image or video data. The method and apparatus can be configured to monitor the image and / or video data for the presence of environmental objects. In other embodiments, the environmental data can be radar, audio, or other data. The vehicle can be configured to identify environmental objects based on the radar, audio, or other data.

[0071] III. Example Machine Learning System FIG. 4 shows a system 400 that depicts a learning phase 402 and an inference phase 404 of a trained machine learning model 432 according to an example embodiment. Some machine learning techniques involve training one or more machine learning systems on an input set of training data to recognize patterns in the training data and provide output inferences and / or predictions regarding the (patterns in the) training data. The resulting trained machine learning model may sometimes be referred to as a trained machine learning system or a trained machine learning model. For example, FIG. 4 shows the learning phase 402 in which one or more machine learning systems 420 are trained on training data 410 to become one or more trained machine learning models 432. Next, during the inference phase 404, the trained machine learning model 432 can receive input data 430 and one or more inference / prediction requests 440 (presumably as part of the input data 430) and, in response, provide one or more inferences and / or predictions 450 as output.

[0072] The machine learning system 420 may include, but is not limited to, an artificial neural network (e.g., a convolutional neural network described herein that uses the segmentation techniques described herein, a recurrent neural network), a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system. During the learning phase 402, the machine learning system 420 can be trained by using training techniques such as, but not limited to, unsupervised, supervised, semi-supervised, reinforcement learning, transfer learning, incremental learning, and / or curriculum learning techniques, by providing at least training data 410 as training input.

[0073] Unsupervised learning involves providing a portion (or all) of the training data 410 to the machine learning system 420. Thereby, the machine learning system 420 can determine one or more output inferences based on the provided portion (or all) of the training data 410. Supervised learning may involve providing a portion of the training data 410 to the machine learning system 420, and the machine learning system 420 determines one or more output inferences based on the provided portion of the training data 410, and the output inferences are accepted or corrected based on the correct results associated with the training data 410. In some examples, supervised learning of the machine learning system 420 can be controlled by a set of rules and / or a set of labels in the training input, and the set of rules and / or the set of labels can be used to correct the inferences of the machine learning system 420.

[0074] Semi-supervised learning may involve obtaining correct results (e.g., partially labeled data) for only a part, and not necessarily all, of the training data 410. During semi-supervised learning, supervised learning is used for a portion of the training data 410 that has the correct results, and unsupervised learning is used for a portion of the training data 410 for which the correct results were not obtained.

[0075] Reinforcement learning involves the machine learning system 420 receiving a reward signal regarding a previous inference, where the reward signal can be a numerical value. During reinforcement learning, the machine learning system 420 can output an inference and receive a reward signal in response, and the machine learning system 420 is configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing the expected total of the numerical values provided by the reward signal over time.

[0076] Transfer learning techniques can include the trained machine learning model 432 being pre-trained on one dataset and further trained using training data 410. More specifically, the machine learning system 420 can be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to a computing device CD1, which is intended to execute the trained machine learning model during the inference phase 404. Next, during the learning phase 402, the pre-trained machine learning model can be further trained using the training data 410, where the training data 410 can be derived from the kernel and non-kernel data of the computing device CD1. This further training of the machine learning system 420 and / or the pre-trained machine learning model using the training data 410 of CD1 can be performed using either supervised learning or unsupervised learning. Once the machine learning system 420 and / or the pre-trained machine learning model is trained at least on the training data 410, the learning phase 402 can be completed. The resulting trained machine learning model can be utilized as at least one of the trained machine learning models 432.

[0077] Incremental learning techniques can include providing input data to a trained machine learning model 432 (and perhaps a machine learning system 420) to continuously expand the knowledge of the trained machine learning model 432. Curriculum learning techniques can include providing the machine learning system 420 with training data arranged in a particular order, such as first providing relatively easy training examples and gradually progressing to more difficult training examples, similar to a school curriculum or learning course. Other techniques can be contemplated for training the machine learning system 420 and / or the trained machine learning model 432.

[0078] In some examples, after the learning phase 402 is complete but before the inference phase 404 begins, the trained machine learning model 432 can be provided to a computing device CD1 on which the trained machine learning model 432 does not yet reside. For example, after the learning phase 402 is complete, the trained machine learning model 432 can be downloaded to the computing device CD1.

[0079] For example, a computing device CD2 storing a trained machine learning model 432 can provide the trained machine learning model 432 to a computing device CD1 by one or more of: transmitting a copy of the trained machine learning model 432 to the computing device CD1, creating a copy of the trained machine learning model 432 for the computing device CD1, providing access to the trained machine learning model 432 to the computing device CD1, and / or otherwise providing the trained machine learning system to the computing device CD1. In some examples, the trained machine learning model 432 can be used by the computing device CD1 immediately after being provided by the computing device CD2. In some examples, after the trained machine learning model 432 is provided to the computing device CD1, the trained machine learning model 432 can be installed and / or otherwise prepared for use before it can be used by the computing device CD1.

[0080] During the inference phase 404, the trained machine learning model 432 can receive input data 430 and generate and output corresponding inferences and / or predictions 450 for the input data 430. Thus, the input data 430 can be used as an input to the trained machine learning model 432 to provide corresponding inferences and / or predictions 450 to kernel and non-kernel components. For example, the trained machine learning model 432 can generate inferences and / or predictions 450 in response to an inference / prediction request 440. In some examples, the trained machine learning model 432 can be executed by a portion of other software. For example, the trained machine learning model 432 can be executed by an inference or prediction daemon to be immediately available for providing inferences and / or predictions in response to requests. The input data 430 can include data from the computing device CD1 executing the trained machine learning model 432 and / or input data from one or more computing devices other than CD1.

[0081] In some cases, the input data 430 may include data from the environment in which the vehicle 100 / 200 operates. As described above, the vehicle 100 / 200 may have various sensors including cameras, radar units, lidar, microphones, radio units, and other sensors. Each of these sensors can convey this environmental data as an input to the trained machine learning model 432. Note that other types of input data are also conceivable.

[0082] The inference and / or prediction 450 may include output images, heat maps, depth maps, numerical values, and / or other output data provided by the trained machine learning model 432 operating on the input data 430 (and the training data 410). In some cases, the trained machine learning model 432 can use the output inference and / or prediction 450 as the input feedback 460. The trained machine learning model 432 can also utilize past inferences as inputs for generating new inferences.

[0083] In some examples, the machine learning system 420 and / or the trained machine learning model 432 may be executed and / or accelerated using one or more computer processors and / or on-device coprocessors. On-device coprocessors may include, but are not limited to, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more digital signal processors (DSPs), and / or one or more application-specific integrated circuits (ASICs). Such on-device coprocessors can speed up the training of the machine learning system 420, as well as the inference and / or prediction 450 generated by the trained machine learning model 432. In some examples, the trained machine learning model 432 is trained, resident, and executable to provide inference and / or prediction 450 on a particular computing device, or otherwise make inferences for a particular computing device.

[0084] In some examples, a single computing device CD_SOLO may include the trained learning model 432, perhaps after training the machine learning system 420 on the computing device CD_SOLO. Thereby, the computing device CD_SOLO can receive an inference / prediction request 440 seeking to provide inference and / or prediction 450, and use the trained machine learning model 432 operating on the input data 430 to provide inference and / or prediction 450 in response, where the inference and / or prediction 450 can be provided using a user interface and / or display, such as as one or more electronic communications, as one or more printed documents, etc.

[0085] In some examples, two or more computing devices, CD_CLI and CD_SRV, can be used to provide inferences and / or predictions 450. For example, a first computing device, CD_CLI, can generate an inference / prediction request 440 and send it to a second computing device, CD_SRV. Upon receiving the inference / prediction request 440 from CD_CLI, CD_SRV can operate on input data 430 using a trained machine learning model 432, perhaps after training a machine learning system 420, and accordingly determine inferences and / or predictions 450. After determining the inferences and / or predictions 450, CD_SRV responds to the request from CD_CLI by providing the inferences and / or predictions 450 to CD_CLI.

[0086] Note that in the above example, CD1, CD2, CD_SOLO, CD_CLI, and CD_SRV can take the form of vehicle 100 or another similar vehicle.

[0087] As described above, vehicle 100 can detect objects in its surrounding environment in various ways. In some embodiments, vehicle 100 can actively transmit electromagnetic signals (e.g., near-infrared light) reflected by target objects near the vehicle 100 using a thermal imaging system. The thermal imaging system can use detector elements to capture the reflected electromagnetic signals. In other embodiments, the thermal imaging system can passively capture infrared radiation (e.g., mid-infrared light or far-infrared light) emitted as heat from target objects near the vehicle 100. The captured reflected electromagnetic signals and / or the captured infrared radiation can enable vehicle 100 to classify objects in its surrounding environment using thermography techniques (e.g., classifying an object as biological or inanimate based on the temperature variation between objects).

[0088] However, the ability of vehicle 100 to accurately classify objects can be limited by daily / seasonal variations in its surrounding environment. Factors such as the amount of sunlight and the ambient temperature within the environment can introduce noise into the captured reflected electromagnetic signals or the captured infrared radiation. Also, since thermography typically uses the temperature contrast between an object and its background environment, changes in the surrounding environment can make it difficult to apply thermography to correctly distinguish objects. As an example, living organisms may have a different thermal contrast from the surrounding environment at night than during the day.

[0089] Some solutions attempt to normalize these variations through a radiometric calibration camera that estimates the temperature within the surrounding environment. That is, this radiometric calibration camera can be configured to use a standard emissivity value across the surrounding environment. However, using a standard emissivity can potentially reduce the accuracy of thermography techniques. For example, since the emissivity of asphalt is significantly different from that of grass, setting a standard emissivity for both of these materials can lead to inaccurate object classification.

[0090] To address these and other issues, the present embodiment provides a machine learning system that can be trained to generate one or more previous thermal maps based on environmental attributes surrounding the vehicle. These previous thermal maps can be used to normalize thermal variations in the environment surrounding the vehicle. Also, the machine learning system described herein can utilize visible / near-infrared reflectivity, two-color infrared radiation, and / or LIDAR data to predict the emissivity of various elements within the environment, thereby improving the classification of objects within such an environment. Other advantages may also be possible.

[0091] FIG. 5 shows an example system 500 that can be configured to classify objects in the environment through which vehicle 100 is passing, according to an embodiment example. System 500 is shown to include an environmental sensor 510, an infrared module 520, a LIDAR / visible module 530, a thermo machine learning (ML) model 540, an emissivity ML model 550, a normalization module 560, a projection module 570, and a classification machine learning ML model 580.

[0092] Environmental sensor 510 can be configured to identify physical properties related to the environment at that time surrounding vehicle 100. This physical property can include a property representing the ambient / background characteristics of the surrounding environment. Environmental sensor 510 can accordingly generate an environmental attribute 512, which is a signal representing the environment at that time surrounding vehicle 100.

[0093] Infrared module 520 can be configured to generate an infrared image 522 corresponding to the environment through which vehicle 100 is passing. In some embodiments, infrared module 520 actively emits near-infrared light pulses reflected from respective points in the environment. The intensity of these reflected pulses (i.e., return pulses) can be measured by infrared module 520 and captured as infrared image 522. In some embodiments, infrared module 520 may passively measure infrared radiation (e.g., mid-infrared radiation or far-infrared radiation) emitted as heat from a target object near vehicle 100. The intensity of the measured infrared radiation can be captured as infrared image 522. In some embodiments, infrared image 522 represents a plurality of different images captured at a plurality of different times from the environment.

[0094] The LIDAR / visible module 530 can be configured to generate LIDAR data / visible light data from the environment through which the vehicle 100 is passing. In some embodiments, the LIDAR / visible module 530 can utilize a laser rangefinder / LIDAR unit 128 to transmit electromagnetic signals in the near-infrared spectrum (e.g., 905 nm or 1550 nm) that are reflected by various objects near the vehicle 100. In some embodiments, the LIDAR / visible module 530 can utilize a camera 130 to transmit electromagnetic signals in the visible light spectrum (e.g., 380 nm to 740 nm) that are reflected by various objects near the vehicle 100. The LIDAR / visible module 530 can capture the reflected electromagnetic signals and, in response, can make various determinations about the objects that reflected the electromagnetic signals. For example, the LIDAR / visible module 530 can determine the reflectivity of an object that reflected an electromagnetic signal based on the radiant energy of the reflected electromagnetic signal. Thus, the LIDAR / visible data 532 can represent the radiant energy of the reflected electromagnetic signals captured by the LIDAR / visible module 530 from the environment.

[0095] The thermal ML model 540 can be implemented as software instructions executable by a processor (e.g., processor 113), as programmable circuitry (e.g., a field programmable gate array (FPGA)), as dedicated circuitry (e.g., an application specific integrated circuit (ASIC)), or as a combination thereof. The thermal ML model 540 can be communicatively coupled to the environmental sensor 510. In operation, the thermal ML model 540 can receive the environmental attributes 512 from the environmental sensor 510 and can process the environmental attributes 512 to generate a previous thermal map 542. The previous thermal map 542 can correspond to a thermal map (e.g., 3-D or 2-D) captured under physical properties similar to the environmental attributes 512.

[0096] Vehicle 100 can normalize the thermal variations in the infrared image 522 using one or more previous thermal maps 542. For example, when the infrared module 520 generates an infrared image 522 representing the environment surrounding vehicle 100, the thermal ML model 540 can also generate a previous thermal map 542 in response, using the environmental attributes 512 received from the environmental sensor 510. Thereby, both the infrared image 522 and the previous thermal map 542 can be sent to a normalization module 560, which can be implemented as software instructions executable by a processor (e.g., processor 113). The normalization module 560 can subtract the previous thermal map 542 from the infrared image 522 to create a normalized map 562. The idea here is that since the normalization module 560 removes the thermal noise / thermal background of the infrared image 522, the resulting normalized map 562 enables objects to exhibit higher contrast from their surrounding environment, enhancing its ability to classify these objects. In some embodiments, when multiple previous thermal maps 542 are used, the normalization module 560 can average or otherwise weight the values of each of the multiple thermal maps before performing the normalization. In some embodiments, the normalization module 560 can also act to edit or otherwise remove ranges within the previous thermal map 542 that contain predefined object types (e.g., living organisms).

[0097] The emissivity ML model 550 can be implemented as software instructions executable by a processor (e.g., processor 113). The emissivity ML model 550 can be communicatively connected to the LIDAR / visible module 530. In operation, the emissivity ML model 550 can receive LIDAR / visible data 532 from the LIDAR / visible module 530 and process the LIDAR / visible data 532 to generate an emissivity map 552. In an example, each point (e.g., pixel) within the emissivity map 552 can be labeled with a corresponding emissivity value. Thus, the emissivity map 552 can be used to predict the emissivity of various objects within the environment surrounding vehicle 100.

[0098] The projection module 570 can be implemented as software instructions executable by a processor (e.g., processor 113). The projection module 570 can take both the normalization map 562 and the emissivity map 552 as inputs. Thereby, the projection module 570 can match the coordinates of the emissivity map 552 with the coordinates of the normalization map 562 and project the emissivity values from the emissivity map 552 onto the normalization map 562. Thereby, the resulting projection map 572 will include the emissivity values of each point within the normalization map 562.

[0099] In some embodiments, the projection module 570 can combine the emissivity map 552 and the normalization map 562 to generate a temperature map. In other words, the projection map 572 can take the form of a temperature map of the environment surrounding the vehicle 100. To do this, the projection module 570 can use the emissivity map 552 to determine the emissivity values of various objects within the environment. Thereby, the projection module 570 can calculate the contact temperature of each object by using the determined emissivity values of the objects and the infrared radiation of the objects as captured by the normalization map 562.

[0100] In some embodiments, the system 500 can directly determine the temperature map through two-color ratio temperature measurement. For example, the infrared module 520 can include an infrared light emitter device and / or a light receiving element / camera that operates in various spectral wavelength bands (e.g., mid-wavelength infrared (3 - 5 μm) and long-wavelength infrared (7 - 14 μm)). As a result, the temperature map can be computed by a computer by calculating the ratio between the spectral intensity information of each of the various wavelength bands.

[0101] The classification ML model 580 can be implemented as software instructions executable by a processor (e.g., processor 113). The classification ML model 580 can be communicatively connected to the projection module 570. During operation, the classification ML model 580 can receive the projection map 572 from the projection module 570 and process the projection map 572 to classify objects within the projection map 572. In an example, the classification ML model 580 can be configured to identify organisms, traffic signals and signs, roads, vegetation, and other environmental features, where the other environmental features can include mail posts, benches, trash cans, sidewalks, and / or any other objects within the environment that may be of interest to the operation of the vehicle 100.

[0102] In an example embodiment, the classification ML model 580 can perform spatio-temporal association of objects to improve classification accuracy. For example, the classification ML model 580 can act to classify an object at timestamps T-1 and T-2. When performing this classification, the classification ML model 580 can be configured to assign an identifier to each of the objects classified at timestamps T-1 and T-2. Thereby, when classifying an object at timestamp T, the classification ML model 580 can assign a high confidence level to the classification of that object if the identifier of the object was present at timestamps T-1 and T-2. For example, if "pedestrian A" was present at timestamps T-1 and T-2, the classification ML model 580 can assign a higher confidence level to the classification of "pedestrian A" at timestamp T. Depending on the embodiment, the classification ML model 580 can perform spatio-temporal association of objects using the Mahalanobis distance, taking into account the position of the object between the timestamp and the uncertainty information from the Kalman filter.

[0103] In an example embodiment, the classification ML model 580 may take the form of a convolutional neural network (CNN). In some cases, this CNN may use an encoder network coupled to a decoder network and may follow a semantic segmentation architecture. For example, the classification ML model 580 may take the form of a U-Net. The classification ML model 580 may take the form of a trained machine learning model 432.

[0104] In an example embodiment, the classification ML model 580 may be configured to detect objects having a confidence threshold. The confidence threshold may vary depending on the type of object detected. For example, for an object that may require a quick response action, such as the brake light of another vehicle, the confidence threshold may be low. However, in other embodiments, the confidence threshold may be the same for all detected objects.

[0105] If the confidence associated with the detected object is higher than the confidence threshold, the classification ML model 580 may consider the object to be correctly recognized and, accordingly, may adjust the control of the vehicle 100 based on that premise.

[0106] If the confidence associated with the detected object is lower than the confidence threshold, the measures taken by the classification ML model 580 may change. In some embodiments, the classification ML model 580 may cause the vehicle 100 to react as if the detected object exists, despite the low confidence level. In other embodiments, the classification ML model 580 may cause the vehicle 100 to react as if the detected object does not exist. In some embodiments, if an object is detected as having a confidence below the confidence threshold, the object may be given a preliminary identification, and the classification ML model 580 may cause the vehicle 100 to adjust its operation in response to the preliminary identification. Such adjustments to the operation may take the form of, among other adjustments, stopping, switching to a human-controlled (e.g., manual) operation mode, or changing the speed (e.g., the speed and / or direction).

[0107] Depending on the embodiment, in response to a determination that an object has a detection reliability below a threshold, the classification ML model 580 may send a request for remote assistance to a remote computing system such as the remote computing system 302, along with the identification of the object. Additionally and / or alternatively, if the view of the vehicle 100's surrounding environment is blocked, and / or if the environmental sensor 510, the infrared module 520, or the LIDAR / visible module 530 is malfunctioning, the vehicle 100 may request remote assistance from the remote computing system.

[0108] As described above, the remote computing system can take various forms. For example, the remote computing system can be a second computing device within a second vehicle separate from the vehicle 100. The remote computing system can be within a threshold distance (e.g., within 100 m or 1000 m) from the vehicle 100 and can communicate via the wireless communication system 146 or the network 304. The remote computing system can be configured to provide data such as the environmental attribute 512, the infrared image 522, the LIDAR / visible data 532, the previous thermal map 542, the emissivity map 552, the normalization map 562, and / or the projection map 572 to the vehicle 100. Depending on the embodiment, the data transmitted to the vehicle 100 by the remote computing system can be determined by the remote computing system within a threshold time limit (e.g., 1 millisecond, 1 second) from when the infrared module 520 generated the infrared image 522.

[0109] In other embodiments, even when the classification ML model 580 detects an object with a reliability that meets or exceeds a threshold, the classification ML model 580 can cause the vehicle 100 to operate according to the detected object (e.g., stop if the object is identified as a stop sign with high reliability), but the vehicle 100 can be configured to request remote assistance simultaneously (or later) while operating according to the detected object.

[0110] FIG. 6A shows an aspect of the thermal ML model 540 according to an example embodiment. In accordance with the above description, the environmental attribute 512 may take the form of the input data 430, the thermal ML model 540 may take the form of the trained machine learning model 432, and the previous thermal map 542 may take the form of the inference and / or prediction 450.

[0111] The environmental attribute 512 may represent the physical properties associated with the environment at that time surrounding the vehicle 100. These physical properties may include the weather condition 602 of the environment at that time, the GPS 604 (which may take the form of the GPS 122) information regarding the position of the vehicle 100 relative to the earth at that time, the ambient temperature 606 of the environment at that time, the solar position 608 of the sun relative to the vehicle 100 at that time, and the road surface temperature 610 of the road under the vehicle 100 at that time. In some embodiments, the solar position 608 can be estimated using the GPS 604 and the time (e.g., the vehicle 100 may be operable to determine the position of the sun based on the date and location). Note that other environmental attributes 512 are also conceivable.

[0112] The thermal ML model 540 may be configured to receive the environmental attribute 512 as an N-dimensional vector. For example, the environmental attribute 512 may be passed to the thermal ML model 540 as a plurality of values X1 to XN (i.e., X1, X2, X3, X4, X5, X6, X7, X8, X9, and X10 to XN) in an N-dimensional vector space. The thermal ML model 540 may be trained to predict one or more previous thermal maps captured under the same physical properties as the environmental attribute 512. The previous thermal maps may be captured by the vehicle 100 on the previous day or by another vehicle operating in the same environment as the vehicle 100 on the previous day. The thermal ML model 540 may be trained with some (e.g., 1000, 10000) pairs of (environmental attribute, thermal map), in which case the environmental attribute represents the training feature and the thermal map represents the label. In some cases, the thermo-machine learning model 514 may be trained only with pairs of (environmental attribute, thermal map) collected at night (e.g., as detailed by the solar position 608).

[0113] In an example embodiment, each previous heat map in the previous heat map 542 can be associated with (i) a probability value indicating the correlation of the previous heat map with the environmental attribute 512, and (ii) a timestamp indicating the date on which the previous heat map was captured. In some examples, the vehicle 100 can determine to utilize the previous heat map with the highest correlation from the previous heat map 542 (e.g., any previous heat map having a correlation exceeding 90% or 80%). In other examples, the vehicle 100 can determine to utilize all previous heat maps that are sufficiently correlated (e.g., having a correlation exceeding 70%) captured in the past N days (e.g., 7 days, 14 days). Other ways of selecting the previous heat map can also be contemplated. It will be understood that other correlation values and / or correlation thresholds are possible and contemplated within the scope of the present disclosure.

[0114] FIG. 6B shows aspects of the emissivity ML model 550 according to an example embodiment. In accordance with the above description, the LIDAR / visible data 532 can take the form of the input data 430, the emissivity ML model 550 can take the form of the trained machine learning model 432, and the emissivity map 552 can take the form of the inference and / or prediction 450. As shown in FIG. 6B, the LIDAR / visible data 532 can include LIDAR data 612 and visible light data 614.

[0115] As described above, the LIDAR data 612 can represent the radiant energy of electromagnetic signals captured in the infrared spectrum from the environment surrounding the vehicle 100. For example, the laser rangefinder / LIDAR unit 128 can be configured to (i) transmit electromagnetic signals in the near-infrared spectrum that are reflected by various objects near the vehicle 100, and (ii) capture the electromagnetic signals reflected from the objects. The visible light data 614 can represent the radiant energy of electromagnetic signals captured in the visible light spectrum from the environment surrounding the vehicle 100. For example, the camera 130 can be configured to (i) transmit electromagnetic signals in the visible light spectrum that are reflected by various objects near the vehicle 100, and (ii) capture the electromagnetic signals reflected from the objects.

[0116] Depending on the embodiment, the LIDAR data 612 / visible light data 614 may further include, as a parameter in the data, the pointing direction of the device used to capture the LIDAR data 612 / visible light data 614. That is, the device (e.g., the laser rangefinder / LIDAR unit 128, the camera 130, or another device) may be operable to discretize the environment surrounding the vehicle 100 into a finite set of pointing directions covering the 360° range of the device. Since the emissivity of an object may depend on the angle, including the pointing direction may improve the prediction made by the emissivity ML model 550.

[0117] Similar to the thermal ML model 540, the emissivity ML model 550 may be configured to receive the LIDAR / visible data 532 as an N-dimensional vector. For example, the LIDAR / visible data 532 may be passed to the emissivity ML model 550 as a plurality of values X1 to XN (i.e., X1, X2, X3, X4, X5, X6, X7, X8, X9, and X10 to XN) in an N-dimensional vector space. After receiving the LIDAR / visible data 532, the emissivity ML model 550 can predict an emissivity map 552 corresponding to the scene captured by the LIDAR / visible data 532. The emissivity ML model 550 may be trained on some (e.g., 1000, 10000) pairs of (LIDAR / visible data, emissivity map), where the LIDAR / visible data represents training features and the emissivity map represents labels.

[0118] The emissivity ML model 550 can operate in two phases. In the first phase, the emissivity ML model 550 can use the inverse correlation between reflectivity and emissivity to convert the radiant energy values in the LIDAR / visible data 532 into the predicted spectral distribution of the emissivity values of each object around the vehicle 100. In the second phase, the emissivity ML model 550 can predict the true emissivity spectrum of each object that most closely correlates with the predicted spectral distribution of the emissivity values. Using the true emissivity spectrum, the emissivity ML model 550 can infer the emissivity value of each object regardless of its wavelength. As a result, the emissivity ML model 550 can correspondingly generate an emissivity map 552 having the emissivity values of each object corresponding to the same wavelength at which the infrared module 520 operates.

[0119] IV. Method Example FIG. 7 shows a method 700 according to an exemplary embodiment. The operations of the method 700 can be used in either the vehicle 100 or 200, the server computing system 306, the remote computing system 302, or the system 500. The operations can be performed, for example, by the control system 106, the computer system 112, or circuitry configured to perform the operations.

[0120] Block 710 includes receiving, at a computing device, one or more infrared images of the environment. In some embodiments, receiving one or more infrared images of the environment can include receiving multiple infrared images from an infrared camera such as a laser rangefinder / LIDAR 128, a camera 130, or an infrared module 520 via a wired or wireless communication link.

[0121] Block 720 includes applying a trained machine learning system, which is trained to determine identified object types in an environment using a computing device, to the infrared image above. Applying the trained machine learning system may involve determining one or more previous heat maps related to the environment. Applying the trained machine learning system may further involve using the one or more previous heat maps and the one or more infrared images to determine the current heat map related to the environment. Applying the trained machine learning system may also involve determining the identified object types based on the current heat map.

[0122] Block 730 includes providing the identified object types using a computing device. In some embodiments, providing the identified object types may include transmitting the identified object types to the vehicle 100 / 200 via a wired or wireless communication link. Other ways of providing the identified object types for use are possible and contemplated.

[0123] In some embodiments, the identified object types are living organisms. For example, the living organisms may take the form of pedestrians, motorcyclists, bicyclists, dogs, or horses.

[0124] In some embodiments, determining the one or more previous heat maps involves receiving the current heat state of the environment from a plurality of sensors communicatively coupled to the computing device and determining the one or more previous heat maps from the plurality of previous heat maps stored in the computing device based on the current heat state.

[0125] In some embodiments, the plurality of sensors includes at least one of an ambient temperature sensor, a GPS sensor, and a solar sensor operable to determine the current position of the sun.

[0126] In some embodiments, the computing device is incorporated into a scanning laser system. In such embodiments, each of the one or more previous thermal maps corresponds to the angle of the scanning laser system, and determining the one or more previous thermal maps is further based on the then-current angle of the scanning laser system.

[0127] In some embodiments, the one or more previous thermal maps include one or more regions that have been modified, edited, or removed based on the removed object type.

[0128] In some embodiments, determining the then-current thermal map related to the environment involves calibrating the thermal contrast of one or more infrared images based on the one or more previous thermal maps.

[0129] In some embodiments, it further involves determining the then-current spatial point cloud of the environment based on data received from a LIDAR system, using the then-current spatial point cloud to identify objects in the environment, calculating the emissivity value of the identified objects, and determining the identified object type based further on the calculated emissivity value.

[0130] In some embodiments, the then-current thermal map includes the then-current spatial point cloud projected onto one or more infrared images.

[0131] In some embodiments, it further includes determining the then-current visible light image based on data from a camera system, using the then-current visible light image to identify objects in the environment, and calculating the emissivity value of the identified objects, and determining the identified object type based further on the calculated emissivity value.

[0132] In some embodiments, the one or more infrared images include a set of temporally consecutive infrared images of the environment. In such embodiments, applying a trained machine learning system to the one or more infrared images involves a spatio-temporal association of the identified object types between the one or more infrared images.

[0133] In some embodiments, the spatio-temporal association includes assigning a unique identifier to each object that is an identified object type in the environment.

[0134] In some embodiments, the trained machine learning system includes a convolutional neural network. In some embodiments, the convolutional neural network is a segmented network having an encoding path and a decoding path.

[0135] In some embodiments, the one or more previous heat maps include a heat map determined by a second computing device located at a different location in the environment than the computing device.

[0136] In some embodiments, the heat map determined by the second computing device is determined within a threshold time limit from when the computing device received one or more infrared images.

[0137] FIG. 8 shows a method 800 according to an example embodiment. The operations of method 800 may be used in either vehicle 100 or 200, server computing system 306, remote computing system 302, or system 500. The operations may be performed, for example, by control system 106, computer system 112, or circuitry configured to perform the operations.

[0138] Block 810 involves receiving one or more infrared images of the environment at a computing device. In some embodiments, receiving one or more infrared images of the environment may include receiving a plurality of infrared images from an infrared camera such as laser range finder / LIDAR 128, camera 130, or infrared module 520 via a wired or wireless communication link.

[0139] Block 820 involves training a machine learning system in one or more infrared images to determine the identified object types within the environment using a computing device. Training the machine learning system may involve training the machine learning system to determine one or more previous heat maps related to the environment. Training the machine learning system may further involve training the machine learning system to determine the then-current heat map related to the environment using the one or more previous heat maps and the one or more infrared images. Training the machine learning system may also involve training the machine learning system to determine the object types identified using the then-current heat map.

[0140] Block 830 involves providing a trained machine learning system using a computing device. Depending on the embodiment, providing the trained machine learning system may include transmitting the trained machine learning system to vehicle 100 / 200 via a wired or wireless communication link. Other ways of providing the trained machine learning system for use are possible and contemplated.

[0141] Depending on the embodiment, the one or more infrared images include infrared images in which spectral intensity information corresponds to each of a plurality of spectral wavelength bands. In these embodiments, training the machine learning system to determine the then-current heat map related to the environment includes calculating the ratio between the respective spectral intensity information to calculate the emissivity of the objects within the environment.

[0142] The specific arrangements shown in the figures should not be regarded as limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Also, some of the illustrated elements may be combined or omitted. Furthermore, exemplary embodiments may include elements not shown in the figures.

[0143] Steps or blocks representing the processing of information may correspond to circuitry configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, steps or blocks representing the processing of information may correspond to a module, a segment, a physical computer (e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)), or a portion of program code (including associated data). The program code may include one or more instructions executable by a processor for implementing specific logical functions or logical operations in a method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including a disk, a hard drive, or other storage medium.

[0144] Computer-readable media may also include non-transitory computer-readable media such as computer-readable media that store data for short periods of time, such as register memory, processor cache, and random access memory (RAM). Computer-readable media may also include non-transitory computer-readable media that store program code and / or data for extended periods of time. Thus, computer-readable media may include, for example, secondary or long-term persistent storage areas such as read-only memory (ROM), optical or magnetic disks, and compact disc read-only memory (CD-ROM). The computer-readable media can also be any other volatile or non-volatile storage system. The computer-readable media may be considered, for example, a computer-readable storage medium or a tangible storage device.

[0145] Although various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are for illustrative purposes only and are not intended to be limiting, and their true scope is indicated by the following claims.

Claims

1. A computer-implemented method, comprising: receiving, at a computing device, one or more infrared images of an environment; applying, using the computing device, a machine learning system trained to determine identified object types within the environment to the one or more infrared images, wherein the determining comprises, at least: determining one or more previous thermal maps related to the environment; determining a current point cloud of the environment based on data received from a LiDAR system; identifying objects within the environment using the current point cloud; calculating an emissivity value of the identified objects; determining a current thermal map related to the environment using the one or more previous thermal maps and the one or more infrared images; determining the identified object types based on the current thermal map and the calculated emissivity values; and providing, using the computing device, the identified object types.

2. The computer-implemented method of claim 1, wherein the identified object types are living organisms.

3. The determining of the one or more previous thermal maps comprises: receiving, at the computing device, a current thermal state of the environment from a plurality of sensors communicatively coupled to the computing device; and determining the one or more previous thermal maps from a plurality of previous thermal maps stored in the computing device based on the current thermal state.

4. The computer-implemented method of claim 3, wherein the plurality of sensors comprises at least one of an ambient temperature sensor, a global positioning system (GPS) sensor, and a solar sensor operable to determine a current position of the sun.

5. The computer-implemented method of claim 3, wherein the computing device is incorporated into a scanning laser system, each of the one or more previous thermal maps is related to a pointing direction of a device that captures laser light in the scanning laser system, and the determining of the one or more previous thermal maps is further based on a current pointing direction of the device that captures laser light in the scanning laser system.

6. A computer-implemented method, comprising: receiving, at a computing device, one or more infrared images of an environment; applying, using the computing device, a machine learning system trained to determine identified object types within the environment to the one or more infrared images, wherein the determining comprises at least: determining one or more previous thermal maps related to the environment; determining a current point cloud of the environment based on data received from a LIDAR system; identifying objects within the environment using the current point cloud; determining a current thermal map related to the environment using the one or more previous thermal maps and the one or more infrared images, including calibrating a thermal contrast of the one or more infrared images based on the one or more previous thermal maps; determining the identified object types based on the current thermal map; whereby the applying is performed; providing, using the computing device, the identified object types. **Claim 7** The computer-implemented method of claim 1, wherein the current thermal map includes the current point cloud projected onto the one or more infrared images. **Claim 8** The computer-implemented method of claim 6, wherein calibrating the thermal contrast of the one or more infrared images includes editing or removing one or more ranges within the one or more previous thermal maps that include predefined object types. **Claim 9** A computer-implemented method, comprising: receiving, at a computing device, one or more infrared images of an environment; applying, using the computing device, a machine learning system trained to determine identified object types within the environment to the one or more infrared images, wherein the determining comprises at least: determining one or more previous thermal maps related to the environment; determining a current visible light image based on data from a camera system; identifying objects within the environment using the current visible light image; calculating an emissivity value of the identified objects. Determining a current thermal map related to the environment using the one or more previous thermal maps and the one or more infrared images; Determining the identified object type based on the current thermal map and the calculated emissivity value; Performed by; Using the computing device to provide the identified object type, a computer-implemented method comprising.

10. The one or more infrared images include a set of temporally consecutive infrared images of the environment, and applying the trained machine learning system to the one or more infrared images includes a spatio-temporal association of the identified object types between the one or more infrared images. The computer-implemented method according to claim 1.

11. The spatio-temporal association includes assigning a unique identifier to each object that is the identified object type within the environment. The computer-implemented method according to claim 10.

12. The trained machine learning system comprises a convolutional neural network. The computer-implemented method according to claim 1.

13. The convolutional neural network is a segmentation network having an encoding path and a decoding path. The computer-implemented method according to claim 12.

14. The one or more previous thermal maps include thermal maps determined by a second computing device located at a location different from the computing device within the environment. The computer-implemented method according to claim 1.

15. The thermal map determined by the second computing device is determined within a threshold time limit from when the computing device received the one or more infrared images. The computer-implemented method according to claim 14.

16. The computing device is part of an autonomous driving vehicle system. The computer-implemented method according to claim 1.

17. A computer-implemented method, comprising: Receiving, at a computing device, one or more infrared images of an environment, the one or more infrared images including infrared images in a state where spectral intensity information corresponds to a plurality of respective spectral wavelength bands; Using the computing device to train a machine learning system on the one or more infrared images to determine identified object types within the environment, including at least training the machine learning system to determine one or more previous thermal maps associated with the environment; using the one or more previous thermal maps and the one or more infrared images to train the machine learning system to determine a current thermal map associated with the environment, including calculating a ratio between respective spectral intensity information to calculate an emissivity of an object within the environment; training the machine learning system to determine the identified object types using the current thermal map; being performed by; using the computing device to provide the trained machine learning system. A computer-implemented method comprising.

18. A computing device, comprising: one or more processors; a data storage storing computer-executable instructions that, when executed by the one or more processors, cause the computing device to receive one or more infrared images captured from an environment; apply a trained machine learning system to the one or more infrared images to determine identified object types within the environment, the determining including at least determining one or more previous thermal maps associated with the environment; determining a current spatial point cloud of the environment based on data received from a LiDAR system; identifying objects within the environment using the current spatial point cloud; calculating an emissivity value of the identified objects; using the one or more previous thermal maps and the one or more infrared images to determine a current thermal map associated with the environment; determining the identified object types using the current thermal map and the calculated emissivity value; being performed by; providing the identified object types. A computing device that causes a function to be performed, comprising.

Images