Smart LiDAR Sensors for Autonomous Vehicles

Abstract

The present application relates to an intelligent laser radar sensor for an autonomous vehicle. A laser radar sensor may include: a memory configured to store a laser radar image and an artificial neural network (ANN); an inference engine configured to analyze the laser radar image using the ANN and generate an inference result; and a communication interface coupled to a computer system of a vehicle to implement an advanced driver assistance system to operate controls based on the inference result and a sensor data stream generated by a sensor configured on the vehicle.

Description

Technical Field

[0001] At least some embodiments disclosed herein relate generally to lidar sensors, and more particularly, but not limited to, lidar sensors for autonomous vehicles. Background Art

[0002] Recent developments in the art of autonomous driving allow computing systems to operate control elements of a motor vehicle without assistance (at least in some circumstances) from a human vehicle operator.

[0003] For example, sensors (e.g., cameras, radar, and / or lidar) may be mounted on a motor vehicle to detect conditions around the vehicle traveling on a road. A computing system mounted on the vehicle analyzes the sensor input to identify the conditions and generates control signals or commands for autonomously adjusting the direction and / or speed of the vehicle with or without any input from a human vehicle operator.

[0004] In some arrangements, when the computing system identifies a situation in which the computing system may not be able to continue to operate the vehicle in a safe manner, the computing system may alert the human vehicle operator and request the human vehicle operator to take over control of the vehicle and drive manually, rather than allowing the computing system to autonomously drive the vehicle.

[0005] Autonomous driving and / or advanced driver assistance systems (ADAS) can use artificial neural networks (ANNs) to identify events and / or objects captured in sensor inputs. Examples of sensor inputs include images from digital cameras, lidar, radar, ultrasonic sonar, etc.

[0006] Typically, an artificial neural network (ANN) uses a network of neurons to process inputs to the network and generate outputs from the network.

[0007] For example, each neuron in the network receives a set of inputs. Some of the inputs to the neurons may be outputs of certain neurons in the network; and some of the inputs to the neurons may be inputs provided to the neural network. The input / output relationships between neurons in the network represent the connectivity of the neurons in the network.

[0008] For example, each neuron may have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function may be in the form of a step function, a linear function, a sigmoid function, etc. Different neurons in the network may have different activation functions.

[0009] For example, each neuron may generate a weighted sum of its input and its bias, and then produce an output that is a function of the weighted sum, calculated using the neuron's activation function.

[0010] The relationship between one or more inputs and one or more outputs of an ANN is typically defined by an ANN model, which includes data representing the connectivity of neurons in the network, as well as the bias, activation function, and synaptic weights of each neuron. A computing device uses a given ANN model to calculate one or more outputs of the network based on a given set of inputs to the network.

[0011] For example, the input to the ANN network may be generated based on the camera input; and the output from the ANN network may be an identification of an item such as an event or an object.

[0012] A spiking neural network (SNN) is an ANN that closely mimics a natural neural network. When the activation level of an SNN neuron is high enough, the SNN neuron produces a spike as an output. The activation level of an SNN neuron mimics the membrane potential of a natural neuron. The output / spike of an SNN neuron can change the activation level of other neurons that receive the output. The current activation level of an SNN neuron as a function of time is typically modeled using differential equations and is considered to be the state of the SNN neuron. Incoming spikes from other neurons can push the activation level of a neuron higher to reach a spiking threshold. Once a neuron spikes, its activation level is reset. Before spiking, the activation level of an SNN neuron can decay over time, as controlled by a differential equation. The temporal element in the behavior of SNN neurons makes SNNs suitable for processing spatiotemporal data. The connectivity of SNNs is typically sparse, which is advantageous in reducing computational workload.

[0013] Typically, an ANN can be trained using supervised methods in which parameters in the ANN are adjusted to minimize or reduce the error between the known outputs produced by the corresponding inputs and the calculated outputs generated by applying the inputs to the ANN. Examples of supervised learning / training methods include reinforcement learning and error correction learning.

[0014] Alternatively or in combination, the ANN can be trained using unsupervised methods in which the exact output produced by a given set of inputs is unknown until training is complete. The ANN can be trained to classify items into multiple categories, or to classify data points into clusters.

[0015] There are a variety of training algorithms that can be used for complex machine learning / training paradigms. Summary of the invention

[0016] One aspect of the present disclosure relates to a lidar sensor, the lidar sensor comprising: a laser pulse generator; a directional control; a beam guide, the beam guide coupled to the laser pulse generator and the directional control to project laser pulses in different directions; a light sensor; a light receiver, the light receiver configured to guide the laser pulse reflected back to the lidar sensor to the light sensor; a timing circuit system, the timing circuit system configured to determine the time gap between the corresponding laser pulse projected by the beam guide and the corresponding laser pulse reflected back to the light sensor; a memory, the memory configured to store a lidar image and an artificial neural network, wherein the lidar image identifies attributes of multiple points that reflect laser pulses back to the light sensor based on input from the directional control and the timing circuit system; an inference engine, the inference engine configured to analyze the lidar image using the artificial neural network and generate an inference result; and a host interface, the host interface configured to provide the inference result to a vehicle's computer system.

[0017] Another aspect of the present disclosure relates to a vehicle, comprising: a plurality of controls for steering, braking, and acceleration; a set of sensors configured to generate a sensor data stream; a lidar sensor, the lidar sensor comprising: a lidar image generator; a memory configured to store lidar images and an artificial neural network; an inference engine configured to analyze the lidar images using the artificial neural network and generate inference results; and a communication interface; a computer system coupled to the communication interface to implement an advanced driver assistance system to operate the controls based on the inference results and the sensor data stream.

[0018] Yet another aspect of the present disclosure relates to a method, comprising: generating a lidar image through a lidar sensor; storing at least one lidar image and an artificial neural network in a memory of the lidar sensor; analyzing the lidar image using the artificial neural network through an inference engine of the lidar sensor to generate an inference result; and transmitting the inference result to a computer system of a vehicle through an interface of the lidar sensor, wherein the computer system is configured to generate a control signal for autonomous driving of the vehicle based on the inference result and a sensor data stream generated by a sensor configured on the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0020] Figure 1A lidar sensor is shown according to one embodiment.

[0021] Figure 2 A system is shown in which a vehicle is configured to collect and process sensor data according to some embodiments.

[0022] Figure 3 An autonomous vehicle is shown according to one embodiment.

[0023] Figure 4 Memory capacity configured to support neural network computations is shown according to one embodiment.

[0024] Figure 5 Communication between an autonomous vehicle and a lidar sensor is shown according to one embodiment.

[0025] Figure 6 Communications within a lidar sensor are shown according to one embodiment.

[0026] Figure 7 A spiking neural network tile implemented in a lidar sensor using memristors is shown according to one embodiment.

[0027] Figure 8 The use of spiking in a spiking neural network implemented using memristors is demonstrated using current spike detection through a memristor array according to one embodiment.

[0028] Fig. 9 A spiking neural network is shown using a network implementation of spiking neural network blocks according to one embodiment.

[0029] Fig.10 demonstrated the processing of spiking neural networks by using memristors to detect spikes and routing the detected spikes between memristor blocks.

[0030] Fig.11 A method of a lidar sensor according to one embodiment is shown. DETAILED DESCRIPTION

[0031] At least some embodiments disclosed herein provide systems, methods, and apparatus for a smart lidar sensor in a motor vehicle or another vehicle with or without an advanced driver assistance system (ADAS).

[0032] For example, an artificial intelligence (AI) engine implemented using a neuromorphic memory device may be configured in a lidar sensor. The AI ​​engine may analyze lidar information to generate inference results for a host system (e.g., an ADAS of an autonomous vehicle). For example, the AI ​​engine may process a lidar image, extract relevant portions of the lidar image, and provide the extracted portions to the ADAS for sensor fusion. For example, the AI ​​engine may process a lidar image to identify characteristics of an object captured in the lidar image and transmit the characteristics to the ADAS for further analysis and / or adjustment of control of the vehicle.

[0033] Lidar sensors can be used in autonomous vehicles for obstacle avoidance, lane keeping, blind spot detection, rear-end collision warnings, intersection traffic warnings, adaptive cruise control, etc. The lidar sensor projects a laser pulse in a certain direction. When the laser pulse is reflected back to the lidar sensor by an object at a certain point, the lidar sensor can calculate the distance between the point and the lidar sensor based on the speed of light and the time delay between the transmission of the laser pulse and the reception of the reflected laser pulse. The distance identifies the position of the point relative to the lidar sensor in the three-dimensional space around the lidar sensor. By projecting laser pulses in different directions, the lidar sensor can scan the three-dimensional space around the lidar sensor to find the nearest set of points that can reflect the laser points back toward the lidar sensor. The nearest set of points around the lidar sensor represents the physical environment of the lidar sensor and objects / obstacles near the lidar sensor. Optionally, the intensity of the reflected laser pulse is also measured. The position of the point set and optionally the intensity of the point set of the reflected laser pulse can come from a lidar image as observed by the lidar sensor. The LiDAR image may be in the form of the position of reflection points in three-dimensional space and / or the reflection intensity of the reflection points.Objects / obstacles may be identified by analysis of the LiDAR image using an artificial neural network.

[0034] Instead of providing a lidar image to the vehicle's ADAS, the lidar sensor may have a neuromorphic memory device to perform analysis of the lidar image and generate inference results, which are provided to the vehicle's ADAS. The neuromorphic memory device may be used to implement a deep neural network (DNN) or a spike neural network to analyze a 3-dimensional map of lidar information (e.g., the location of the reflection point of the laser pulse and the reflection intensity of the laser pulse from the reflection point). The analysis results of the DNN or SNN may be sent to the vehicle's ADAS for sensor fusion, further analysis, and / or driving decisions. This arrangement can greatly reduce the amount of data transmitted to (and analyzed by) one or more processors of the ADAS. For example, when the lidar information is locally converted into an inference result using a DNN or SNN at the lidar sensor, the lidar sensor may generate a 3-dimensional map of the lidar information at a resolution and frame rate that exceeds the communication bandwidth supported by the standardized communication line between the lidar sensor and one or more processors of the ADAS. Improved resolution and frame rate can make autonomous driving capabilities better and safer.

[0035] For example, an AI engine can be integrated into the optical sensor of a solid-state LiDAR chip. A solid-state LiDAR can adjust the transmission direction of a laser pulse without physically adjusting the orientation of the laser transmitter. For example, a set of optional transmitters can emit laser transmissions in a specific pattern and phase to produce a directional emission of laser pulses. For example, a metasurface with finely tunable liquid crystals can be controlled to adjust the direction of the laser transmission. In some embodiments, micro-mirrors are used to exert directional control on the laser transmission.

[0036] Figure 1 A lidar sensor 101 is shown according to one embodiment.

[0037] exist Figure 1In the laser radar sensor 101, a laser pulse generator 117 is included. Based on the directional control 125, the beam guide 119 can adjust the direction of the laser pulse relative to the laser radar sensor 101. The optical receiver 121 guides the laser pulse reflected at a certain point on the laser pulse transmission path to the optical sensor 123. The timing circuit system 115 determines the round-trip time for the laser pulse to leave the laser pulse generator 117 and reach the optical sensor 123 after being reflected at the point on the laser pulse transmission path. The directional control 125 identifies the direction of the point; and the timing circuit system 115 measures the distance to the point. Optionally, the optical sensor 123 also measures the intensity of the reflection at the point. The laser radar image 107 contains the attributes of a set of points scanned in different directions. The attributes of each point can include the direction and range of the point relative to the laser radar sensor 101 and, optionally, the reflection intensity at the point. Each point can be considered as a pixel of the laser radar image. In some cases, each pixel is in three-dimensional space, and its reflection intensity is used as an attribute of the pixel. In other cases, each pixel may be mapped to a plane (eg, by projecting onto a spherical surface in the direction of the laser pulse), with range and / or intensity as attributes of the pixel.

[0038] exist Figure 1 , the lidar image 107 is stored in a memory 111 of the lidar sensor 101 in a cyclic manner, wherein the latest lidar image 107 overwrites the earliest lidar image 107. For example, the memory 111 may include an integrated circuit memory cell; and the integrated circuit memory cell may be a non-volatile memory (e.g., NAND, FERAM, NOR, PCM, MRAM, cross-point memory). Further, the memory 111 of the lidar sensor 101 stores an artificial neural network (ANN) 113. The inference engine 103 of the lidar sensor 101 is configured to apply the ANN 113 to the lidar image 107 to generate an inference result 109. The host interface 105 is configured to provide the inference result 109 to a host system, such as a computer of a vehicle implementing ADAS for autonomous driving. When the methods and techniques described therein (e.g., Fig.11 When the method of the invention is implemented at least in part by means of instructions executed by a programmable device (e.g., inference engine 103 and / or a controller or processor), the instructions may be stored in memory 111 or another machine-readable medium.

[0039] Optionally, memory 111 and / or inference engine 103 may be formed on an integrated circuit die that is stacked on an integrated circuit die of light sensor 123. A high bandwidth connection may be provided on the integrated circuit die between memory 111 and inference engine 103. In some embodiments, at least a portion of inference engine 103 is implemented in a portion of memory 111. For example, a memristor crossbar array may be used to store parameters of ANN 113 and to compute matrix multiplications and accumulations involving ANN 113.

[0040] For example, inference result 109 may include an identification of an object captured in lidar image 107 , a position and velocity of the object relative to lidar sensor 101 , a size of the object, one or more classifications of the object, and / or an extracted portion of lidar image 107 showing the object.

[0041] The memory 111 may be implemented using integrated circuit memory. Generally, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and can retain stored data even when power is not supplied. The memory 111 may include volatile memory and / or non-volatile memory. The memory 111 may be implemented using different types of memory or the same type of memory.

[0042] Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase change memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point storage, and memory devices. Cross-point memory devices can use transistor-free memory elements, each of which has a memory cell and a selector, and the memory cell and the selector are stacked together in a column. The memory element column is connected by two layers of lines running in a vertical direction, wherein a layer of lines running in one direction in the layer is located above the memory element column, and the other layer of lines is located in the other direction and below the memory element column in the layer. Each memory element can be selected individually at the intersection of a line on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), etc. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0043] The inference engine 103 may include a neural network accelerator configured to perform matrix arithmetic calculations more efficiently than a general-purpose processing device (e.g., a controller or a central processing unit (CPU)). Optionally, the inference engine 103 also includes one or more general-purpose processors. Calculations involving the ANN 113 have matrix arithmetic calculations (such as matrix multiplication and accumulation operations), which can be computationally intensive for general-purpose processors. Using a neural network accelerator to perform matrix arithmetic calculations can reduce the data to be transmitted to one or more processors and reduce the computational workload of the one or more processors.

[0044] When the ANN 113 includes a spiking neural network (SNN), the simulation of one or more differential equations used to control the activation levels of the SNN neurons can be computationally intensive for a general purpose processor. Optionally, a neural network accelerator can use dedicated hardware to simulate one or more differential equations and thereby improve the computational efficiency of implementing the SNN.

[0045] For example, the laser radar sensor 101 can be configured in Figure 2 In the vehicle shown.

[0046] Figure 2 A system is shown in which a vehicle 211 is configured to collect and process sensor data, according to some embodiments.

[0047] Figure 2 The vehicle 211 in FIG. 2 has a lidar sensor 101 , a data storage device 212 , other sensors 222 , an ANN 225 , and an ADAS 205 configured to process sensor data (including input from a sensor 222 ) to generate control signals for the vehicle 211 .

[0048] Typically, one or more sensors (e.g., 222, 101) may be configured on the vehicle 211 to generate sensor data that is input to the ADAS 205 and / or the data storage device 212. The lidar sensor 101, the data storage device 212, and / or the ADAS 205 may be configured to generate inference results using the ANN 225. The inference results may include control signals for operating or driving the vehicle 211, recommendations for maintenance services for the vehicle 211, and the like.

[0049] In some embodiments, at least a portion of the data generated by the sensors (e.g., 222, 101) is used both in the ADAS 205 for driver assistance and in the ANN 225 for maintenance prediction. Optionally, the output of the ANN 225 can be used in the data storage device 212 and the ADAS 205. The ANN 225 can be part of the ADAS 205.

[0050] Sensors 222 and 101 may be configured in digital cameras, laser radars, radars, ultrasonic sonars, etc. Other types of sensors may also be used, such as brake sensors, speed sensors, acceleration sensors, airbag sensors, GPS (Global Positioning System) receivers, audio sensors / microphones, vibration sensors, force / stress sensors, deformation sensors, motion sensors, temperature sensors, etc. Some of sensors 222 and 101 may be configured primarily to monitor the environment of vehicle 211; and some of sensors 222 and 101 may be configured primarily to monitor the operating conditions of one or more components of vehicle 211, such as an internal combustion engine, an exhaust system, an electric motor, brakes, tires, a battery, etc.

[0051] The ANN 225 of the vehicle 211 is configured to process sensor input data from the sensor 222 and / or the lidar sensor 101 to control the vehicle 211 .

[0052] Typically, the output of one or more sensors 222 and / or lidar sensor 101 as a function of time is provided as a sensor data stream to ADAS 205 and / or ANN 225 to provide driver assistance (eg, autonomous driving) and maintenance prediction.

[0053] At least a portion of the sensor data stream may be provided to the data storage device 212 for storage and / or processing. For example, a portion of the ANN 225 may be implemented in the data storage device 212. The inference engine of the data storage device 212 may process the sensor data stream to generate inference results for further processing by the ADAS 205. Thus, the input data stream to the data storage device 212 may include at least a portion of the sensor data stream from the sensor 222; and the output data stream from the data storage device 212 may include inference results generated for the ADAS 205 of the vehicle 211 using the ANN 225 in the data storage device 212.

[0054] At least a portion of the ANN 225 associated with the lidar image 107 generated in the lidar sensor 101 can be implemented as the ANN 113 and by the inference engine 103 of the lidar sensor 101. Optionally, some sensor data generated by the sensor 222 and / or inference results generated by the data storage device 212 and / or the ADAS 205 can be provided to the lidar sensor 101 for sensor fusion and / or reducing data traffic between communication connections in the vehicle 211.

[0055] The ANN 225 of the vehicle 211 located in the data storage device 212 and / or in the lidar sensor 101 may include an SNN configured to classify time-based changes in the sensor data and / or detect deviations from known patterns of the sensor data of the vehicle 211. When the ANN 225 detects a deviation from a known pattern, the sensor data corresponding to the deviation may be stored in the data storage device 212 and / or in the memory 111 of the lidar sensor 101 for further analysis and / or further training of the ANN 225.

[0056] The data storage device 212 of the vehicle 211 can be configured to record sensor data over a period of time, which can be used in the ANN 225 for predictive maintenance and / or for further training the ANN 225. A maintenance service facility (e.g., 227) can download the sensor data 221 from the data storage device 212 and provide the sensor data 221 and corresponding inference result data 223 to the server 219 to facilitate training of the ANN 225.

[0057] Optionally or in combination, the data storage device 212 and / or the lidar sensor 101 may be configured with a machine learning module for customizing and / or training the ANN 225 installed in the vehicle 211 and / or in the data storage device 212 .

[0058] The vehicle 211 may have a wireless communication device for communicating with a remote server 219 via a wireless signal 213 and a communication network 217. The remote server 219 is typically located away from the road 214 on which the vehicle 211 is traveling. For example, the vehicle 211 may provide some sensor data 221 to the server 219 and receive updates of the ANN 225 from the server 219.

[0059] The communication network 217 may be a cellular telephone network having one or more base stations (e.g., 215) for receiving wireless signals (e.g., 213). Alternatively or in combination, the communication network 217 may include the Internet, wherein the wireless local area network signal (e.g., 213) transmitted by the vehicle 213 is received in an access point (e.g., 215) for further transmission to the server 219. In some embodiments, the vehicle 211 communicates with the server 219 using a communication link 216 to a satellite 218 or a communication balloon.

[0060] Server 219 may also communicate with one or more maintenance service facilities (eg, 227 ) to receive sensor data 221 and / or desired inference data 223 for a vehicle (eg, 211 ).

[0061] For example, the expected inference result data 223 may be generated by a human operator inspecting the sensor data 221 (e.g., a lidar image from the lidar sensor 101, an input from the sensor 222) and / or the relevant condition of the vehicle 211. For example, the expected inference result data 223 may include an inspection record and / or a service record of a component of the vehicle (e.g., 211). For example, the inspection record and / or the service record may indicate the degree of wear of a component inspected during service at a maintenance service facility (e.g., 227), the identification of a failed or faulty component, etc. The sensor data 221 of the vehicle (e.g., 211) obtained during a time period associated with the expected inference result data 223 may be used to train the ANN 225 at the server 219 to improve the inference capability of the ANN 225.

[0062] The updated ANN 225 may be installed in the vehicle 211 at a maintenance service facility 227. Alternatively, the updated ANN 225 may be transmitted to the vehicle 211 to update the vehicle 211 wirelessly.

[0063] A portion of the ANN 225 may be configured in the lidar sensor 101, the portion being responsible for processing the lidar image 107 and / or sensor fusion based on the lidar image 107. The inference engine 103 of the lidar sensor 101 processes the lidar image 107 and optionally the input from the sensor 222 to generate an inference result 109. Based on the input from the sensor 222 and / or the inference result 109 from the lidar sensor 101, the ADAS 205 of the vehicle 211 may generate control of autonomous driving and / or driver assistance for the vehicle 211.

[0064] Figure 3 An autonomous vehicle 211 is shown according to one embodiment. For example, Figure 2 Vehicles in the system 211 can use Figure 3 An autonomous vehicle 211 is implemented.

[0065] Typically, the vehicle 211 may include an infotainment system 249, a communication device 239, a lidar sensor 101, one or more other sensors 222, and a computer system 231 connected to some controls of the vehicle 211, such as a steering control 241 for the direction of the vehicle 211, a brake control 243 for stopping the vehicle 211, an acceleration control 245 for the speed of the vehicle 211, etc. In some embodiments, Figure 2 The vehicle 211 in the system has a similar configuration and / or similar components.

[0066] Figure 3The vehicle 211 is equipped with an advanced driver assistance system (ADAS) 205. The ADAS 205 of the vehicle 211 may have an artificial neural network (ANN) 225 for object detection, recognition, identification, and / or classification based on the images generated by the sensors 222 and 101. A portion of the ANN 225 may be Figure 1 The laser radar sensor 101 and / or Figure 2 is implemented in the data storage device 212.

[0067] The computer system 231 of the vehicle 211 may include one or more processors 233, a data storage device 212, and a memory 235 storing firmware (or software) 247, including computer instructions (e.g., instructions 151 described below) and data models for the ADAS 205. The computer instructions may include, for example, instructions for causing the various components of the vehicle 211 to perform the methods and techniques described herein (including the instructions described below) (through execution by the one or more processors 233). Fig.11 method) instructions.

[0068] The sensors 222 and 101 of the vehicle 211 may include visible light cameras, infrared cameras, laser radars, radar or sonar systems, peripheral sensors, global positioning system (GPS) receivers, satellite positioning system receivers, brake sensors and / or airbag sensors. Further, the sensors 222 and 101 of the vehicle 211 may include audio sensors (e.g., microphones) configured to monitor noise from various components and locations in the vehicle 211, vibration sensors, pressure sensors, force sensors, stress sensors and / or deformation sensors configured to measure loads on components of the vehicle 211, acceleration sensors and / or gyroscope sensors that measure the movement of some components of the vehicle 211, etc. Such sensors can be used to monitor the operating status and / or operating conditions of components for predictive maintenance.

[0069] The sensor 222 and the lidar sensor 101 may provide real-time sensor data streams to the computer system 231. For example, the sensor data generated for the vehicle 211 may include images of objects captured using a camera that uses imaging light visible to the human eye or a camera that uses imaging infrared light, or a sonar, radar, or lidar system. Preferably, at least the lidar image 107 is processed by the inference engine 103 of the lidar sensor 101 to generate an inference result as output data of the lidar sensor 101 and thereby reduce the computational workload of the host computer system 231 and the data traffic to the host computer system.

[0070] For example, the lidar sensor 101 may generate the lidar image 107 at a resolution and frequency that exceeds the communication bandwidth between the lidar sensor 101 and the computer system 231. However, because the inference engine 103 of the lidar sensor 101 reduces the lidar image 107 to a data volume that is much smaller than the inference result 109 of the lidar image 107, the inference result 109 can be transmitted from the lidar sensor 101 and the computer system 231 using the limited communication bandwidth between the lidar sensor 101 and the computer system 231.

[0071] For example, the lidar image 107 may include a point cloud on a surface closest to the lidar sensor 101 as seen by the laser pulse. The ANN 113 may be used to analyze the point cloud to identify objects and / or environmental features having a surface. For example, point clouds corresponding to different objects or features may be separated from each other for identification. For example, a set of points on the surface of an object (e.g., another vehicle, a person, a building) may be analyzed to determine the class / classification of the object. As the point cloud of the surface of the object changes over a period of time, the ANN 113 may be used to determine the speed and direction of travel of the object. Inference results 109 may include a set of points extracted on the surface of the object, identification / classification of the object, the speed and direction of travel of the object, the orientation of the object, etc.

[0072] The infotainment system 249 of the vehicle 211 may be used to present data and / or inferences from the lidar sensor 101. For example, compressed lidar images with reduced resolution and refresh rate may be generated in the lidar sensor 101 and transmitted to the infotainment system 249 for presentation as objects or obstacles around the vehicle 211. Optionally, the communication device 239 may establish a connection with a mobile device of an occupant of the vehicle 211 for presentation.

[0073] When the vehicle 211 is configured with the ADAS 205 , the output of the ADAS 205 may be used to control (eg, 241 , 243 , 245 ) the acceleration of the vehicle 211 , the speed of the vehicle 211 , and / or the direction of the vehicle 211 during autonomous driving.

[0074] For different types of ANN-related data, the memory capacity of the lidar sensor 101 can be divided into different parts. Different parts can be configured separately to optimize access and storage of corresponding data according to its mode of use by the inference engine 103.

[0075] The model of the artificial neural network (ANN) 113 may include parameters that specify the static properties of each artificial neuron in the ANN 113 and the connectivity of neurons in the ANN 113. The model data of the ANN 113 is static and does not change during the prediction calculation using the ANN 113. Therefore, the usage mode of the model data is mainly read. However, the model data of the ANN 113 can change when the updated ANN 113 is installed. For example, the vehicle 211 can download the updated ANN 113 from the server 219 to the data storage device 212 of the vehicle 211 to update its prediction capabilities. The model data of the ANN 113 may also change during or after the ANN 113 is trained using machine learning techniques (e.g., 271 or 275). It is preferred to configure the memory of the data storage device 212 to store the model data in separate partitions or regions, wherein the partitions or regions operate according to configuration parameters, and the configuration parameters optimize the memory unit for a specific usage mode of the model data (e.g., mainly read, infrequently updated). For example, when the memory cells are implemented using NAND logic gate-based flash memory, the memory cells in the ANN model partition / region can be configured to operate in a multi-level cell (MLC) mode, a triple-level cell (TLC) mode, or a quad-level cell (QLC) mode, where two, three, or four bits are stored per memory cell for increased storage capacity.

[0076] The input data provided to the artificial neurons in the ANN 113 may include external inputs and internal inputs. The external inputs are typically generated by sensors (e.g., 209) of the vehicle 211 but not by the artificial neurons in the ANN 113. The external inputs may be stored in a cyclic manner so that input data of a recent period of driving of a predetermined length may be found in the data storage device 212. Therefore, it is preferred to configure a separate memory partition or region of the data storage device 212 to store the external input data, wherein the partition or region operates according to configuration parameters that optimize the memory cells for the storage mode (e.g., enhanced persistence, cyclic rewrite) of the external input data. For example, when the memory cells are implemented using flash memory based on NAND logic gates, the memory cells in the ANN input partition / region may be configured to operate in a single-level cell (SLC) mode, wherein each memory cell stores one bit of data to improve the persistence of the cyclic rewrite operation.

[0077] In some embodiments, artificial neurons may have state variables that change over time in response to inputs during prediction calculations. For example, the activation level of a spiking neuron may change over time and be considered a dynamic state variable of the spiking neuron. In some embodiments, such state variable data for artificial neurons has a storage usage pattern similar to external input data; and therefore, the state variable data may be stored in a partition or region configured for external input data. In other embodiments, the state variable data for artificial neurons is maintained in a buffer and stored at a lower frequency than the external input; and therefore, another partition / region may be configured to store the dynamic state variable data for the artificial neuron.

[0078] The output data generated by the artificial neurons in the ANN 113 can be buffered for further access by the inference engine 101 and / or one or more processors 233 of the computer system 231. The output data may include external output and internal output. External input is generated by the artificial neuron as an output from the ANN 113, such as a classification or prediction result made by the ANN 113. The output of the ANN 113 is usually further processed by one or more processors 233 of the computer system 231. The external input can be saved regularly (for example, in a manner similar to storing state variable data). Some of the internal outputs and / or external outputs can be internal inputs to the artificial neurons in the ANN 113. Generally, it may not be necessary to store the internal output of the buffer from the data storage device to the memory component. In some embodiments, when the buffer capacity of the lidar sensor 101 is not enough to hold the entire state variable data and / or internal output, the lidar sensor 101 can use a swap partition / area to expand the capacity of the buffer. The swap partition / area can be configured to optimize random access and improve persistence.

[0079] External outputs and / or dynamic states of neurons can be stored in separate output partitions or regions in a cyclic manner, so that external output data and / or dynamic states of neurons can be stored periodically and the most recent set of external outputs and / or dynamic states can be found in the lidar sensor 101. External outputs and / or dynamic states of neurons can be stored selectively because the ANN can regenerate some of such data based on external inputs stored in input partitions or regions. Preferably, the output partitions or regions are configured to store one or more sets of external outputs and / or dynamic states that cannot be generated based on external inputs stored in input partitions or regions. When data is stored in a cyclic manner in input / output partitions or regions, the earliest stored data set is erased to make room for the latest data set. The ANN input / output partitions / regions can be configured so that an optimized sequential write stream copies data from a buffer of a data storage device to a memory cell in a memory component of the data storage device.

[0080] Figure 4 Memory capacity 281 is shown configured to support neural network computations according to one embodiment. Figure 4 Configuration Figure 1 The memory capacity 281 of the memory 111 of the lidar sensor 101 is used to support the neural network calculation. Figure 4 The memory capacity 281 of the data storage device 212 is configured to support neural network computations.

[0081] Figure 4 The storage capacity 281 may be implemented using one or more memory components, each memory component being configured within an integrated circuit and / or on an integrated circuit die.

[0082] A set of regions 283, 285, 287, ... may be created on the memory capacity 281. Each of the regions (e.g., 283, 285, or 287) corresponds to a named portion of the memory capacity 281. Logical addresses are defined within each region. The address map 291 is configured to map between the logical addresses defined in the regions 283, 285, 287, ... and the physical addresses of the memory cells in the memory component.

[0083] Address map 291 may include region optimization settings 292 for regions 283 , 285 , and 287 .

[0084] For example, ANN model region 283 may be a memory / storage partition configured for model data of artificial neural network (ANN) 113. Region optimization settings 292 optimize memory operations in ANN model region 283 according to the data usage pattern of the ANN model (e.g., mostly read, infrequently updated centers).

[0085] For example, the neuron input region 285 may be a memory / storage partition configured for external input data to the artificial neural network (ANN) 113. The region optimization settings 292 optimize memory operations in the neuron input region 285 according to the data usage pattern of the external input data (e.g., supporting cyclic rewriting of a continuous input data stream for sequential writing to enhance durability).

[0086] For example, neuron output region 287 may be a memory / storage partition configured for external output data provided from artificial neural network (ANN) 113. Region optimization settings 292 optimize memory operations in neuron output region 287 based on data usage patterns of external output data (e.g., improving durability of data that is regularly rewritten with random read / write accesses).

[0087] Buffers (e.g. Figure 5 252) in can be configured to store temporary / intermediate data of the artificial neural network (ANN) 113, such as internal input / output of artificial neurons in the ANN 113.

[0088] Optionally, a swap area may be configured in the memory volume 281 to expand the buffer (e.g. Figure 5 252) capacity.

[0089] Optionally, the address map 291 contains a mapping between logical memory addresses received in the host interface 105 for accessing data of the artificial neuron and the identity of the artificial neuron. Thus, a read or write command to access one type of data of an artificial neuron in one region may enable the controller (e.g. Figure 5 251) in accessing another type of data of an artificial neuron in another region.

[0090] For example, in response to a request to write external input data of a neuron into the memory capacity 281, the address map 291 can be used to calculate the address of the model parameters of the neuron in the ANN model area 283 and read the model parameters into the buffer 252 to enable the inference engine 103 to perform calculations on the output of the neuron. The output of the neuron can be saved in the buffer 252 as an internal input to other neurons (e.g., to reduce write amplification). Further, the identities of other neurons connected to the neuron can also be retrieved from the ANN model area 283 into the buffer 252, which allows the inference engine 103 and / or the processor to further process the propagation of the output in the ANN 113. Retrieving model data from the ANN model area 283 can be performed in parallel with storing external input data in the neuron input area 285. Therefore, the processor 233 of the computer system 231 of the vehicle 211 does not have to explicitly send a read command to retrieve model data from the ANN model area 283.

[0091] Similarly, in response to reading the output data of the neuron, the address map 291 can be used to calculate the address of the model parameters of the neuron stored in the ANN model area 283 and read the model parameters into the buffer 252 so that the inference engine 103 applies the internal input in the buffer 252 to perform the calculation of the output of the neuron. The calculated output can be provided as a response to the reading of the neuron output data without requiring the lidar sensor 101 to store the output data in the memory component. Therefore, the processor 233 and / or the inference engine 103 can control the calculation of the neuron by writing input to the neuron and / or reading output from the neuron.

[0092] Typically, the incoming external input data to the ANN 113 can be raw sensor data 221 directly generated by the sensor (e.g., 209) without processing by the processor 233 and / or the inference engine 103. Alternatively, indirect sensor data 221 that has been processed by the processor 233 for the ANN 113 according to the signal from the sensor 222 can be provided as external input data. The incoming external input data can be accepted in the host interface 105 and written to the neuron input area 285 in a cyclic manner, and automatically buffered in the buffer 252 so that the inference engine 103 generates neuron outputs using the model stored in the ANN model area 283. The output generated by the inference engine 103 can be further buffered as an internal input to further apply the model in the ANN model area 283. When the external output becomes available, the controller 251 can report the completion of the write request with an indication of the availability of the external output. Optionally, the controller 251 and / or the inference engine 103 can generate an internal read command to propagate a signal in the ANN 113 when the external output is generated. Alternatively, host processor 233 may control the propagation of signals in ANN 113 by selectively reading the outputs of neurons; and data storage device 212 may actively buffer data in buffer 252 that may be needed to accelerate ANN computations.

[0093] The communication protocol / interface can be configured to allow the lidar sensor to perform neural network acceleration in real time while reducing data traffic to the host system.

[0094] For example, a host processor (e.g., 233) of vehicle 211 can provide a write command to lidar sensor 101 to store a model of an artificial neural network in a model partition (e.g., 283).

[0095] In order to use the ANN model in classification and / or prediction, the host processor (e.g., 233) of the vehicle 211 can optionally flow the input data of the ANN 113 into the neuron input partition (e.g., 285). The inference engine 103 can automatically apply the radio image 107 from the tone detector 115 and the input data (if any) from the host processor 233 to the model stored in the ANN model partition (e.g., 283) according to the address map 291. The lidar sensor 101 makes the calculated output available for propagation in the ANN 113. Preferably, the calculated output is available to the inference engine 103 through the buffer 252 without storing the intermediate output in the memory component (e.g., 261 to 263). Therefore, data communication between the host processor (e.g., 233) and the lidar sensor 101 for conveying the output of the neuron can be reduced. When the output has been propagated to the output neuron in the ANN 113, the lidar sensor 101 can provide a response to the request from the host processor (e.g., 233). The response indicates that external output from the neuron in ANN 113 is available. In response, the host processor (e.g., 233) of vehicle 211 can optionally issue a read command for retrieving the external output for further processing.

[0096] Figure 5 FIG. 2 shows the communication between autonomous vehicle 211 and lidar sensor 101 according to one embodiment. For example, Figure 1 The laser radar sensor 101 shown in Figure 2 or 3 in the vehicle 211 as implemented in Figure 5 Communication shown in .

[0097] exist Figure 5 In the present invention, since some of the calculations involving the ANN 225 are performed by the inference engine 103 or the neural network accelerator 259 within the lidar sensor 101, a simplified set of instructions 151 for performing neural network calculations can be configured for one or more processors 233 of a host system (e.g., 231).

[0098] The sensor 222 of the vehicle 211 may generate an optional continuous input stream 155 as part of the sensor data 221 for the lidar sensor 101 of the vehicle 211. The sensor data 221 in the input stream 155 may be generated at fixed, predetermined time intervals (e.g., during operation of the vehicle 211) for sensor fusion in the lidar sensor 101. Alternatively, the operation of sensor fusion may be performed by the processor 233.

[0099] The timing circuit system 115 and the directional control 125 can provide data to form a stream of the lidar image 107. The lidar image 107 and the optional input stream 155 are applied to the input neurons in the ANN 113 stored in the ANN model area 283. The input neurons in the ANN 113 are configured to accept external inputs to the ANN 113; and the output neurons are configured to provide external outputs from the ANN 113.

[0100] One or more processors 233 may execute instructions 151 for processing sensor data 221 and some of the output data 157 from the neuronal output region 287 of the lidar sensor 101 .

[0101] For example, the one or more processors 233 may write the sensor data 221 as part of the input stream 155 into the neuron input region 285 and read the output data 157 generated by the inference engine 103 using the ANN data in the model region 283 from the neuron output region 287 .

[0102] The lidar sensor 101 stores the input stream 155 into the neuron input area 285 in a cyclic manner, wherein the earliest input set currently stored in the neuron input area 285 corresponding to the earliest time instance of data sampling for the data set is erased to store the latest input set.

[0103] For each input data set, the inference engine 103 applies the model of the ANN 113 stored in the ANN model area 283. The inference engine 103 (or one or more processors 233) can control the propagation of signals within the neural network. When the output neuron of the ANN 113 generates its output in response to the input data set, the lidar sensor 101 can provide an indication to the processor 233 that the neuron output is ready for retrieval. The indication can be configured to respond to a request from the one or more processors 233 to write the input data set to the neuron input area 285. The one or more processors 233 can optionally retrieve the output data 157 (e.g., according to conditions and / or criteria programmed in the instructions).

[0104] In some embodiments, trigger parameters are configured in the lidar sensor 101. When the output parameters in the external output 167 meet the requirements specified by the trigger parameters, the lidar sensor 101 provides a response to a request from the one or more processors 233 to write the input data set to the neuron input area 285.

[0105] Figure 6 1 shows the communication within the lidar sensor 101 according to one embodiment. For example, Figure 6 The communication can be Figure 1The laser radar sensor 101 shown in FIG.

[0106] exist Figure 6 In the example, model area 283 stores the model 163 of ANN 113. In response to receiving a lidar image 107 and an optional set of external inputs 165 within a time instance from the input stream 155 in buffer 252, data storage device 212 can write the external inputs 165 to input area 285, while retrieving a neuron model 162 containing a portion of the ANN model 163, the portion of the ANN model corresponding to the parameters of the input neuron and / or the identity of the neuron connected to the input neuron. Buffer 252 allows inference engine 103 to combine neuron model 162, lidar image 107, and optional external input 175 to generate output 177 of the input neuron.

[0107] Typically, neuron outputs 177 may be included as part of internal outputs 166 for further propagation within the ANN 113 and / or as part of external outputs 167 of one or more processors 233 .

[0108] The internal output 166 is stored in the buffer 252 as an internal input 166 for further propagation in the ANN 113 in a manner similar to the generation of the neuron output 177 from the external input 165. For example, a portion of the internal input 166 can cause the controller 251 and / or the inference engine 103 to retrieve the corresponding neuron model 162 associated with the internal input so that the internal input is applied to the corresponding neuron model 162 in the inference engine 103 to generate its neuron output 177.

[0109] When a complete set of external outputs 167 is available in buffer 252 , external outputs 167 may be stored in output area 287 .

[0110] Optionally, the lidar sensor 101 (or storage device 212) does not store each set of external outputs 167 corresponding to a stored set of external inputs 165 sampled at a time instance. For example, the lidar sensor 101 (or storage device 212) can be configured to store one set of external outputs 167 each time a predetermined number of sets of external inputs (e.g., 165) have been counted. Alternatively or in combination, the one or more processors 233 can determine whether to store the external outputs 167. For example, the lidar sensor 101 can be configured to store the external outputs 167 in response to the one or more processors 233 retrieving the external outputs 167 for further processing. For example, the lidar sensor 101 (or storage device 212) can be configured to store the external outputs 167 in response to a write command from the one or more processors 233 after the external outputs 167 are processed in the one or more processors 233.

[0111] Inference engine 103 and / or neural network accelerator 259 may be configured to implement and / or accelerate a spiking neural network using one or more crossbar arrays of memristors.

[0112] Resistive random access memory (ReRAM or RRAM) works by changing the resistance across a dielectric solid-state material, which may be referred to as a memristor. In general, a memristor is a two-terminal electrical component with a non-constant resistance. Its resistance may change based on the history of the current flowing through the memristor and / or the history of the voltage applied to the memristor. When the power to the memristor is turned off, the memristor may remember its most recent resistance. Therefore, memristors may be used to implement non-volatile memory.

[0113] The memristor crossbar array can be configured to perform a multiplication and accumulation (MAC) operation through an analog circuit system. The current that reaches the bit line through the word line and through a group of memristors in the crossbar array is summed in the bit line, which corresponds to an accumulation operation. The current corresponds to the multiplication of the voltage applied to the word line and the parameter related to the resistance of the memristor, which corresponds to a multiplication operation. The current in the bit line can be compared with the spike threshold to determine whether the activation level of the neuron represented by the bit line has reached the spike level. The memristor array can be connected to the bit line respectively and programmed to have a threshold corresponding to the activation level threshold of the spike neuron. A current detector can be configured for each memristor connected to the output of the bit line to determine whether the level of the current in the bit line corresponds to a level exceeding the memristor threshold. The detection result of the current detector can represent a spike response in the output of the spike neuron represented by the bit line. The spike response can be routed to another part of the SNN or to the output of the SNN.

[0114] Alternatively, an analog-to-digital converter (ADC) can be used to measure the current in the bit line and convert it into digital data for spike detection in the comparator. However, the ADC consumes more space and energy than the memristor and current detector. Therefore, using a memristor array and current detector to eliminate the need for an ADC when detecting spikes can improve the space and energy efficiency of the SNN implementation.

[0115] Figure 7 A spiking neural network block implemented using memristors is shown according to one embodiment.

[0116] exist Figure 7In the embodiment of the present invention, a memristor crossbar array 301 receives an input voltage spike 303 to generate a current through a memristor array 305, which can act as a switch array. Each memristor in the array 305 is programmed to have a threshold for spike detection. When the current flowing through the memristor is less than the threshold, the memristor has substantially the same resistance; and the switch implemented by the memristor is considered to be in the off state. However, when the current flowing through the memristor reaches its threshold, the resistance of the memristor decreases (e.g., close to zero), causing the current flowing through the memristor to increase significantly and spike; and in this case, the switch implemented by the memristor is considered to be in the on state. Once the switch implemented by the memristor is in the on state, the current flowing through the memristor can increase to at least a predefined level, which is independent of the threshold of the memristor and is higher than the current that can flow through the memristor in the array 301 before reaching the threshold of the memristor in the array 301.

[0117] The current level detector 307 may be configured to detect whether the current flowing through the memristors in the array 305 has reached a level corresponding to the on-state of the memristors in the array 305. The detection may be performed based on detecting whether at least a predefined level of current flows through the corresponding memristors in the array 305. Thus, the current detection operation of the current level detector 307 is independent of the thresholds of the memristors in the array 305. Based on whether at least a predefined level of current is present in the memristors in the array, the corresponding detector in the current level detector 307 array generates a digital signal indicating whether a spike in current has been detected for the memristors in the array 305. Collection of the outputs from the current level detector 307 provides an output spike 309.

[0118] Figure 8 The use of spiking in a spiking neural network implemented using memristors is demonstrated using current spike detection through a memristor array according to one embodiment. For example, Figure 7 The crossbar array 301 and the memristor array 305 can be configured as follows: Figure 8 Implemented as shown in .

[0119] exist Figure 8 In the embodiment of the present invention, each of the memristors in the crossbar array 301 is connected between a word line (e.g., 331) and a bit line (e.g., 341). The word lines 331, ..., 333, 335, ..., 337 are configured to receive an input voltage; the bit lines 341, 343, ..., 345 are configured to provide an output current; and the memristor array 305 is configured to generate an output current spike corresponding to the spike of the neuron represented by the bit lines 341, 343, ..., 345 according to the activation level threshold of the spiking neuron.

[0120] For example, when a voltage is applied to word line 331, the voltage generates a current that flows through memristors 311, 321, ... to bit lines 341, 343, ..., 345, respectively. The contribution of the voltage applied to word line 331 to the current in bit lines 341, 343, ..., 345 is proportional to the weight and response of the neuron to the input represented by the voltage applied to word line 331. The weight and response of the neuron can be implemented by programming the resistance of memristors 311, 321, ..., respectively.

[0121] The bit lines 341, 343, ..., 345 sum the currents contributed by the voltages applied to the word lines 331, ..., 333, 335, ..., 337 to the bit lines 341, 343, ..., 345. Thus, the currents in the bit lines 341, 343, ..., 345 correspond to the product of the weight and response of the neuron (implemented by the programmed resistance of the crossbar array 301) and the voltages of the word lines 331, ..., 333, 335, ..., 337 representing the inputs to the neurons.

[0122] For example, contributions of the voltage on word lines 331, ..., 333, 335, ..., 337 to bit line 341 are summed by currents flowing from word lines 331, ..., 333, 335, ..., 337 through memristors 311, ..., 313, 315, ..., 317; and contributions of the voltage on word lines 331, ..., 333, 335, ..., 337 to bit line 343 are summed by currents flowing from word lines 331, ..., 333, 335, ..., 337 through memristors 321, ..., 323, 325, ..., 327; etc.

[0123] Thus, the crossbar array 301 performs a multiply and accumulate (MAC) operation by converting voltages on word lines 331 , . . . , 333 , 335 , . . . , 337 into currents on bit lines 341 , 343 , . . . , 345 .

[0124] exist Figure 8 , 345, the memristor array 305 is used to generate current spikes based on the thresholds of the memristors 319, 329, ... connected to the bit lines 341, 343, ..., respectively.

[0125] For example, memristor 319 is connected to bit line 341 and is programmed to have a threshold value corresponding to a threshold value of an activation level of a spike neuron. When the current on bit line 341 is less than the current threshold value, memristor 319 has a first resistance corresponding to an off state; and when the current on bit line 341 reaches the current threshold value, memristor 319 has a second resistance corresponding to an off state, which is significantly lower than the first resistance.

[0126] The current level detector 351 is configured to detect whether the current on the bit line 341 is at a level corresponding to the on-state of the memristor 319 , which indicates whether the spiking neuron corresponding to the bit line 341 generates a spike.

[0127] Therefore, the outputs of the current level detectors 351 , 353 , . . . , 355 correspond to the output spikes 309 of the bit lines 341 , 343 , . . . , 345 .

[0128] For example, a spike series may be applied to word line 331. The spike series is a time sequence of voltages applied to word line 331. The voltages in the spike series may be arranged at fixed time intervals. The voltages in the spike series may be selected from two predetermined voltages. One of the voltages corresponds to the presence of a spike at a certain time instance in the spike series; and the other voltage corresponds to the absence of a spike at a certain time instance in the spike series.

[0129] For example, a data item containing multiple bits can be converted into a series of spikes over the same number of time instances. The voltage applied at each of the time instances is based on the value of the corresponding bit in the data item.

[0130] When a set of spike series are applied to a set of word lines 331, ..., 333, 335, ..., 337, respectively, the outputs of the current level detectors 351, 353, ..., 355 generate a set of output spike series, which can be further processed by another spike neural network block.

[0131] The resistance value of the memristor in the crossbar array 301 can be set to model the behavior of the spiking neural network. The resistance value of the memristor in the crossbar array 301 can be changed by programming the memristor by selectively applying current and / or voltage. For example, after selecting a memristor (e.g., 311) for programming, the voltage applied to the memristor (e.g., 311) can be increased to put the memristor (e.g., 311) in an on state; and then a programming pulse can be applied to the memristor (e.g., 311) to adjust the resistance of the memristor (e.g., 311) to a desired value. After programming, the memristor (e.g., 311) remembers its programmed resistance during its off state until the memristor (e.g., 311) is selected again for programming in the on state. A separate circuit can be used to select a memristor (e.g., 311) and program the resistance of the memristor (e.g., 311).

[0132] Fig. 9 A spiking neural network is shown using a network implementation of spiking neural network blocks according to one embodiment.

[0133] exist Fig. 9 In the embodiment, a plurality of SNN blocks 361, 363, 365, ... are configured on the integrated circuit die. Each SNN block (e.g., 361, 363 or 365) can be Figure 7 and / or Figure 8 Each SNN block (e.g., 361, 363, or 365) is configured with associated routing logic (e.g., 362, 364, or 366) to provide input spike data (e.g., 303) from the communication network 373 to the SNN block and transmit its output spike data (e.g., 309) to other SNN blocks through the communication network 373.

[0134] For example, each SNN block (e.g., 361, 363, or 365) and router 371 may have a unique address for data transmission through communication network 373. The input spike series address to the SNN block (e.g., 361) may be received by the routing logic (e.g., 362) of the SNN block, which generates corresponding voltages for the word lines (e.g., 331, ..., 333, 335, ..., 337) of the block (e.g., 301) to cause the block (e.g., 301) to generate output spikes 309 through current level detectors (e.g., 351, 353, ..., 355) connected to the bit lines (e.g., 341, 343, ..., 345) of the SNN block (e.g., 301). The routing logic (e.g., 362) then transmits the output spikes to router 371. Router 371 is configured to route the output spike to the next one or more blocks (e.g., 363 and / or 365) based on the neural connectivity in the spiking neural network.

[0135] The router 371 is configured on the integrated circuit die to route input / output spikes to or from the SNN blocks (e.g., 361, 363, and / or 365) (e.g., according to a routing table that is configured according to the neural connectivity in the spiking neural network). Thus, the SNN blocks (e.g., 361, 363, or 365) on the integrated circuit die can be reconfigured by the router 371 to model different spiking neural networks with different neuron connectivity configurations.

[0136] Fig.10 demonstrated the processing of spiking neural networks by using memristors to detect spikes and routing the detected spikes between memristor blocks.

[0137] exist Fig.10 In FIG. 1 , input data 381 to a spiking neural network is converted by a spike encoder 383 to generate spikes 385 as input to one or more SNN blocks 387. Each SNN block in the SNN block 387 may be Figure 7 or Figure 8, wherein an output spike in the current in the bit lines (e.g., 341, 343, ..., 345) is detected by a memristor array 305 whose threshold is configured according to the threshold of the activation level of the spiking neuron. The current level detectors 351, 353, ..., 355 generate digital data of the output spike 309, which can be provided as input spikes 389 to one or more additional SNN blocks 391. Additional layers of SNN blocks can be added in a manner similar to the use of the second layer block 391.

[0138] Fig.10 The blocks used in (e.g., 387 and 391) may be blocks 361, 363, 365, ... configured on an integrated circuit die. Some of the blocks 361, 363, 365, ... on the integrated circuit die may be reconfigured to model different portions of spiking neurons at different times in the network.

[0139] For example, the first layer SNN blocks 387 may initially be configured on blocks 361, 363, 365 by programming the resistance of the memristors (e.g., 311 to 317, 321 to 327, ...) in the crossbar array 301 of each block (e.g., 301) and by programming the thresholds of the memristors (e.g., 319, 329, ...) for spike detection. Subsequently, some of the blocks 361, 363, 365 may be reprogrammed as SNN blocks 391 for second layer processing.

[0140] Fig.11 A method of a lidar sensor according to one embodiment is shown. For example, Fig.11 The method can be Figure 2 or 3 vehicles in 211 Figure 1 or 5 is implemented in the lidar sensor 101.

[0141] At box 401 , the lidar sensor 101 generates a lidar image 107 .

[0142] For example, the laser radar sensor 101 may include a laser pulse generator 117, a directional control 125, and a beam guide 119 coupled to the laser pulse generator 117 and the directional control 125 to project laser pulses in different directions. The laser radar sensor 101 may include a light sensor 123 and a light receiver 121, wherein the light receiver is configured to guide the laser pulse reflected back to the laser radar sensor 101 to the light sensor 123. The timing circuit system of the laser radar sensor 101 is configured to determine the time gap between the laser pulse projected in a certain direction by the beam guide 119 and the laser pulse reflected back to the light sensor 123.

[0143] The memory 111 of the lidar sensor 101 is configured to store at least one lidar image 107 and an artificial neural network 113. The lidar image 107 represents the surrounding environment of the lidar sensor 101 as seen through the laser pulses. For example, the lidar image 107 identifies the properties of multiple points that will reflect subsequent pulses back to the light sensor 123 based on input from the directional control 125 and the timing circuit system 115, such as the direction to the point, the range to the point, the position of the point, the intensity of the reflected laser pulse, etc.

[0144] For example, the beam guide may be a solid-state device with no movable mechanical components. Alternatively, the beam guide may be implemented by micro-mirrors.

[0145] For example, the memory 111 and the inference engine 103 may be configured on at least one first integrated circuit die; and the light sensor 123 may be configured on a second integrated circuit die. The at least one first integrated circuit die and the second integrated circuit die may be stacked and enclosed in the same integrated circuit package to provide high communication bandwidth between the memory 111 and the inference engine 103. For example, the memory 111 and the inference engine 103 in the at least one first integrated circuit die and the light sensor 123 in the second integrated circuit die may be connected by at least through-silicon vias.

[0146] At box 403 , at least one lidar image 107 and an artificial neural network 113 for processing the lidar image 107 are stored in the memory 111 of the lidar sensor 101 .

[0147] For example, the artificial neural network 113 may include a spiking neural network, or a deep neural network, or any combination thereof.

[0148] At box 405 , the inference engine 103 of the lidar sensor 101 analyzes the lidar image 107 using the artificial neural network 113 to generate an inference result 109 .

[0149] For example, the lidar sensor 101 may include a host interface 105 configured to provide inference results 109 to the computer system 231 of the vehicle 211; and, because the lidar sensor 101 is configured to provide inference results 109 rather than lidar images 107, the lidar sensor 101 may be configured to generate lidar images 107 at a rate and resolution that exceeds the communication bandwidth between the host interface 105 and the computer system of the vehicle 211.

[0150] For example, the lidar image 107 may be analyzed to determine the identity or classification of an object captured in the lidar image 107 .

[0151] For example, LiDAR image 107 may be analyzed by ANN 113 to extract portions showing objects and remove other portions to reduce the amount of image data to be further analyzed for sensor fusion.

[0152] For example, a portion of the sensor data stream 221 may be provided as an input stream to the lidar sensor 101; and the inference engine 103 uses the ANN 113 to perform sensor fusion by combining the lidar image 107 or a portion of the captured object shown in the lidar image with the portion of the sensor data stream 221 to generate an inference result 109.

[0153] Optionally, the analysis of the LiDAR image 107 by the inference engine 103 may include matrix multiplication and accumulation operations performed using the memristor crossbar array 301. The memristors in the crossbar array 301 are configured to store parameters of the artificial neural network 113.

[0154] At box 407 , the interface 105 of the lidar sensor 101 transmits the inference result 109 to the computer system 231 of the vehicle 211 to generate a control signal for autonomous driving of the vehicle 211 based on the inference result 107 and the sensor data stream 221 generated by the sensor 222 configured on the vehicle 211 .

[0155] For example, the ANN 113 may include a spiking neural network (SNN) implemented using a crossbar array 301 of first memristors (311 to 317; 321 to 327; ...). Each of the first memristors is connected between a word line (e.g., 331, ..., 333, 335, ..., or 337) and a bit line (e.g., 341, 343, ..., or 345) to convert a voltage applied to the word line into a current in the bit line; and each corresponding bit line (e.g., 341) of the crossbar array 301 sums the current converted by the columns (e.g., 311 to 317) of the first memristors to the corresponding bit line (e.g., 341).

[0156] Further, the SNN is implemented using second memristors 319, 329, ... respectively connected to the bit lines 341, 343, ... of the crossbar array 301. Each of the second memristors 319, 329, ... is programmed to have a current threshold according to the activation level threshold of the spike neuron. Therefore, the spike of the neuron can be detected by detecting a high current level (e.g., higher than a predetermined level) higher than the threshold of the second memristor 319, 329, ...

[0157] Optionally, the inference engine 103 includes a neural network accelerator 259 implemented using a memristor crossbar array 301. The memristor crossbar array is configured to perform matrix multiplication and accumulation operations by converting voltages on word lines 331, ..., 333, 335, ..., 337 of the crossbar array 301 into currents summed in bit lines 341, 343, ..., 345 of the crossbar array 301. The memristors (311 to 317; 321 to 327; ...) in the crossbar array 301 are configured to store parameters of the artificial neural network 113; and the resistance of the memristors (311 to 317; 321 to 327; ...) is programmed according to the parameters of the artificial neural network 113.

[0158] Typically, server 219 , computer system 231 , data storage device 212 , and lidar sensor 101 may each be implemented as one or more data processing systems.

[0159] The present disclosure includes methods and apparatus for performing the methods described above, including data processing systems for performing these methods, and computer-readable media containing instructions that, when executed on a data processing system, cause the system to perform these methods.

[0160] A typical data processing system may include an interconnect (eg, a bus and system core logic) that interconnects one or more microprocessors and memory. A microprocessor is typically coupled to a cache memory.

[0161] The interconnect interconnects one or more microprocessors and memory together, and also interconnects the one or more microprocessors and memory to one or more input / output (I / O) devices through one or more I / O controllers. The I / O devices may include display devices and / or peripheral devices, such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (such as a printer, scanner, mouse, and / or keyboard) are optional.

[0162] The interconnect may include one or more buses that are interconnected through various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB (Universal Serial Bus) adapter for controlling USB peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0163] The memory may include one or more of the following: ROM (read only memory), volatile RAM (random access memory), and non-volatile memory such as a hard drive, flash memory, etc.

[0164] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires continuous power to refresh or maintain the data in the memory. Nonvolatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other types of memory systems that maintain data even after the system loses power. Nonvolatile memory can also be random access memory.

[0165] The non-volatile memory may be a local device directly coupled to the rest of the components in the data processing system. A network storage device may also be used that is remote from the system and coupled to the data processing system through a network interface such as a modem or Ethernet interface.

[0166] In the present disclosure, in order to simplify the description, some functions and operations are described as being performed by software codes or caused by software codes. However, such expressions are also used to specify that a function is generated by a processor (such as a microprocessor) executing code / instructions.

[0167] Alternatively or in combination, the functions and operations described herein may be implemented using special purpose circuitry, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA), with or without software instructions. Embodiments may be implemented using hardwired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source of instructions executed by a data processing system.

[0168] Although one embodiment may be implemented on fully functional computers and computer systems, various embodiments may be distributed as computer products in various forms and may apply regardless of the specific type of machine or computer readable media used to actually effect the distribution.

[0169] At least some aspects disclosed may be embodied at least in part in software. That is, the techniques may be performed in a computer system or other data processing system in response to its processor (e.g., a microprocessor) executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0170] The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a "computer program." A computer program typically includes one or more instructions that are placed at various times in various memories and storage devices in the computer and, when read and executed by one or more processors in the computer, cause the computer to perform the necessary operations to perform the elements involved in various aspects.

[0171] Machine-readable media can be used to store software and data that, when executed by a data processing system (e.g., a computer system 231 in a vehicle 211, a server 219, a system with one or more processors 233, a device 101 with an inference engine 103 and / or a controller 251), causes the system to perform various methods. Executable software and data can be stored in various places, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache (e.g., memory 111, memory 235, data storage device 212, buffer 252). Portions of this software and / or data can be stored in any of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or in the same communication session. Data and instructions can be obtained in full before executing the application. Alternatively, portions of data and instructions can be obtained dynamically and in a timely manner when execution is required. Therefore, it is not necessary for data and instructions to be all located on machine-readable media at a specific time instance.

[0172] Examples of computer readable media include, but are not limited to, non-transitory, recordable, and non-recordable media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CD ROM), digital versatile disk (DVD), etc.), etc. A computer readable medium may store instructions.

[0173] The instructions may also be embodied in digital and analog communication links for electrical, optical, acoustic or other forms of propagation signals (such as carrier waves, infrared signals, digital signals, etc.). However, propagation signals (such as carrier waves, infrared signals, digital signals, etc.) are not tangible machine-readable media and are not configured to store instructions.

[0174] Generally, a machine-readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible to a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors, etc.).

[0175] In various embodiments, hard-wired circuitry can be used in combination with software instructions to implement the techniques. Therefore, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source of instructions executed by a data processing system.

[0176] The above description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described to avoid obscuring the description. References to one embodiment or embodiments in this disclosure are not necessarily references to the same embodiment; and, such references mean at least one.

[0177] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope as set forth in the following claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A laser radar sensor, comprising: Laser pulse generator; Directional controls; a beam guide coupled to the laser pulse generator and the directional control to project laser pulses in different directions; Light sensor; a light receiver configured to direct laser pulses reflected back to the lidar sensor to the light sensor; timing circuitry configured to determine a time gap between a respective laser pulse projected by the beam guide and the respective laser pulse reflected back to the light sensor; a memory configured to store a lidar image and an artificial neural network, wherein the lidar image identifies attributes of a plurality of points that reflected laser pulses back to the light sensor based on input from the directional control and the timing circuitry; an inference engine configured to analyze the lidar image using the artificial neural network and generate an inference result; as well as a host interface configured to provide the inference result to a computer system of the vehicle; The memory and the inference engine in at least one of the first integrated circuit die and the light sensor in the second integrated circuit die are connected by at least through-silicon vias.

2. The lidar sensor of claim 1, wherein the beam guide is a solid-state device having no movable mechanical components.

3. The lidar sensor of claim 1, wherein the at least one first integrated circuit die and the second integrated circuit die are enclosed in the same integrated circuit package.

4. The lidar sensor of claim 3, wherein the artificial neural network comprises a spike neural network, or a deep neural network, or any combination thereof.

5. The lidar sensor of claim 3, wherein the lidar sensor is configured to generate lidar images at a rate and resolution that exceeds a communication bandwidth between the host interface and the computer system of the vehicle.

6. The lidar sensor of claim 3, wherein the attributes include a reflection intensity of the point, a range to the point, and a direction to the point.

7. The lidar sensor of claim 3, wherein the inference engine comprises a crossbar array of memristors configured to perform matrix multiplication and accumulation operations.

8. The lidar sensor of claim 7, wherein the memristors in the crossbar array are configured to store parameters of the artificial neural network; and the multiplication and accumulation operations are performed by the memristors converting voltages on word lines of the crossbar array into currents summed in bit lines of the crossbar array.

9. A vehicle comprising: multiple controls for steering, braking, and acceleration; a set of sensors configured to generate a sensor data stream; A laser radar sensor, the laser radar sensor comprising: Light sensor; LiDAR image generator; a memory configured to store the lidar image and the artificial neural network; an inference engine configured to analyze the lidar image using the artificial neural network and generate an inference result; and Communication interface; a computer system coupled to the communication interface to implement an advanced driver assistance system to operate the control based on the inference result and the sensor data stream, The memory and the inference engine in at least one of the first integrated circuit die and the light sensor in the second integrated circuit die are connected by at least through-silicon vias.

10. The vehicle of claim 9, wherein the inference identifies an object in the lidar image that reflects a lidar signal.

11. The vehicle of claim 10, wherein the inference result comprises a portion extracted from the lidar image using the artificial neural network.

12. The vehicle of claim 9, wherein a portion of the sensor data stream is provided to the lidar sensor to perform sensor fusion based on the lidar image and the portion of the sensor data.

13. The vehicle of claim 9, wherein the inference engine comprises a memristor crossbar array configured to store parameters of the artificial neural network and perform matrix multiplication and accumulation operations in the artificial neural network.

14. The vehicle of claim 9, wherein the at least one first integrated circuit die and the second integrated circuit die are enclosed in a same integrated circuit package.

15. A method comprising: Generate a lidar image by using a lidar sensor; storing at least one lidar image and an artificial neural network in a memory of the lidar sensor; Analyzing the lidar image using the artificial neural network through an inference engine of the lidar sensor to generate an inference result; as well as transmitting the inference result to a computer system of the vehicle through an interface of the lidar sensor, wherein the computer system is configured to generate a control signal for autonomous driving of the vehicle based on the inference result and a sensor data stream generated by a sensor configured on the vehicle, The memory and the inference engine in at least one first integrated circuit die and the optical sensor of the lidar sensor in the second integrated circuit die are connected by at least through-silicon vias.

16. A method according to claim 15, wherein the analysis includes determining the identity or classification of objects captured in the lidar image.

17. The method of claim 16, wherein the analyzing comprises extracting a portion showing the object from the lidar image.

18. The method according to claim 17, further comprising: receiving, in the lidar sensor, a portion of the sensor data stream; as well as A sensor fusion operation is performed by the inference engine using the artificial neural network based on the lidar image and the portion of the sensor data stream to generate the inference result.

19. The method of claim 15, wherein the analyzing comprises performing matrix multiplication and accumulation operations using a crossbar array of memristors configured to store parameters of the artificial neural network.

20. The method of claim 15, wherein the at least one first integrated circuit die and the second integrated circuit die are enclosed in a same integrated circuit package.

Images