Storage device with neural network accelerator for automotive predictive maintenance

Abstract

The present application relates to a storage device with a neural network accelerator for predictive maintenance of automobiles. For example, a data storage device of a vehicle includes: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; at least one storage media component having a non-volatile memory to store at least a portion of the sensor data stream; a controller; and a neural network accelerator coupled to the controller. The neural network accelerator is configured to perform at least a portion of a calculation based on an artificial neural network and the sensor data stream to predict maintenance service of the vehicle.

Description

Technical Field

[0001] At least some embodiments disclosed herein relate generally to maintenance services for vehicles, and more particularly, but not limited to, to a storage device with a neural network accelerator for predictive maintenance of motor vehicles. Background Art

[0002] Conventionally, automobile maintenance is scheduled based on predetermined operational milestones. For example, a routine maintenance service may be scheduled every three or six months, or after a predetermined driving distance (e.g., 3,000 miles, 6,000 miles, or 15,000 miles).

[0003] When a component of a motor vehicle breaks down or malfunctions during operation of the vehicle, the accident can be a safety hazard. After the accident occurs, a service trip for the vehicle will soon be scheduled at an inconvenient time.

[0004] Recent developments in the field of autonomous driving technology allow computing systems to operate control elements of a motor vehicle without the assistance of a human operator of the vehicle, at least under some conditions.

[0005] For example, sensors (e.g., cameras and radars) may be installed on a motor vehicle to detect conditions of the vehicle's surroundings as it travels on a road. A computing system installed on the vehicle analyzes the sensor inputs to identify the conditions and generates control signals or commands to automatically adjust the direction and / or speed of the vehicle, with or without any input from a human operator of the vehicle.

[0006] In some arrangements, when the computing system identifies a situation in which the computing system may not be able to continue operating the vehicle in a safe manner, the computing system alerts the human operator of the vehicle and requests the human operator to take over control of the vehicle and drive manually, rather than allowing the computing system to automatically drive the vehicle.

[0007] U.S. Patent No. 9,533,579, entitled “Electronic Control Apparatus for Electrically-Driven Vehicle” and issued on January 3, 2017, discloses an electronic control apparatus for a vehicle with a self-diagnostic function.

[0008] Autonomous driving and / or advanced driver assistance systems (ADAS) often involve artificial neural networks (ANNs) for recognizing events and / or objects captured in sensor inputs.

[0009] Typically, an artificial neural network (ANN) uses a network of neurons to process inputs to the network and generate outputs from the network.

[0010] For example, each neuron in the network receives a set of inputs. Some inputs to neurons may be outputs of certain neurons in the network; and some inputs to neurons may be inputs provided to the neural network. The input / output relationships among neurons in the network represent neuronal connectivity in the network.

[0011] For example, each neuron may have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function may be in the form of a step function, a linear function, a logarithmic sigmoid function, etc. Different neurons in the network may have different activation functions.

[0012] For example, each neuron may produce a weighted sum of its inputs and its bias and then produce an output that is a function of the weighted sum computed using the neuron's activation function.

[0013] The relationship between the input(s) and output(s) of an ANN is typically defined by an ANN model, which includes data representing the connectivity of the neurons in the network and the bias, activation function, and synaptic weights of each neuron. Using a given ANN model, a computing device calculates the output(s) of the network from a given set of inputs to the network.

[0014] For example, input to the ANN network may be generated based on camera input; and output from the ANN network may be recognition of items such as events or objects.

[0015] A spiking neural network (SNN) is an ANN that closely mimics a natural neural network. When the activation level of the neuron is high enough, the SNN neuron produces a spike as an output. The activation level of the SNN neuron mimics the membrane potential of a natural neuron. The output / spike of the SNN neuron can change the activation level of other neurons that receive the output. The current activation level of the SNN neuron, which varies over time, is usually modeled using differential equations and is considered to be the state of the SNN neuron. Incoming spikes from other neurons push up the activation level of the neuron to reach the spike threshold. Once a neuron reaches a spike, its activation level is reset. Before reaching a spike, the activation level of the SNN neuron decays over time, as controlled by the differential equation. The temporal element in the behavior of the SNN neuron makes SNN suitable for processing spatiotemporal data. The connectivity of SNNs is usually sparse, which helps reduce the computational workload.

[0016] Typically, an ANN can be trained using supervised methods, in which parameters in the ANN are adjusted to minimize or reduce the error between known outputs produced by corresponding inputs and calculated outputs produced by applying the inputs to the ANN. Examples of supervised learning / training methods include reinforcement learning and error correction learning.

[0017] Alternatively or in combination, the ANN may be trained using unsupervised methods, where the exact outputs produced by a given set of inputs are not known until training is complete. The ANN may be trained to classify items into multiple categories, or to classify data points into clusters.

[0018] A variety of training algorithms can be used for complex machine learning / training paradigms. Summary of the invention

[0019] One aspect of the present invention relates to a data storage device for a vehicle, comprising: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; at least one storage media component having a non-volatile memory to store at least a portion of the sensor data stream; a controller; and a neural network accelerator coupled to the controller, the neural network accelerator being used to perform at least a portion of a calculation based on an artificial neural network and the sensor data stream to predict maintenance service for the vehicle.

[0020] In another aspect, the present invention relates to a storage media component comprising: an input / output interface configured to receive a sensor data stream from at least one sensor configured on a vehicle; at least one media unit having a non-volatile memory to store at least a portion of the sensor data stream; control logic; and a neural network accelerator coupled to the control logic, the neural network accelerator being used to perform at least a portion of a calculation based on an artificial neural network and the sensor data stream during a calculation to predict maintenance service for the vehicle.

[0021] In a further aspect, the present invention relates to a method comprising: receiving, in a data storage device of a vehicle, a sensor data stream from at least one sensor configured on the vehicle; storing at least a portion of the sensor data stream in the data storage device; performing at least a portion of a calculation based on an artificial neural network and the sensor data stream by a neural network accelerator configured in the data storage device; and predicting maintenance service for the vehicle based on the calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0023] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.

[0024] Figure 2 An autonomous vehicle with a data storage device is shown according to one embodiment.

[0025] Figures 3 to 5The training of an artificial neural network for maintenance service prediction according to some embodiments is described.

[0026] Figure 6 A method of predictive maintenance according to one embodiment is presented.

[0027] Figure 7 A data storage device for accelerating neural network computations according to one embodiment is presented.

[0028] Figure 8 A storage media assembly for accelerating neural network computations is presented according to one embodiment.

[0029] Fig. 9 A method for accelerating neural network computations in a motor vehicle is presented according to one embodiment. DETAILED DESCRIPTION

[0030] At least some embodiments disclosed herein provide systems, methods, and apparatus for processing sensor data generated in a motor vehicle or another vehicle with or without an advanced driver assistance system (ADAS) to facilitate predictive maintenance.

[0031] Before a component of a motor vehicle fails or malfunctions during operation of the vehicle, it may be indicated whether the component requires replacement or maintenance. Such an indication may not be apparent to a typical driver or passenger. However, sensor data may be collected and analyzed to predict the probability of component failure. The prediction may be used to schedule maintenance service, which may reduce or eliminate the chance of an incident where a component of the vehicle fails or malfunctions during operation of the vehicle on the road. In addition, the prediction allows a service trip to be scheduled at a convenient time.

[0032] For example, sensors may be installed in an automotive system to collect data during its routine operation; and the sensor data may be used to predict whether and how soon a component will need to be replaced or maintained. The sensor data may be provided as input to an artificial neural network (ANN) (e.g., a spiking neural network (SNN)) of an artificial intelligence (AI) system to train itself (e.g., using unsupervised machine learning techniques) over a period of time when normal vehicle operation is expected. The training customizes the neural network to the specific operating environment(s) of the driver, passenger, or user of the vehicle and the personalized operating habits of the vehicle occupant(s). Subsequently, the artificial neural network may detect an abnormal condition when the operating data deviates from the normal pattern. The AI ​​system may be used to recommend maintenance services and / or identify components that may need to be replaced or maintained.

[0033] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.

[0034] Figure 1 The system includes a vehicle 111 having a data storage device 101. Optionally, the vehicle 111 has an advanced driver assistance system (ADAS) 105 and one or more sensors 103 that provide sensor data input to the ADAS 105 and / or the data storage device 101. The data storage device 101 is configured to use an artificial neural network (ANN) 125 to predict / identify the need for maintenance service based on the data collected by the sensor 103. The ADAS 105 can be omitted without affecting the predictive maintenance feature. In some implementations, at least a portion of the data generated by the sensor 103 is used both in the ADAS 105 for driver assistance and in the ANN 124 for maintenance prediction. Optionally, the output of the ANN 124 can be used for both the data storage device 101 and the ADAS 105.

[0035] The sensors 103 may include digital cameras, lidars, radars, ultrasonic sonars, brake sensors, speed sensors, acceleration sensors, airbag sensors, GPS (Global Positioning System) receivers, audio sensors / microphones, vibration sensors, force / stress sensors, deformation sensors, motion sensors, temperature sensors, etc. Some sensors 103 may be configured primarily to monitor the environment of the vehicle 111; and other sensors 103 may be configured primarily to monitor the operating conditions of one or more components of the vehicle 111, such as the internal combustion engine, exhaust system, electric motor, brakes, tires, battery, etc.

[0036] The output of the sensor(s) 103 over time is provided as a sensor data stream to the ADAS 105 and / or the ANN 125 to provide driver assistance (eg, autonomous driving) and maintenance prediction.

[0037] For example, the vehicle 111 may have a wireless communication device to communicate with a remote server 119 via wireless signals 113 and a communication network 117. The remote server 119 is typically configured at a location away from the road 102 on which the vehicle 111 operates. For example, the vehicle 111 may provide some sensor data 121 to the server 119 and receive updates of the ANN 125 from the server 119.

[0038] One example of a communication network 117 is a cellular telephone network having one or more base stations (e.g., 115) to receive wireless signals (e.g., 113). Another example of a communication network 117 is the Internet, where wireless local area network signals (e.g., 113) transmitted by the vehicle 113 are received in an access point (e.g., 115) for further communication to a server 119. In some embodiments, the vehicle 111 communicates with the server 119 using a communication link 107 to a satellite 109 or a communication balloon.

[0039] The server 119 may also communicate with one or more maintenance service facilities (e.g., 127) to receive maintenance service data 123 of a vehicle (e.g., 111). The maintenance service data 123 may include inspection records and / or service records of components of the vehicle (e.g., 111). For example, the inspection records and / or service records may indicate the degree of component wear, identification of failed or malfunctioning components, etc., of components inspected during service at a maintenance service facility (e.g., 127). The sensor data 121 and maintenance service data 123 of the vehicle (e.g., 111) in the time period before the service may be used to train the ANN 125 to predict the probability that the component requires maintenance service. The updated ANN 125 may be used to predict and recommend maintenance service for the vehicle 111 based on the sensor data 121 received in the most recent time period. Alternatively, the updated ANN 125 may be transmitted to the vehicle 111; and the vehicle 111 may use the data generated from the sensor 103 during the routine operation of the vehicle 111 to predict and recommend maintenance service.

[0040] The data storage device 101 of the vehicle 111 may be configured to record sensor data for a certain period of time that can be used in the ANN for predictive maintenance. Maintenance predictions typically last for a relatively long period of time (e.g., days, weeks, and / or months). In contrast, sensor data recorded for reviewing an accident, collision, or near-collision involving an autonomous vehicle typically lasts for a short period of time (e.g., 30 seconds to a few minutes). Therefore, a typical black box data recorder configured to record sensor data for review / analysis of an accident or collision is not sufficient for predictive maintenance.

[0041] Optionally, the data storage device 101 stores sensor data that derives the time period of a trip to a maintenance service facility (e.g., 127). The maintenance service facility (e.g., 127) can download the sensor data 121 from the data storage device 101 and provide the sensor data 121 and corresponding maintenance service data 123 to the server 119 to facilitate the training of the ANN 125.

[0042] Optionally or in combination, the data storage device 101 is configured with a machine learning module to customize and / or train the ANN 125 installed in the vehicle 111 for predictive maintenance.

[0043] For example, the machine learning module of the data storage device 101 may be used to calibrate the ANN 125 to take into account the typical / daily environment in which the vehicle 111 is operated and / or the driving preferences / habits of the driver(s) of the vehicle 111 .

[0044] For example, during a period of time when the vehicle is expected to operate in a typical / daily environment with healthy components, the sensor data generated by the sensors 103 can be used to train the ANN 125 to recognize sensor data patterns that represent trouble-free operation. Such patterns can vary for different vehicles (e.g., 111) based on their routine operating environments and the driving habits / characteristics of their drivers. The training allows the ANN 125 to detect deviations from the recognized normal patterns and report anomalies for maintenance prediction.

[0045] For example, the ANN 125 may include an SNN configured to classify time-based variations in sensor data and / or detect deviations from known patterns in sensor data of the vehicle 111 operating under normal / health conditions but in a personalized environment (e.g., a driver / passenger's daily route) and / or under personalized driving habits / patterns.

[0046] Figure 2 An autonomous vehicle 111 is shown with a data storage device 101 according to one embodiment. For example, Figure 1 Vehicles in the system 111 can use Figure 2 The automatic driving vehicle 111 is implemented.

[0047] Figure 2 The vehicle 111 is configured to have an advanced driver assistance system (ADAS) 105. The ADAS 105 of the vehicle 111 may have an artificial neural network (ANN) 125 for object detection, recognition, identification, and / or classification. The ANN 125 and / or another neural network (e.g., configured in the data storage device 101) may be used to predict the probability that a component of the vehicle 111 requires maintenance service (e.g., repair, replacement, or adjustment).

[0048] Preferably, the data storage device 101 is configured to at least partially process the sensor data for predictive maintenance while alleviating the computational burden on the processor 133 responsible for operating the ADAS 105 and / or other components such as the infotainment system 149 .

[0049] The vehicle 111 typically includes an infotainment system 149, a communication device 139, one or more sensors 103, and a computer system 131 connected to some controls of the vehicle 111 (e.g., a steering control 141 for the direction of the vehicle 111, a brake control 143 for stopping the vehicle 111, an acceleration control 145 for the speed of the vehicle 111, etc.). In some embodiments, Figure 1 The vehicle 111 in the system may have a similar configuration and / or similar components.

[0050] Some sensors 103 are required for the operation of ADAS 105; and some sensors 103 are used to collect data related to the health of components of the vehicle 111, which may not be used by ADAS 105. Optionally, the sensor data generated by the sensors 103 can also be used to predict the likelihood of an imminent component failure. This prediction can be used in ADAS 105 to take emergency action to put the vehicle in a safe state (e.g., by reducing speed and / or pulling into a parking space).

[0051] The computer system 131 of the vehicle 111 includes one or more processors 133 , the data storage device 101 , and a memory 135 storing firmware (or software) 147 , including computer instructions and data models for the ADAS 105 .

[0052] The one or more sensors 103 of the vehicle may include a visible light camera, an infrared camera, a lidar, a radar or sonar system, a peripheral sensor, a global positioning system (GPS) receiver, a satellite positioning system receiver, a brake sensor, and / or an airbag sensor. In addition, the sensors 103 may include an audio sensor (e.g., a microphone) configured to monitor noise from various components and locations in the vehicle 111, a vibration sensor, a pressure sensor, a force sensor, a stress sensor, and / or a deformation sensor configured to measure the load on the components of the vehicle 111, an accelerometer and / or a gyroscope sensor that measures the motion of some components of the vehicle 111, etc. Such sensors can be used to monitor the operating state and / or health of the components for predictive maintenance.

[0053] The sensor(s) 103 may provide a real-time sensor data stream to the computer system 131. The sensor data generated by the sensors 103 of the vehicle 111 may include images of objects captured using a camera that uses light visible to the human eye for imaging or a camera that uses infrared light for imaging, or a sonar, radar, or LIDAR system. The image data obtained from at least one sensor of the vehicle is part of the collected sensor data for recording in the data storage device 101 and / or as an input to the ANN 125. For example, the camera can be used to obtain road information on which the vehicle 111 is traveling, which can be processed by the ANN 125 to generate control signals for the vehicle 111. For example, the camera can be used to monitor the operating status / health of components of the vehicle 111, which can be processed by the ANN 125 to predict or schedule maintenance services.

[0054] The sensor data generated by the sensors 103 of the vehicle 111 may include an audio stream that captures characteristics of sounds at locations on the vehicle 111 (e.g., locations near the engine, motor, drivetrain, wheels, doors, windows, etc.). The audio data acquired from at least one sensor 103 of the vehicle 111 may be part of the collected sensor data for recording in the data storage device 101 and / or as input to the ANN 125. For example, the audio stream may be used to monitor the operating state / health of components of the vehicle 111 (e.g., internal combustion engine, exhaust system, motor, brakes), which may be processed by the ANN 125 to predict or schedule maintenance service.

[0055] The infotainment system 149 may be used to present predicted or scheduled maintenance services. Optionally, the communication device 139 may establish a connection with a mobile device of the driver of the vehicle 111 to notify the driver of recommended maintenance services and / or recommended service data, to schedule an appointment, etc.

[0056] When the vehicle 111 is configured with the ADAS 105 , the output of the ADAS 105 may be used to control (eg, 141 , 143 , 145 ) the acceleration of the vehicle 111 , the speed of the vehicle 111 , and / or the direction of the vehicle 111 during autonomous driving.

[0057] Figures 3 to 5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.

[0058] exist Figure 3 In the example, the supervised machine learning module 171 is used to train the artificial neural network 125 to minimize the difference between the service predictions 129 generated from the sensor data 121 and the maintenance service data 123.

[0059] For example, the maintenance service data 123 may identify measured wear of a component over time to predict a time to a recommended service. The sensor data 121 may be used in the ANN 125 to generate a predicted time to a recommended service. The supervised machine learning module 171 may adjust the artificial neural network 125 to reduce / minimize the difference between the time predicted based on the sensor data 121 and the time calculated from the wear measurements.

[0060] For example, maintenance service data 123 may identify a component that was replaced or repaired in maintenance service facility 127. Sensor data 121 recorded during a period of time before the component was replaced or repaired may be used to calculate the time to replacement or repair. In addition, segments of the sensor data stream during the period of time before replacement or repair may be used in ANN 125 to generate predictions of the time to replacement or repair. Supervised learning 171 may be used to tune ANN 125 to reduce the predicted time to replacement or repair and the actual time to replacement or repair.

[0061] Figure 2 The supervised learning 171 may be applied to the server 119 based on the sensor data of the vehicle population and its maintenance service data 123 to generate a general ANN for the vehicle population.

[0062] Figure 2 The supervised learning 171 of the vehicle 111 can be applied to the vehicle 111 based on the sensor data of the vehicle and its maintenance service data 123 to produce a customized / personalized ANN for the vehicle population. For example, a general ANN can be initially used in the vehicle 111; and the sensor data of the vehicle 111 and its maintenance service data 123 can be used to further train the ANN 125 of the vehicle to customize / personalize the ANN 125 in the vehicle 111.

[0063] exist Figure 4 In the example embodiment, an unsupervised machine learning module 175 is used to train or improve the artificial neural network 125 to facilitate anomaly detection 173. The unsupervised machine learning module 175 is configured to adjust the classification, clustering, or identified pattern of the ANN (e.g., SNN) in the sensor data 121 so that the degree of deviation from the classification, clustering, or identified pattern in the sensor data 121 generated within a recent time period can be used to signal anomaly detection 173. Anomaly detection 173 allows the vehicle 111 to be scheduled for inspection at the maintenance service facility 127. Optionally, after inspection, the maintenance service data 123 can be used to apply supervised learning 171 to produce a more accurate prediction of the service, such as in Figure 3 middle.

[0064] Generally, it may be assumed that the vehicle 111 operates in a normal / healthy state for a certain period of time. For example, after initially delivering a new vehicle 111 for service, it may be assumed that the vehicle 111 provides trouble-free service for at least a certain period of time (e.g., several months). For example, after a period of time following a component replacement or repair, it may be assumed that the component provides trouble-free service for at least a certain period of time (e.g., several months or a year). Thus, sensor data 121 acquired during this period of time may be pre-classified as "normal" for use in, for example, Figure 4 Unsupervised learning in 175 or Figure 5 Supervised learning in 171 is used to train ANN125.

[0065] For example, sensor data 121 collected during a "normal" service period of a vehicle 111 or component may be classified into several clusters via unsupervised learning 175. Different clusters may correspond to different types of normal conditions (e.g., driving on different routes, driving on roads with different surface conditions, driving on days with different weather conditions, driving at different times of the day, driving on different days of the week, driving in different moods of the driver's driving habits). When subsequent sensor data 121 is classified outside of a "normal" cluster, an anomaly is detected.

[0066] Optionally, supervised machine learning 171 may be used to train the ANN 125, such as Figure 5 1. During a "normal" service period of the vehicle 111 or component, the expected classification 177 can be used to label the sensor data 121. Supervised learning 171 can be used to minimize the classification difference between the prediction 179 made from the sensor data 121 using the ANN 125 and the expected classification 177. In addition, when the sensor data 121 is known to be "abnormal" (e.g., in a maintenance service facility 127 or after a diagnosis is made by a user, driver, or passenger of the vehicle 111), the expected classification 177 can be changed to "abnormal" to further train the ANN 125 to directly recognize abnormalities (e.g., rather than relying on deviations from a known "normal" cluster to infer abnormalities).

[0067] Thus, the ANN 125 may be trained to identify anomalous sensor data and estimate the severity of the anomaly to schedule maintenance service.

[0068] Figure 6 A method of predictive maintenance according to one embodiment is presented. For example, Figure 6 The method can be found in Figure 1 or 2 in the data storage device 101 in the vehicle 111 or in Figure 2 is implemented in a computer system 131 in the vehicle 111 .

[0069] At box 201 , sensors (eg, 103 ) installed in a vehicle 111 generate a sensor data stream (eg, 121 ) during operation of the vehicle 111 on a road 102 .

[0070] At block 203, the sensor data stream (e.g., 121) is provided to an artificial neural network (ANN) 125. For example, the ANN 125 may include a spiking neural network (SNN).

[0071] At block 205 , an artificial neural network (ANN) 125 generates a prediction of maintenance service based on the sensor data stream (eg, 121 ).

[0072] At block 207 , a data storage device 101 disposed on the vehicle stores at least a portion of the sensor data stream (eg, 121 ).

[0073] At box 209 , an artificial neural network (ANN) is trained using a sensor data stream (eg, 121 ) collected over a predetermined period of time since the vehicle left the factory or maintenance service facility 127 .

[0074] For example, an artificial neural network (ANN) may be configured to identify components of vehicle 111 that require repair or replacement during maintenance service and / or identify an estimated time period until a component fails or malfunctions, or a suggested time period before a component fails or malfunctions to a recommended component maintenance service. Thus, the performance of predictive maintenance service may avoid incidents of component failure or malfunction while vehicle 111 is operating on road 102.

[0075] For example, sensor 103 may be a microphone mounted near the component, a vibration sensor attached to the component, a pressure sensor mounted in the component, a force or stress sensor mounted on or attached to the component, a deformation sensor attached to the component, or an accelerometer configured to measure a motion parameter of the component.

[0076] Optionally, the data storage device 101, the computer system 131 of the vehicle 111 and / or a server 119 remote from the vehicle may have a machine learning module configured to train an artificial neural network (ANN) 125 during a time period in which the vehicle 111 is assumed to be in a healthy state, such as a predetermined time period starting from when the vehicle 111 leaves a factory or maintenance service facility 127.

[0077] For example, the machine learning module may use unsupervised machine learning 175 to train the ANN 125 to recognize / classify normal patterns 121 of sensor data and thus have the ability to detect anomalies based on deviations from normal patterns, such as Figure 4 Alternatively, supervised machine learning 171 may be used, such as Figure 3 or as described in 5.

[0078] For example, unsupervised machine learning 175 may be applied by the data storage device 101 or computer system 131 of the vehicle during a predetermined time period in which the vehicle 111 or component is known to be operable without failure or degradation.

[0079] Alternatively or in combination, some sensor data 121 stored in the data storage device 101 of the vehicle 111 may be uploaded to the server 119 to train the ANN 125 .

[0080] In at least some embodiments disclosed herein, the data storage device 101 is configured to accelerate computation of an artificial neural network (ANN) 125 of the vehicle 111 .

[0081] For example, in addition to supporting typical operations of data access and storage, the data storage device 101 may be further configured to perform at least a portion of computations involving an artificial neural network (ANN) 125, such as generating predictions (e.g., 129 or 173) or classifications (e.g., 179) from the sensor data 121 and / or performing unsupervised machine learning 175 (e.g., in Figure 4 ) and / or supervised machine learning 171 (e.g., in Figure 3 or as described in 5) to adjust ANN 125.

[0082] For example, the computations configured in the data storage device 101 can be used to reduce the amount of data to be transmitted to the processor(s) 133 for use or application of the ANN 125 and / or reduce the computational tasks of the processor(s) 133 in evaluating the output of the ANN 125 and / or in training the ANN 125. This arrangement can result in faster output and / or lower energy usage from the data storage device 101 because data does not have to be moved in and out of memory to a dedicated independent neural network accelerator. The computational power of the data storage device 101 in processing data related to the artificial neural network 125 enables the computer system 131 of the motor vehicle 111 to monitor the health of the vehicle components (e.g., in a non-real-time or pseudo-real-time manner) with reduced or no impact on the processing of critical tasks (e.g., autonomous driving by the ADAS 105). In addition, the computational power of the data storage device 101 can be used to accelerate the processing of sensor data of the ADAS 105 and thus improve the processing of critical tasks.

[0083] Figure 7 A data storage device 101 for accelerating neural network computation according to one embodiment is shown. For example, Figure 7 The data storage device 101 may be used to implement Figure 1 Or the data storage device 101 of the vehicle 111 described in 2.

[0084] exist Figure 7 , the data storage device 101 has a host interface 157 configured to communicate with a processor (e.g., 133). For example, the communication between the processor (e.g., 133) and the host interface 157 may be based at least in part on a communication protocol for a peripheral component interconnect express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a universal serial bus (USB) bus, and / or a storage area network (SAN).

[0085] For example, the host interface 157 may be used to receive sensor data 121 generated by the sensors 103 of the vehicle 111 to optionally store a portion of the sensor data 121 in the storage media components ( 161 - 163 ).

[0086] For example, each of the storage media components (161-163) may be a memory integrated circuit configured to store data. For example, the media component (161 or 163) may include one or more integrated circuit dies embedded in an integrated circuit package. The integrated circuit die may have many memory cells formed thereon to store data.

[0087] Typically, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and can retain stored data even when not powered.

[0088] Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase change memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point storage, and memory devices. Cross-point memory devices use transistor-free memory elements, each of which has memory cells and selectors stacked together as columns. The columns of memory elements are connected via two vertical wire layers, one of which is located above the columns of memory elements and the other is located below the columns of memory elements. Each memory element can be individually selected at the intersection of one wire on each of the two layers. Cross-point memory devices are fast and non-volatile and can be used as a unified memory pool for processing and storage. Further examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM), etc. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0089] The data storage device 101 may have a controller 151 including a volatile local memory 153 and at least one processing device 155 .

[0090] The local memory of the controller 151 may be an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines for controlling the operation of the processing device 155, including handling communications between the data storage device 101 and the processor(s) (e.g., 133) of the vehicle 111 and other functions described herein. The local memory 151 of the controller 151 may include a read-only memory (ROM) for storing microcode and / or storage registers and / or volatile memory such as dynamic random access memory (DRAM) and static random access memory (SRAM) for storing, for example, memory pointers, fetched data, etc.

[0091] exist Figure 7 , data storage device 101 includes a neural network accelerator 159 coupled to a controller 151 and / or storage media components (161 to 163).

[0092] For example, the neural network accelerator 159 may be configured to perform matrix arithmetic calculations. Calculations involving the ANN 125 have matrix multiplication and accumulation operations, which may be computationally intensive for a general-purpose processor (e.g., 133). Using the neural network accelerator 159 to perform matrix arithmetic calculations may reduce the data to be transmitted to the processor(s) 133 of the vehicle 111 and reduce the computational workload of the processor(s) 133.

[0093] For example, when ANN 125 includes a spiking neural network (SNN), simulation of the differential equation(s) used to control the activation levels of the SNN neurons may be computationally intensive for a general purpose processor (e.g., 133). Neural network accelerator 159 may use specialized hardware to simulate the differential equation(s) and thus improve the computational efficiency of computer system 131 as a whole.

[0094] In some implementations, neural network accelerator 159 is an integrated circuit device separate from controller 151 and / or storage media components (161-163). Alternatively or in combination, neural network accelerator 159 is integrated with controller 151 in an integrated circuit package. Furthermore, neural network accelerator 159 may be integrated in at least one of the storage media components (161-163), such as Figure 8 As described in .

[0095] Figure 8 A storage media component 160 for accelerating neural network computations according to one embodiment is shown. For example, Figure 7 Each or some of the storage media components (161 to 163) may be used Figure 8 The storage media component 160 is implemented.

[0096] exist Figure 8In the embodiment of the present invention, the storage media component 160 may be housed within an integrated circuit package. The input / output (I / O) interface 171 of the storage media component 160 is configured to process input / output signals in the pins of the integrated circuit package. For example, the input / output signals may include an address signal for specifying a location in the media unit 175 and a data signal representing data to be written in the media unit 175 at the location specified by the address signal or data to be retrieved from the location in the media unit 175.

[0097] exist Figure 8 In the example, the neural network accelerator 159 is coupled to the control logic 173 and / or the media unit 175 to perform calculations for evaluating the output of the ANN 125 and / or training the ANN 125.

[0098] For example, the input / output interface 171 may receive an address identifying a matrix stored in a media unit and to be operated on via the neural network accelerator 159. The storage media component 160 may provide the computational results of the neural network accelerator 159 as output data in response to the address, store the output data in a buffer for further operation, and store the output data in a location specified via the address signal in the media unit 175. Thus, the computation performed by the neural network accelerator 159 may be within the storage media component 160, which is proximate to the media unit 175 in which the matrix data is stored. For example, each of the media units 175 may be an integrated circuit die having a memory cell of a non-volatile memory formed thereon.

[0099] For example, the state data of the SNN neurons may be stored in the media unit 175 according to a predetermined pattern. The neural network accelerator 159 may automatically update the state of the SNN neurons according to the differential equation(s) to control the activation level of the SNN neurons over time. Optionally, the neural network accelerator 159 is configured to process spikes of neurons in the neural network. Alternatively, the data storage device 101 and / or the neural network accelerator 159 of the processor(s) 133 may be configured to process spikes of neurons and / or accumulation of inputs to the SNN.

[0100] Fig. 9 A method for accelerating neural network computation in a motor vehicle according to one embodiment is presented. For example, Fig. 9 The method can be used Figure 7 The storage media component 101 and / or Figure 8 The storage media component 160 is Figure 1 or 2 is implemented in the vehicle 111. For example, Fig. 9 The methods can be combined Figure 6 The method used.

[0101] At block 221 , the data storage device 101 of the vehicle 111 receives a sensor data stream from at least one sensor (eg, 103 ) configured on the vehicle 111 .

[0102] At block 223 , the data storage device 101 stores at least a portion of the sensor data stream.

[0103] At box 225, the neural network accelerator 159 configured in the data storage device 101 performs at least a portion of the calculation based on the artificial neural network 125 and the sensor data stream.

[0104] At box 227, maintenance service for vehicle 111 is predicted based at least in part on calculations performed by a neural network accelerator 159 configured within the data storage device 101.

[0105] Optionally, at box 229, an artificial neural network (ANN) is trained in vehicle 111 using, at least in part, a neural network accelerator and using sensor data streams collected over a predetermined time period (e.g., a period after a new vehicle 111 is delivered from a factory or after a component is replaced in a maintenance service facility 127).

[0106] For example, the neural network accelerator 159 may be configured on an integrated circuit device separate from the controller 151 of the data storage device and / or separate from the storage media components (161-163).

[0107] For example, the neural network accelerator 159 may be configured on an integrated circuit device including the controller 151 of the data storage device 101 , or on an integrated circuit device including the storage media component ( 160 , 1061 , or 163 ) of the data storage device 101 .

[0108] For example, the neural network accelerator 159 may be configured to use data stored in the data storage device 101 to perform calculations, such as matrix arithmetic calculations for an ANN and / or differential equation simulations for an SNN.

[0109] Examples of matrix arithmetic calculations include matrix multiplication and accumulation operations. After performing calculations using the data stored in the data storage device 101 to produce the results of the matrix arithmetic calculations, the neural network accelerator 159 may provide the results as outputs of the data storage device 111 when the data is retrieved (e.g., in response to a read command). Alternatively or in combination, the results of the matrix arithmetic calculations may be buffered in the data storage device 101 as operands for the next matrix calculation performed in conjunction with the data matrix retrieved from the non-volatile memory via the read command received in the host interface 157.

[0110] When the artificial neural network (ANN) 125 includes a spiking neural network (SNN), the neural network accelerator may be configured to simulate differential equations that control activation levels of neurons in the spiking neural network (SNN). Optionally, the storage medium component is configured to store the states of the neurons in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to automatically update the states of the neurons over time according to the differential equation. For example, the neural network accelerator 159 may be configured to train the spiking neural network (SNN) to detect anomalies via unsupervised machine learning.

[0111] Server 119 , computer system 131 , and / or data storage device 101 may each be implemented as one or more data processing systems.

[0112] The present invention includes methods and apparatus for performing the methods described above, including data processing systems that perform these methods, and computer-readable media containing instructions that, when executed on a data processing system, cause the system to perform these methods.

[0113] A typical data processing system may include an interconnect (eg, a bus and system core logic) that interconnects (several) microprocessors and memory. A microprocessor is typically coupled to a cache memory.

[0114] The interconnect interconnects the microprocessor(s) and memory together and also interconnects the microprocessor(s) and memory to input / output (I / O) devices via I / O controller(s). The I / O devices may include display devices and / or peripheral devices such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some I / O devices, such as a printer, scanner, mouse, and / or keyboard, are optional.

[0115] The interconnect may include one or more buses connected to each other through various bridges, controllers and / or adapters. In one embodiment, the I / O controller includes a USB (Universal Serial Bus) adapter for controlling USB peripherals and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

[0116] The memory may include one or more of the following: ROM (read only memory), volatile RAM (random access memory), and non-volatile memory such as a hard drive, flash memory, and the like.

[0117] Volatile RAM is typically implemented as dynamic RAM (DRAM) that constantly requires power in order to refresh or maintain the data in the memory. Nonvolatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other types of memory systems that maintain data even after power is removed from the system. Nonvolatile memory can also be random access memory.

[0118] The non-volatile memory may be a local device directly coupled to the rest of the components in the data processing system. Non-volatile memory remote from the system may also be used, such as a network storage device coupled to the data processing system through a network interface such as a modem or Ethernet interface.

[0119] In the present invention, some functions and operations are described as being executed by software codes or caused by software codes to simplify the description. However, such expressions are also used to specify that the functions are generated by a processor such as a microprocessor executing code / instructions.

[0120] Alternatively or in combination, the functions and operations as described herein may be implemented using dedicated circuitry with or without software instructions, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments may be implemented using hardwired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source for the instructions executed by the data processing system.

[0121] While one embodiment may be implemented in fully functional computers and computer systems, the various embodiments may be distributed in a variety of forms as a computing product and may be applicable regardless of the specific type of machine or computer readable media used to actually implement the distribution.

[0122] At least some of the disclosed aspects may be embodied at least in part in software. That is, the techniques may be implemented in a computer system or other data processing system in response to its processor (e.g., a microprocessor) executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache memory, or remote storage device).

[0123] The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions called a "computer program". A computer program typically includes one or more instructions that are placed at different times in various memories and storage devices in a computer and that, when read and executed by one or more processors in a computer, cause the computer to perform the operations necessary to implement the elements involved in various aspects.

[0124] Machine-readable media can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory, and / or cache memory. Portions of this software and / or data can be stored in any of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or in the same communication session. Data and instructions can be obtained as a whole before executing an application. Alternatively, portions of data and instructions can be dynamically obtained in a timely manner when execution requires. Therefore, at a specific time instance, data and instructions do not need to be located on a machine-readable medium as a whole.

[0125] Examples of computer-readable media include, but are not limited to, non-volatile media, recordable media, and non-recordable media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CD ROM), digital versatile disks (DVD), etc.). Computer-readable media may store instructions.

[0126] The instructions may also be embodied in digital and analog communication links using electrical, optical, acoustic or other forms of propagated signals such as carrier waves, infrared signals, digital signals, etc. However, propagated signals such as carrier waves, infrared signals, digital signals, etc. are not tangible machine-readable media and are not configured to store instructions.

[0127] Generally, a machine-readable medium includes any mechanism that provides (ie, stores and / or transmits) information in a form accessible by a machine (eg, a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors, etc.).

[0128] In various embodiments, hard-wired circuits may be combined with software instructions to implement the techniques. Therefore, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source of instructions executed by a data processing system.

[0129] The above description and accompanying drawings are illustrative and should not be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described in order to avoid making the description unclear. References to one embodiment or embodiments in the present invention are not necessarily references to the same embodiment; and such references represent at least one embodiment.

[0130] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made to the invention without departing from the broader spirit and scope as set forth in the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A data storage device for a vehicle, comprising: a hardware host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; at least one storage media component having non-volatile memory configured to store at least a portion of the sensor data stream; the controller; and a neural network accelerator coupled to the controller, the neural network accelerator configured to: determining whether a first portion of the sensor data stream was acquired during a time period when the vehicle is expected to operate in a normal mode; pre-classifying the first portion of the sensor data stream obtained during the time period as normal and pre-classifying it into a plurality of clusters corresponding to different types of normal operating conditions associated with the vehicle; utilizing an artificial neural network to perform at least a portion of a calculation based on the first portion of the sensor data stream to predict a time for maintenance service of the vehicle; detecting an anomaly associated with a component of the vehicle based on a second portion of the sensor data classified outside of the plurality of clusters; estimating a severity of the anomaly associated with the component of the vehicle to facilitate scheduling the maintenance service; and The artificial neural network is trained during the time period when the vehicle is expected to operate in the normal mode by adjusting the artificial neural network based on the sensor data stream and the actual time of the maintenance service for the vehicle to reduce the difference between the predicted time of future maintenance service and the actual time of the future maintenance service.

2. A data storage device according to claim 1, wherein the neural network accelerator is configured to use data stored in the data storage device to perform at least matrix arithmetic calculations.

3. The data storage device of claim 2, wherein the matrix arithmetic calculations comprise matrix multiplication and accumulation operations.

4. A data storage device according to claim 3, wherein the neural network accelerator is configured to provide the result of the matrix arithmetic calculation as an output of the data storage device when retrieving data.

5. The data storage device of claim 3, wherein the neural network accelerator is configured to buffer results of matrix arithmetic calculations for use in a next matrix calculation performed in conjunction with a matrix of data retrieved from the non-volatile memory via a read command received in the hardware host interface.

6. The data storage device of claim 1, wherein the artificial neural network comprises a spiking neural network.

7. The data storage device of claim 6, wherein the neural network accelerator is configured to simulate differential equations that control activation levels of neurons in the spiking neural network.

8. The data storage device of claim 7, wherein the storage media component is configured to store states of the neurons in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to update the states of the neurons over time according to the differential equation.

9. The data storage device of claim 7, wherein the neural network accelerator is configured to train the spiking neural network to detect the anomaly via unsupervised machine learning.

10. A storage media component comprising: a hardware input / output interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; at least one media unit having non-volatile memory to store at least a portion of the sensor data stream; control logic; and a neural network accelerator coupled to the control logic, the neural network accelerator configured to: determining whether a first portion of the sensor data stream was acquired during a time period when the vehicle is expected to operate in a normal mode; pre-classifying the first portion of the sensor data stream obtained during the time period as normal and pre-classifying it into a plurality of clusters corresponding to different types of normal operating conditions associated with the vehicle; utilizing an artificial neural network to perform at least a portion of a calculation based on the first portion of the sensor data stream to predict a time for maintenance service of the vehicle; detecting an anomaly associated with a component of the vehicle based on a second portion of the sensor data classified outside of the plurality of clusters; estimating a severity of the anomaly associated with the component of the vehicle to facilitate scheduling the maintenance service; and The artificial neural network is trained during the time period when the vehicle is expected to operate in the normal mode by adjusting the artificial neural network based on the sensor data stream and the actual time of the maintenance service for the vehicle to reduce the difference between the predicted time of future maintenance service and the actual time of the future maintenance service.

11. The storage media assembly of claim 10, further comprising: An integrated circuit package, wherein the hardware input / output interface, the media unit, the control logic and the neural network accelerator are sealed within the integrated circuit package.

12. The storage media assembly of claim 11, wherein the media unit comprises an integrated circuit die on which memory cells of the nonvolatile memory are formed.

13. The storage media assembly of claim 12, wherein the integrated circuit package includes pins for connecting to a controller of a data storage device of the vehicle.

14. The storage media component of claim 12, wherein the neural network accelerator is configured to perform at least matrix arithmetic calculations using data stored in the at least one media unit.

15. The storage media component of claim 12, wherein the artificial neural network comprises a spiking neural network; and the neural network accelerator is configured to simulate differential equations that control activation levels of neurons in the spiking neural network.

16. A method comprising: In a data storage device of a vehicle, receiving a sensor data stream from at least one sensor disposed on the vehicle; storing at least a portion of the sensor data stream in the data storage device; determining whether a first portion of the sensor data stream was acquired during a time period when the vehicle is expected to operate in a normal mode; pre-classifying the first portion of the sensor data stream obtained during the time period as normal and pre-classifying it into a plurality of clusters corresponding to different types of normal operating conditions associated with the vehicle; performing at least a portion of a computation using an artificial neural network and the first portion of the sensor data stream via a neural network accelerator configured in the data storage device; predicting a time for maintenance service of the vehicle based on the calculation; detecting an anomaly associated with a component of the vehicle based on a second portion of the sensor data classified outside of the plurality of clusters; estimating a severity of the anomaly associated with the component of the vehicle to facilitate scheduling the maintenance service; and The artificial neural network is trained during the time period when the vehicle is expected to operate in the normal mode by adjusting the artificial neural network based on the sensor data stream and the actual time of the maintenance service for the vehicle to reduce the difference between the predicted time of future maintenance service and the actual time of the future maintenance service.

17. The method of claim 16, wherein the neural network accelerator is configured on an integrated circuit device separate from a controller of the data storage device.

18. The method of claim 16, wherein the neural network accelerator is configured on an integrated circuit device that includes a controller of the data storage device.

19. The method of claim 16, wherein the neural network accelerator is configured on an integrated circuit device that includes a storage media component of the data storage device.

20. The method of claim 16, wherein the calculation comprises at least matrix arithmetic calculation or differential equation simulation.

Images