System, method and apparatus for optimizing neural network computing for predictive maintenance of vehicles

Abstract

The present application is directed to systems, methods, and apparatus for optimizing neural network computations for predictive maintenance of a vehicle. For example, a data storage device for a vehicle includes: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; at least one storage media component having a non-volatile memory; and a controller. The non-volatile memory is configured into a plurality of partitions (e.g., namespaces), the partitions having different sets of memory operation settings configured for different types of data associated with an artificial neural network ANN. The partitions include a model partition configured to store model data for the ANN. The sensor data stream is applied to the ANN to predict maintenance service for the vehicle. The memory cells of the model partition may be configured for reading, rare updates, improved storage capacity, and / or for access in parallel with the input / output of the ANN.

Description

Technical Field

[0001] At least some embodiments disclosed herein relate generally to maintenance services for vehicles, and more specifically, but not limited to, to storage and access of neural network models for predictive maintenance of automobiles. Background Art

[0002] Conventionally, automobile maintenance is scheduled based on predetermined operating milestones. For example, a routine maintenance service may be scheduled every three or six months, or after a predetermined distance (e.g., 3,000 miles, 6,000 miles, or 15,000 miles) has been traveled.

[0003] Such incidents can be safety threatening when components of a motor vehicle malfunction or fail during vehicle operation. After such incidents occur, it is important to schedule a trip to obtain vehicle service as soon as possible, even at an inconvenient time.

[0004] Recent developments in the field of autonomous driving technology allow computing systems to operate control elements of a motor vehicle without the assistance of a human operator of the vehicle, at least under some conditions.

[0005] For example, sensors (e.g., cameras and radars) may be mounted on a motor vehicle to detect the vehicle's surroundings as it travels on a roadway. A computing system mounted on the vehicle analyzes the sensor inputs to identify conditions and generates control signals or commands for autonomous adjustments to the vehicle's direction and / or speed, with or without any input from a human operator of the vehicle.

[0006] In some arrangements, when the computing system identifies a situation in which the computing system may be unable to continue operating the vehicle in a safe manner, the computing system alerts the human operator of the vehicle and requests the human operator to take over control of the vehicle and drive manually instead of allowing the computing system to autonomously drive the vehicle.

[0007] U.S. Patent No. 9,533,579, entitled “Electronic Control Apparatus for Electrically-Driven Vehicle” and published on January 3, 2017, discloses an electronic control apparatus for a vehicle having a self-diagnostic function.

[0008] Autonomous driving and / or advanced driver assistance systems (ADAS) often involve artificial neural networks (ANNs) for recognizing events and / or objects captured in sensor inputs.

[0009] Generally speaking, an artificial neural network (ANN) uses a network of neurons to process inputs to the network and produce outputs from the network.

[0010] For example, each neuron in the network receives a set of inputs. Some of the inputs to the neurons may be outputs of certain neurons in the network; and some of the inputs to the neurons may be inputs provided to the neural network. The input / output relationships among neurons in the network represent the neuron connectivity in the network.

[0011] For example, each neuron may have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function may be in the form of a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network may have different activation functions.

[0012] For example, each neuron may produce a weighted sum of its input and its bias and then produce an output that is a function of the weighted sum, computed using the neuron's activation function.

[0013] The relationship between the input and output of an ANN is generally defined by an ANN model, which contains data representing the connectivity of neurons in the network, as well as the bias, activation function, and synaptic weights of each neuron. Using a given ANN model, a computing device calculates the output of the network from a given set of inputs sent to the network.

[0014] For example, input to the ANN network may be generated based on camera input; and output from the ANN network may be recognition of items such as events or objects.

[0015] Spiking neural networks (SNNs) are a type of ANN that closely simulates natural neural networks. When the activation level of a neuron is high enough, the SNN neuron generates a pulse as an output. The activation level of an SNN neuron simulates the membrane potential of a natural neuron. The output / pulse of an SNN neuron can change the activation level of other neurons that receive the output. The current activation level of an SNN neuron over time is typically modeled using differential equations and is considered to be the state of the SNN neuron. Incoming pulses from other neurons can push the activation level of a neuron higher to reach the threshold of the pulse. Once a neuron generates a pulse, its activation level is reset. Before a pulse is generated, the activation level of an SNN neuron may decay over time, as controlled by a differential equation. The temporal element in the behavior of SNN neurons makes SNNs suitable for processing spatiotemporal data. The connectivity of SNNs is typically sparse, which helps reduce computational workload.

[0016] In general, ANNs can be trained using supervised methods in which parameters in the ANN are adjusted to cause the error between the known outputs produced by the corresponding inputs and the calculated outputs produced from applying the inputs to the ANN. Examples of supervised learning / training methods include reinforcement learning, and learning with error correction.

[0017] Alternatively or in combination, the ANN may be trained using unsupervised methods in which the exact output produced by a given set of inputs is not known until training is complete.The ANN may be trained to classify items into categories, or to classify data points into clusters.

[0018] Multiple training algorithms can be used for complex machine learning / training paradigms. Summary of the invention

[0019] In one aspect, the present disclosure is directed to a data storage device for a vehicle, comprising: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; a non-volatile memory configured into a plurality of partitions, the partitions respectively having different memory operation setting sets configured for different types of data related to an artificial neural network, wherein the partitions include a model partition configured to store model data of the artificial neural network; and a controller configured to operate memory cells in the partitions according to the memory operation setting set; wherein the artificial neural network is configured to predict maintenance service for the vehicle based on the sensor data stream.

[0020] In another aspect, the present disclosure is directed to a vehicle comprising: at least one processor; at least one sensor configured to generate a sensor data stream; and a data storage device having: a host interface configured to receive the sensor data stream; a non-volatile memory configured into a plurality of partitions, the partitions respectively having different memory operation setting sets configured for different types of data associated with an artificial neural network, wherein the partitions include a model partition configured to store model data of the artificial neural network; and a controller configured to operate memory cells in the partitions according to the memory operation setting set; wherein the artificial neural network is configured to predict maintenance service for the vehicle based on the sensor data stream.

[0021] In another aspect, the present disclosure is directed to a method comprising: configuring a non-volatile memory of a data storage device into a plurality of partitions; storing, for each of the partitions, different sets of memory operation settings for different types of data related to an artificial neural network, wherein the partitions include a model partition configured to store model data of the artificial neural network; receiving, in the data storage device, a sensor data stream from at least one sensor configured on a vehicle; operating memory cells in the partitions according to the set of memory operation settings in response to the sensor data stream; and predicting maintenance service for the vehicle based on the sensor data stream using the artificial neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0023] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.

[0024] Figure 2 An autonomous vehicle with a data storage device is shown according to one embodiment.

[0025] Figure 3-Figure 5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.

[0026] Figure 6 A method of predictive maintenance according to one embodiment is shown.

[0027] Figure 7 A data storage device for accelerating neural network computations according to one embodiment is shown.

[0028] Figure 8 A storage media assembly for accelerating neural network computations is shown according to one embodiment.

[0029] Fig. 9 A method of accelerating neural network computations in a motor vehicle is shown according to one embodiment.

[0030] Fig.10 A data storage device configured to support neural network computations is shown according to one embodiment.

[0031] Fig.11 The configuration of a namespace for an artificial neural network (ANN) model according to one embodiment is described.

[0032] Fig.12 The configuration of a namespace for input sent to an artificial neuron according to one embodiment is described.

[0033] Fig.13 The configuration of a namespace for outputs from an artificial neuron according to one embodiment is described.

[0034] Figure 14-16 A method of predictive maintenance supported by model partitioning, input partitioning, and output partitioning according to one embodiment is shown. DETAILED DESCRIPTION

[0035] At least some embodiments disclosed herein provide systems, methods, and apparatus for processing sensor data generated in a motor vehicle, or another vehicle with or without an advanced driver assistance system (ADAS), to facilitate predictive maintenance.

[0036] Before a component of a motor vehicle fails or malfunctions during operation of the vehicle, there may be an indication whether the component requires replacement or maintenance. Such an indication may not be noticeable to a typical driver or passenger. However, sensor data may be collected and analyzed to predict the probability of component failure. The prediction may be used to schedule maintenance service, which may reduce or eliminate the chance of an incident where a component of the vehicle fails or malfunctions during operation of the vehicle on the roadway. Furthermore, the prediction allows a service trip to be scheduled at a convenient time.

[0037] For example, sensors may be installed in an automobile system to collect data during its routine operation; and the sensor data may be used to predict whether a component needs to be replaced or maintained and how soon it will be needed. The sensor data may be provided as input to an artificial neural network (ANN) (e.g., a spiking neural network (SNN)) sent to an artificial intelligence (AI) system to train itself (e.g., using unsupervised machine learning techniques) over a period of time in which the vehicle is expected to operate normally. The training is customized to the neural network for the specific operating environment of the driver, passenger, or user of the vehicle and the personalized operating habits of the vehicle's passengers. Subsequently, the artificial neural network may detect abnormal conditions when the operating data deviates from the normal pattern. The AI ​​system may be used to recommend maintenance services and / or identify components that may need to be replaced or maintained.

[0038] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.

[0039] Figure 1The system includes a vehicle 111 having a data storage device 101. Optionally, the vehicle 111 has an advanced driver assistance system (ADAS) 105 and one or more sensors 103 that provide sensor data input to the ADAS 105 and / or the data storage device 101. The data storage device 101 is configured to use an artificial neural network (ANN) 125 to predict / identify the need for maintenance service based on data collected by the sensor 103. ADAS 105 can be omitted without affecting the predictive maintenance feature. In some embodiments, at least a portion of the data generated by the sensor 103 is in the ADAS 105 for driving assistance and in the ANN 125 for maintenance prediction. Optionally, the output of the ANN 124 can be used in the data storage device 101 and in the ADAS 105.

[0040] Sensors 103 may include digital cameras, lidars, radars, ultrasonic sonars, brake sensors, speed sensors, acceleration sensors, airbag sensors, global positioning system (GPS) receivers, audio sensors / microphones, vibration sensors, force / strain sensors, deformation sensors, motion sensors, temperature sensors, etc. Some of sensors 103 may be configured primarily to monitor the environment of vehicle 111; and other sensors 103 may be configured primarily to monitor the operating conditions of one or more components of vehicle 111 (e.g., internal combustion engine, exhaust system, electric motor, brake pads, tires, battery, etc.).

[0041] The time-varying output of the sensors 103 is provided as a sensor data stream to the ADAS 105 and / or the ANN 125 to provide driving assistance (eg, autonomous driving) and maintenance prediction.

[0042] For example, the vehicle 111 may have a wireless communication device that communicates with a remote server 119 via wireless signals 113 and a communication network 117. The remote server 119 is typically configured at a location away from the road 102 on which the vehicle 111 is traveling. For example, the vehicle 111 may provide some sensor data 121 to the server 119 and receive updates of the ANN 125 from the server 119.

[0043] One example of a communication network 117 is a cellular telephone network having one or more base stations (e.g., 115) that receive wireless signals (e.g., 113). Another example of a communication network 117 is the Internet, where wireless local area network signals (e.g., 113) transmitted by the vehicle 111 are received in an access point (e.g., 115) for further transmission to a server 119. In some embodiments, the vehicle 111 uses a communication link 107 to a satellite 109 or a communication balloon that communicates with the server 119.

[0044] The server 119 may also communicate with one or more maintenance service facilities (e.g., 127) to receive maintenance service data 123 of the vehicle (e.g., 111). The maintenance service data 123 may include inspection records and / or service records of components of the vehicle (e.g., 111). For example, the inspection records and / or service records may indicate the degree of wear of the components inspected at the maintenance service facility (e.g., 127) during the service of the components, the identification of faulty or malfunctioning components, etc. The sensor data 121 and maintenance service data 123 of the vehicle (e.g., 111) in the time period before the service may be used to train the ANN 125 to predict the probability that the component needs maintenance service. The updated ANN 125 may be used to predict and recommend maintenance service for the vehicle 111 based on the sensor data 121 received in the most recent time period. Alternatively, the updated ANN 125 may be transmitted to the vehicle 111; and the vehicle 111 may use the data generated from the sensor 103 during the routine operation of the vehicle 111 to predict and recommend maintenance service.

[0045] The data storage device 101 of the vehicle 111 may be configured to record sensor data over a time period used in the ANN for predictive maintenance. Maintenance predictions are typically for relatively long time periods (e.g., days, weeks, and / or months). In contrast, sensor data recorded for reviewing an accident, collision, or near-collision involving an autonomous vehicle is typically for a short time period (e.g., 30 seconds to minutes). Therefore, a typical black box data recorder configured to record sensor data for reviewing / analyzing an accident or collision is insufficient for predictive maintenance.

[0046] Optionally, the data storage device 101 stores sensor data about time periods that cause trips to maintenance service facilities (e.g., 127). The maintenance service facility (e.g., 127) can download the sensor data 121 from the data storage device 101 and provide the sensor data 121 and corresponding maintenance service data 123 to the server 119 to facilitate training of the ANN 125.

[0047] Optionally, or in combination, the data storage device 101 is configured with a machine learning module to customize and / or train the ANN 125 installed in the vehicle 111 for predictive maintenance.

[0048] For example, the machine learning module of the data storage device 101 may be used to calibrate the ANN 125 to take into account the typical / daily environment in which the vehicle 111 is operated and / or the driving preferences / habits of the driver of the vehicle 111 .

[0049] For example, during a time period when the vehicle is expected to operate in a typical / daily environment with healthy components, the sensor data generated by the sensors 103 can be used to train the ANN 125 to recognize patterns of sensor data that represent trouble-free operation. Such patterns may be different for different vehicles (e.g., 111) based on their routine operating environments and the driving habits / characteristics of their drivers. The training allows the ANN 125 to detect deviations from the recognized normal patterns and report anomalies for maintenance predictions.

[0050] For example, the ANN 125 may include an SNN configured to classify time-based changes in sensor data and / or detect deviations from known sensor data patterns for the vehicle 111 operating under normal / healthy conditions but in a personalized environment (e.g., a driver / passenger's daily route) and / or under personalized driving habits / patterns.

[0051] Figure 2 An autonomous vehicle 111 with a data storage device 101 is shown according to one embodiment. For example, Figure 2 111 Implementation of Autonomous Vehicles Figure 1 Vehicle 111 in the system.

[0052] Figure 2 The vehicle 111 is configured with an advanced driver assistance system (ADAS) 105. The ADAS 105 of the vehicle 111 may have an artificial neural network (ANN) 125 for object detection, recognition, identification, and / or classification. The ANN 125 and / or another neural network (e.g., configured in the data storage device 101) may be used to predict the probability that a component of the vehicle 111 requires maintenance service (e.g., repair, replacement, or adjustment).

[0053] Preferably, the data storage device 101 is configured to process the sensor data, at least in part, for predictive maintenance, placing a reduced computational burden on the processor 133 responsible for operating the ADAS 105 and / or other components (eg, the infotainment system 149 ).

[0054] The vehicle 111 typically includes an infotainment system 149, a communication device 139, one or more sensors 103, and a computer system 131 connected to some controls of the vehicle 111 (e.g., a steering control 141 for the direction of the vehicle 111, a brake control 143 for stopping the vehicle 111, an acceleration control 145 for the speed of the vehicle 111, etc.). In some embodiments, Figure 1 The vehicle 111 in the system has a similar configuration and / or similar components.

[0055] Some of the sensors 103 are required for operation of the ADAS 105; and some of the sensors 103 are used to collect data related to the health of components of the vehicle 111, which may not be used in the ADAS 105. Optionally, the sensor data generated by the sensors 103 can also be used to predict the likelihood of an imminent component failure. Such predictions can be used in the ADAS 105 to take emergency actions to put the vehicle in a safe state (e.g., by reducing speed and / or stopping).

[0056] The computer system 131 of the vehicle 111 includes one or more processors 133 , the data storage device 101 , and a memory 135 storing firmware (or software) 147 including computer instructions and data models for the ADAS 105 .

[0057] The one or more sensors 103 of the vehicle may include a visible light camera, an infrared camera, a lidar, a radar or sonar system, a peripheral device sensor, a global positioning system (GPS) receiver, a satellite positioning system receiver, a brake sensor, and / or an airbag sensor. In addition, the sensors 103 may include an audio sensor (e.g., a microphone) configured to monitor noise from various components and locations in the vehicle 111, a vibration sensor, a pressure sensor, a force sensor, a strain sensor, and / or a deformation sensor configured to measure the load on the components of the vehicle 111, an accelerometer and / or a gyroscope sensor that measures the movement of some components of the vehicle 111, etc. Such sensors can be used to monitor the operating state and / or health of the components for predictive maintenance.

[0058] The sensor 103 may provide a real-time sensor data stream to the computer system 131. The sensor data generated by the sensor 103 of the vehicle 111 may include an image of an object captured using a camera, or a sonar, radar, or lidar system, wherein the camera is a camera that captures the image using light visible to the human eye, or a camera that captures the image using infrared light. The image data obtained from at least one sensor of the vehicle is part of the collected sensor data recorded in the data storage device 101 and / or as an input to the ANN 125. For example, the camera can be used to obtain roadway information for the vehicle 111 to travel, and the ANN 125 can process the roadway information to generate control signals for the vehicle 111. For example, the camera can be used to monitor the operating status / health of the components of the vehicle 111, and the ANN 125 can process the operating status / health to predict or schedule maintenance services.

[0059] The sensor data generated by the sensors 103 of the vehicle 111 may include an audio stream that captures characteristics of sounds at locations on the vehicle 111 (e.g., locations near the engine, motor, transmission system, wheels, doors, windows, etc.). The audio data obtained from at least one sensor 103 of the vehicle 111 may be part of the collected sensor data recorded in the data storage device 101 and / or as input to the ANN 125. For example, the audio stream may be used to monitor the operating status / health of components of the vehicle 111 (e.g., internal combustion engine, exhaust system, electric motor, brakes), which the ANN 125 may process to predict or schedule maintenance service.

[0060] The infotainment system 149 may be used to present the predicted or scheduled maintenance service. Optionally, the communication device 139 may establish a connection to a mobile device of the driver of the vehicle 111 to notify the driver of the recommended maintenance service and / or recommended service data, schedule the appointment, etc.

[0061] When the vehicle 111 is configured with the ADAS 105 , the output of the ADAS 105 may be used to control (eg, 141 , 143 , 145 ) the acceleration of the vehicle 111 , the speed of the vehicle 111 , and / or the direction of the vehicle 111 during autonomous driving.

[0062] Figure 3-Figure 5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.

[0063] exist Figure 3 In the example, the supervised machine learning module 171 is used to train the artificial neural network 125 to minimize the difference between the service prediction 129 generated from the sensor data 121 and the maintenance service data 123.

[0064] For example, the maintenance service data 123 may identify measured wear of a component over time to predict when a recommended service will be performed. The sensor data 121 may be used in the ANN 125 to generate a predicted time when the recommended service will be performed. The supervised machine learning module 171 may adjust the artificial neural network 125 to reduce / minimize the difference between the time predicted based on the sensor data 121 and the time calculated from the wear measurements.

[0065] For example, maintenance service data 123 may identify a component that was replaced or repaired in maintenance service facility 127. Sensor data 121 recorded during a period of time before the component was replaced or repaired may be used to calculate the time when the replacement or repair will be performed. In addition, segments of the sensor data stream during the period of time before the replacement or repair may be used in ANN 125 to generate a prediction of the time when the replacement or repair will be performed. Supervised learning 171 may be used to adjust ANN 125 to reduce the predicted time when the replacement or repair will be performed and the actual time when the replacement or repair is performed.

[0066] Figure 2 The supervised learning 171 may be applied in the server 119 to generate a general ANN for the vehicle population based on the sensor data of the vehicle population and its maintenance service data 123 .

[0067] Figure 2 The supervised learning 171 can be applied in the vehicle 111 to generate a customized / personalized ANN for a group of vehicles based on the vehicle's sensor data and its maintenance service data 123. For example, a general ANN can be initially used in the vehicle 111; and the sensor data of the vehicle 111 and its maintenance service data 123 can be used to further train the vehicle's ANN 125 for customization / personalization of the ANN 125 in the vehicle 111.

[0068] exist Figure 4 In the example of FIG. 1 , an unsupervised machine learning module 175 is used to train or optimize the artificial neural network 125 to facilitate anomaly detection 173. The unsupervised machine learning module 175 is configured to adjust the ANN (e.g., SNN) for classifications, clusters, or recognized patterns in the sensor data 121 so that the degree of deviation from the classifications, clusters, or recognized patterns in the sensor data 121 generated within a recent time period can be used to signal the detection 173 of an anomaly. The anomaly detection 173 allows the vehicle 111 to be scheduled for inspection at the maintenance service facility 127. Optionally, after the inspection, such as in Figure 3 In the example embodiment, the maintenance service data 123 may be used to apply supervised learning 171 to produce more accurate predictions of service.

[0069] Generally, it may be assumed that the vehicle 111 is operating in normal / healthy conditions for a certain period of time. For example, after a new vehicle 111 is initially put into service, it may be assumed that the vehicle 111 provides trouble-free service for at least a period of time (e.g., several months). For example, after a period of time after a component is replaced or repaired, it may be assumed that the component provides trouble-free service for at least a period of time (e.g., several months or a year). Therefore, the sensor data 121 obtained during this period of time may be pre-classified as "normal" for use in the following examples. Figure 4 Unsupervised learning in 175 or Figure 5Supervised learning in 171 training ANN 125.

[0070] For example, sensor data 121 collected during a "normal" service period of a vehicle 111 or component may be classified into several clusters via unsupervised learning 175. Different clusters may correspond to different types of normal conditions (e.g., driving on different routes, driving on roads with different road conditions, driving in different weather conditions, driving at different times of the day, driving on different days of the week, driving under different emotions of the driver's driving habits). When continuous sensor data 121 is classified outside of a "normal" cluster, an anomaly is detected.

[0071] Optionally, as Figure 5 , supervised machine learning 171 may be used to train the ANN 125. During a "normal" service cycle of a vehicle 111 or component, an expected classification 177 may be used to label the sensor data 121. Supervised learning 171 may be used to minimize the classification difference between a prediction 179 made from the sensor data 121 using the ANN 125 and the expected classification 177. In addition, when the sensor data 121 is known to be "abnormal" (e.g., after a diagnosis made in a maintenance service facility 127 or by a user, driver, or passenger of the vehicle 111), the expected classification 177 may be changed to "abnormal" for further training the ANN 125 to directly recognize abnormalities (e.g., instead of relying on deviations from a known "normal" cluster to infer abnormalities).

[0072] Thus, the ANN 125 may be trained to identify anomalous sensor data and schedule maintenance service by estimating the severity of the anomaly.

[0073] Figure 6 A method of predictive maintenance according to one embodiment is shown. For example, Figure 6 The method can be implemented in Figure 1 or 2 in the data storage device 101 in the vehicle 111 or Figure 2 A computer system 131 in the vehicle 111.

[0074] At box 201 , sensors (eg, 103 ) installed in a vehicle 111 generate a sensor data stream (eg, 121 ) during operation of the vehicle 111 on a road 102 .

[0075] At block 203, the sensor data stream (eg, 121) is provided to an artificial neural network (ANN) 125. For example, the ANN 125 may include a spiking neural network (SNN).

[0076] At block 205 , an artificial neural network (ANN) 125 generates a prediction of maintenance service based on the sensor data stream (eg, 121 ).

[0077] At block 207 , the data storage device 101 disposed on the vehicle stores at least a portion of the sensor data stream (eg, 121 ).

[0078] At box 209 , an artificial neural network (ANN) is trained using sensor data streams (eg, 121 ) collected over a predetermined period of time since the vehicle left the factory or maintenance service facility 127 .

[0079] For example, an artificial neural network (ANN) may be configured to identify components of vehicle 111 that require repair or replacement during maintenance service and / or identify a predicted time period until a component fails or malfunctions, or a suggested time period until a recommended maintenance service for a component prior to a component failure or malfunction. Thus, the performance of the predicted maintenance service may avoid an incident in which a component fails or malfunctions while vehicle 111 is operating on road 102.

[0080] For example, the sensor 103 may be a microphone mounted near the component, a vibration sensor attached to the component, a pressure sensor mounted in the component, a force or stress sensor mounted to or attached to the component, a deformation sensor attached to the component, or an accelerometer configured to measure a motion parameter of the component.

[0081] Optionally, the data storage device 101, the vehicle 111 of the computer system 131 and / or a server 119 remote from the vehicle may have a machine learning module that is configured to train an artificial neural network (ANN) 125 during a time period in which the vehicle 111 is assumed to be in a healthy state (e.g., a predetermined time period from when the vehicle 111 leaves a factory or maintenance service facility 127).

[0082] For example, if Figure 4 As described in , the machine learning module can use unsupervised machine learning 175 to train the ANN 125 to recognize / classify normal patterns of sensor data 121 and thus have the ability to detect anomalies based on deviations from normal patterns. Figure 3 As described in or 5, supervised machine learning 171 can be used.

[0083] For example, unsupervised machine learning 175 may be applied by the data storage device 101 or computer system 131 of the vehicle 111 during a predetermined time period during which the vehicle and / or components are known to operate without impairment or degradation.

[0084] Alternatively or in combination, some of the sensor data 121 stored in the data storage device 101 of the vehicle 111 may be uploaded to the server 119 for use in training the ANN 125 .

[0085] In at least some embodiments disclosed herein, the data storage device 101 is configured to accelerate computations of an artificial neural network (ANN) 125 of the vehicle 111 .

[0086] For example, in addition to supporting typical operations of data access and storage, the data storage device 101 may be further configured to perform at least a portion of computations involving an artificial neural network (ANN) 125, such as generating predictions (e.g., 129 or 173) or classifications (e.g., 179) from the sensor data 121 and / or performing unsupervised machine learning 175 (e.g., Figure 4 ) and / or supervised machine learning 171 (e.g., as described in Figure 3 or as described in 5) to adjust ANN 125.

[0087] For example, the calculations configured in the data storage device 101 can be used to reduce the amount of data transmitted to the processor 133 for use or application by the ANN 125 and / or reduce the computational tasks of the processor 133 when evaluating the output of the ANN 125 and / or when training the ANN 125. Such an arrangement can result in faster output from the data storage device 101 and / or reduced energy usage, because data in and out of the memory does not have to be moved to a dedicated independent neural network accelerator. The computing power of the data storage device 101 in processing data related to the ANN 125 allows the computer system 131 of the motor vehicle 111 to monitor the health of the vehicle components (e.g., in a non-real-time manner, or a pseudo-real-time manner), with reduced impact, or no impact, on the processing of mission-critical tasks (e.g., autonomous driving by ADAS105). In addition, the computing power of the data storage device 101 can be used to accelerate the processing of sensor data for the ADAS105 and thus improve the processing of mission-critical tasks.

[0088] Figure 7 1 shows a data storage device 101 for accelerating neural network calculations according to one embodiment. For example, Figure 7 The data storage device 101 can be used to implement Figure 1 or the data storage device 101 of the vehicle 111 described in 2

[0089] exist Figure 7, the data storage device 101 has a host interface 157 configured to communicate with a processor (e.g., 133). For example, the communication between the processor (e.g., 133) and the host interface 157 may be based at least in part on a communication protocol for a peripheral component interconnect express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a universal serial bus (USB) bus, and / or a storage area network (SAN).

[0090] For example, the host interface 157 may be used to receive sensor data 121 generated by sensors 103 of the vehicle 111 to optionally store a portion of the sensor data 121 in the storage media components 161 - 163 .

[0091] For example, each of the storage media components 161 to 163 may be a memory integrated circuit configured to store data. For example, the media component 161 or 163 may include one or more integrated circuit dies embedded in an integrated circuit package. The integrated circuit die may have a plurality of memory cells formed thereon to store data.

[0092] Generally speaking, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and can retain stored data even when not powered.

[0093] Examples of non-volatile memory include flash memory, memory cells formed based on "NAND" logic gates, "NOR" logic gates, phase change memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point memory devices, and memory devices. Cross-point memory devices use memory elements with fewer transistors, each of which has a memory cell and a selector stacked together in a column. The memory element columns are connected via two vertical wire layers, one of which is above the memory element columns and the other is below the memory element columns. Each memory element can be selected individually at the intersection of one wire on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM) memory, etc. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0094] The data storage device 101 may have a controller 151 including a volatile local memory 153 and at least one processing device 155 .

[0095] The local memory of the controller 151 may be an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines for controlling the operation of the processing device 155, including controlling the communication between the data storage device 101 and the processor (e.g., 133) of the vehicle 111, and other functions described herein. The local memory 153 of the controller 151 may include a read-only memory (ROM) for storing microcode and / or memory registers for storing, for example, memory pointers, extracted data, etc., and / or volatile memory, such as dynamic random access memory (DRAM) and static random access memory (SRAM).

[0096] exist Figure 7 , data storage device 101 includes a neural network accelerator 159 coupled to a controller 151 and / or storage media components 161 to 163.

[0097] For example, the neural network accelerator 159 may be configured to perform matrix arithmetic calculations. The calculations involve the ANN 125 having matrix multiplication and accumulation operations, which may be computationally intensive for a general purpose processor (e.g., 133). Using the neural network accelerator 159 to perform matrix arithmetic calculations may reduce the data to be transmitted to the processor 133 of the vehicle 111 and reduce the computational workload of the processor 133.

[0098] For example, when ANN 125 includes a spiking neural network (SNN), simulation of differential equations used to control activation levels of SNN neurons may be computationally intensive for a general purpose processor (e.g., 133). Neural network accelerator 159 may simulate differential equations using specialized hardware and thus improve the computational efficiency of computer system 131 overall.

[0099] In some embodiments, neural network accelerator 159 is an integrated circuit device separate from controller 151 and / or storage media components 161 to 163. Alternatively or in combination, neural network accelerator 159 is integrated with controller 151 in an integrated circuit package. Figure 8 As illustrated in FIG. 1 , the neural network accelerator 159 may be integrated in at least one of the storage media components 161 to 163 .

[0100] Figure 8 A storage media component 160 for accelerating neural network computations according to one embodiment is shown. For example, Figure 8 The storage media component 160 implements Figure 7 Each or some of the storage media components 161 to 163 in.

[0101] exist Figure 8In the embodiment of the present invention, the storage media component 160 can be housed within an integrated circuit package. The input / output (I / O) interface 170 of the storage media component 160 is configured to process input / output signals in the pins of the integrated circuit package. For example, the input / output signals can include address signals that specify locations in the media unit 174, and data signals representing data written to the media unit 174 at the locations specified by the address signals, or data retrieved from the locations in the media unit 174.

[0102] exist Figure 8 , neural network accelerator 159 is coupled to control logic 178 and / or media unit 174 to perform calculations used in evaluating the output of ANN 125 and / or in training ANN 125.

[0103] For example, the input / output interface 170 may receive an address identifying a matrix stored in a media unit and operated via the neural network accelerator 159. The storage media component 160 may provide the computational results of the neural network accelerator 159 as output data in response to the address, store the output data in a buffer for further operation, and store the output data in a location specified via the address signal in the media unit 174. Thus, the computation performed by the neural network accelerator 159 may be within the storage media component 160, which is proximate to the media unit 174 in which the matrix data is stored. For example, each of the media units 174 may be an integrated circuit die on which a memory cell of a non-volatile memory is formed.

[0104] For example, state data of SNN neurons may be stored in the media unit 174 according to a predetermined pattern. The neural network accelerator 159 may automatically update the state of the SNN neurons over time according to the differential equations used to control the activation levels of the SNN neurons. Optionally, the neural network accelerator 159 is configured to process pulses of neurons in the neural network. Alternatively, the neural network accelerator 159 and / or the processor 133 of the data storage device 101 may be configured to process pulses of neurons and / or accumulation of inputs sent to the SNN.

[0105] Fig. 9 A method of accelerating neural network computation in a motor vehicle according to one embodiment is shown. For example, Fig. 9 The method can be used Figure 7 The data storage device 101 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig. 9 The method can be used with Figure 6 The methods are used in combination.

[0106] At block 221 , the data storage device 101 of the vehicle 111 receives a sensor data stream from at least one sensor (eg, 103 ) configured on the vehicle 111 .

[0107] At block 223 , the data storage device 101 stores at least a portion of the sensor data stream.

[0108] At box 225, the neural network accelerator 159 configured in the data storage device 101 performs at least a portion of the calculation based on the artificial neural network 125 and the sensor data stream.

[0109] At box 227, maintenance service for vehicle 111 is predicted based at least in part on calculations performed by a neural network accelerator 159 configured within the data storage device 101.

[0110] Optionally, at box 229, in the vehicle 111, an artificial neural network (ANN) is trained at least in part using a neural network accelerator and using sensor data streams collected over a predetermined time period (e.g., a period after a new vehicle 111 leaves the factory or after a component is replaced in a maintenance service facility 127).

[0111] For example, the neural network accelerator 159 may be configured on an integrated circuit device that is separate from the controller 151 of the data storage device and / or separate from the storage media components 161-163.

[0112] For example, the neural network accelerator 159 may be configured on an integrated circuit device including the controller 151 of the data storage device 101 , or configured on an integrated circuit device including the storage media component 160 , 161 , or 163 of the data storage device 101 .

[0113] For example, the neural network accelerator 159 may be configured to perform calculations using data stored in the data storage device 101, such as matrix arithmetic calculations for an ANN and / or differential equation simulations for an SNN.

[0114] Examples of matrix arithmetic calculations include matrix multiplication and accumulation operations. After performing calculations using the data stored in the data storage device 101 to produce the results of the matrix arithmetic calculations, the neural network accelerator 159 may provide the results as outputs of the data storage device 101 when the data is retrieved (e.g., in response to a read command). Alternatively or in combination, the results of the matrix arithmetic calculations may be buffered in the data storage device 101 as operands for the next matrix calculation performed in conjunction with the data matrix retrieved from the non-volatile memory via a read command received in the host interface 157.

[0115] When the artificial neural network (ANN) 125 includes a spiking neural network (SNN), the neural network accelerator may be configured to simulate differential equations that control activation levels of neurons in the spiking neural network (SNN). Optionally, the storage media component is configured to store states of neurons in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to automatically update the states of the neurons over time according to the differential equation. For example, the neural network accelerator 159 may be configured to train a spiking neural network (SNN) to detect anomalies via unsupervised machine learning.

[0116] The computations performed by the neural network accelerator 159 based on the artificial neural network (ANN) 125 involve different data types with different usage patterns of the data storage device 101.

[0117] For example, making predictions using the artificial neural network (ANN) 125 includes using data specifying a model of the artificial neural network (ANN) 125, input data provided to the artificial neuron, and output data generated by the artificial neuron.

[0118] The storage capacity of the data storage device 101 can be divided into different portions of the different types of ANN-related data. The different portions can be individually configured to optimize access and storage of corresponding data based on the usage mode of the neural network accelerator 159 and / or the processor 133 of the computer system 131 in which the data storage device 101 is configured.

[0119] The model of the artificial neural network (ANN) 125 may include parameters that specify the static properties of individual artificial neurons in the ANN 125 and the connectivity of neurons in the ANN 125. The model data of the ANN 125 is static and does not change during the prediction calculations performed using the ANN 125. Therefore, the usage mode is usually to read the model data. However, when the updated ANN 125 is installed, the model data of the ANN 125 may change. For example, the vehicle 111 may download the updated ANN 125 from the server 119 to the data storage device 101 of the vehicle 111 to update its prediction capabilities. The model data of the ANN 125 may also change during or after training the ANN 125 using machine learning techniques (e.g., 171 or 175). It is preferred to configure a separate partition or namespace of the data storage device 101 to store the model data, wherein the partition or namespace optimizes the configuration parameters of the memory unit to operate according to a specific usage mode for the model data (e.g., mostly read, rarely updated). For example, when memory cells are implemented using NAND logic gate-based flash memory, the memory cells in the ANN model partition / namespace can be configured to operate in a multi-level cell (MLC) mode, a triple-level cell (TLC) mode, or a quad-level cell (QLC) mode, where each memory cell stores two, three, or four bits for increased storage capability.

[0120] The input data provided to the artificial neurons in the ANN 125 may include external inputs and internal inputs. The external inputs are usually generated by the sensors 103 of the vehicle 111 rather than by the artificial neurons in the ANN 125. The external inputs may be stored in a cyclic manner so that the input data for the latest time period of the predetermined driving length can be found in the data storage device 101. Therefore, it is preferred that a separate partition or namespace of the data storage device 101 is configured to store the external input data, wherein the partition or namespace operates according to configuration parameters that optimize the memory cells (e.g., enhance the durability of cyclic overwrite) for the storage mode of the external input data. For example, when the memory cells are implemented using flash memory based on NAND logic gates, the memory cells in the ANN input partition / namespace may be configured to operate in a single-level cell (SLC) mode, wherein each memory cell stores one bit of data for improved durability in cyclic overwrite operations.

[0121] In some embodiments, an artificial neuron may have state variables that change over time in response to inputs during a predictive computation. For example, the activation level of a spiking neuron may change over time and be considered a dynamic state variable of the spiking neuron. In some embodiments, such state variable data for an artificial neuron has a similar storage usage pattern as external input data; and therefore, the state variable data may be stored in a partition or namespace configured for external input data. In other embodiments, the state variable data for an artificial neuron is maintained in a buffer and is stored less frequently than external inputs; and therefore, another partition / namespace may be configured for storing dynamic state variable data for an artificial neuron.

[0122] The output data generated by the artificial neurons in the ANN 125 can be buffered for further access by the neural network accelerator 159 and / or the processor 133 of the computer system 131. The output data may include external outputs and internal outputs. The external input is generated by the artificial neuron as an output from the ANN 125, such as the result of classification or prediction performed by the ANN 125. The output of the ANN 125 is usually further processed by the processor 133 of the computer system 131. The external input can be saved periodically (e.g., in a manner similar to storing state variable data). Some of the internal outputs and / or external outputs may be internal inputs sent to the artificial neurons in the ANN 125. In general, it may not be necessary to store the internal outputs from the buffer of the data storage device to the storage media component. In some embodiments, when the buffering capacity of the data storage device 101 is insufficient to save the entire state variable data and / or internal output, the data storage device 101 can use the swap partition / namespace to expand the capacity of the buffer. The swap partition / namespace can be configured to optimize random access and to improve durability.

[0123] External outputs and / or dynamic states of neurons may be stored in a separate output partition or namespace in a cyclic manner, such that external output data and / or dynamic states of neurons may be stored periodically and the latest set of external outputs and / or dynamic states may be found in the data storage device 101. External outputs and / or dynamic states of neurons may be selectively stored since some of such data may be regenerated by the ANN from external inputs stored in the input partition or namespace. Preferably, the output partition or namespace is configured to store one or more sets of external outputs and / or dynamic states that cannot be generated from external inputs stored in the input partition or namespace. When storing data in an input / output partition or namespace in a cyclic manner, the oldest stored data set is erased to make room for the latest data set. The ANN input / output partition / namespace may be configured for an optimized continuous write stream to copy data from a buffer of the data storage device to a memory cell in a storage media component of the data storage device.

[0124] Fig.10 A data storage device 101 configured to support neural network computations according to one embodiment is shown. For example, the data storage device 101 may be used to Figure 1 or in a vehicle 111 in 2 to facilitate predictive maintenance and / or support ADAS 105 .

[0125] Similar to Figure 7 The data storage device 101, Fig.10 The data storage device 101 includes a host interface 157 and a controller 151 .

[0126] Similar to Figure 7 The storage media components 161 to 163 in the data storage device 101 can be implemented using a storage media component set Fig.10 The storage capacity 181 of the data storage device 101 is .

[0127] A set of namespaces 183, 185, 187, ... may be created on storage capacity 181 of data storage device 101. Each of the namespaces (e.g., 183, 185, or 187) corresponds to a named portion of storage capacity 181. Logical addresses are defined within each namespace. Address mapping 191 is configured to map logical addresses defined in namespaces 183, 185, 187, ... to storage media components (e.g., Figure 7 Mapping between the physical addresses of the memory cells in 161 to 163) described in .

[0128] Address map 191 may include namespace optimization settings 192 for namespaces 183 , 185 , and 187 .

[0129] For example, the ANN model namespace 183 may be a memory / storage partition configured for model data of the artificial neural network (ANN) 125. The namespace optimization settings 192 optimize memory operations in the ANN model namespace 183 based on the data usage pattern of the ANN model (e.g., mostly read, less update-centric).

[0130] For example, the neuron input namespace 185 may be a memory / storage partition configured for external input data sent to the artificial neural network (ANN) 125. The namespace optimization settings 192 optimize memory operations in the neuron input namespace 185 according to the data usage pattern of the external input data (e.g., enhanced durability to support cyclic overwrites for continuous input data streams for continuous writes).

[0131] For example, the neuron output namespace 187 may be a memory / storage partition / configured for external output data provided from the artificial neural network (ANN) 125. The namespace optimization settings 192 optimize memory operations in the neuron output namespace 187 based on the data usage pattern of the external output data (e.g., improving durability of data that is periodically overwritten by random read / write accesses).

[0132] The data storage device 101 includes a buffer 152 configured to store temporary / intermediate data of the artificial neural network (ANN) 125 , such as internal inputs / outputs of artificial neurons in the ANN 125 .

[0133] Optionally, a swap namespace may be configured in storage capacity 181 to expand the capacity of buffer 152 .

[0134] Optionally, address map 191 includes a mapping between logical memory addresses received in host interface 157 to access data of artificial neurons and identities of artificial neurons. Thus, a read or write command to access one type of data of an artificial neuron in one namespace may cause controller 151 to access another type of data of an artificial neuron in another namespace.

[0135] For example, in response to a request to write external input data for a neuron to the storage capacity 181 of the data storage device 185, the address map 191 may be used to calculate the address of the model parameters of the neuron in the ANN model namespace 183 and read the model parameters into the buffer 152 to allow the neural network accelerator 159 to perform calculations of the output of the neuron. The output of the neuron may be stored in the buffer 152 as an internal input to other neurons (e.g., to reduce write amplification). In addition, the identities of other neurons connected to the neuron may also be retrieved from the ANN model namespace 183 into the buffer 152, which allows the neural network accelerator 159 and / or processor to further process the propagation of the output in the ANN 125. The retrieval of the model data from the ANN model namespace 183 may be performed in parallel with the storage of the external input data into the neuron input namespace 185. Therefore, the processor 133 of the computer system 131 of the vehicle 111 does not have to explicitly send a read command to retrieve the model data from the ANN model namespace 183.

[0136] Similarly, in response to reading the output data of a neuron, address map 191 may be used to calculate the address of the model parameters of the neuron stored in ANN model namespace 183 and read the model parameters into buffer 152 to allow neural network accelerator 159 to apply the internal input in buffer 152 to perform the calculation of the output of the neuron. The calculated output may be provided as a response to reading the output data of the neuron without having to cause data storage device 101 to store the output data in storage media components (e.g., 161 to 163). Thus, processor 133 and / or neural network accelerator 159 may control the calculation of the neuron by writing input to the neuron and / or reading output from the neuron.

[0137] In general, the external input data passed into the ANN 125 may be raw sensor data 121 directly generated by the sensor 103 without being processed by the processor 133 and / or the neural network accelerator 159. Alternatively, indirect sensor data 121 that has been processed by the processor 133 of the ANN 125 from the signal from the sensor 103 may be provided as the external input data. The incoming external input data may be accepted in the host interface 157 and written to the neuron input namespace 185 in a cyclic manner, and automatically buffered in the buffer 152 of the neural network accelerator 159 to generate neuron output using the model stored in the ANN model namespace 183. The output generated by the neural network accelerator 159 may be further buffered as internal input to further apply the model in the ANN model namespace 183. When the external output becomes available, the data storage device 101 may report the completion of the write request by an indication of the availability of the external output. Optionally, the controller 151 and / or the neural network accelerator 159 may generate an internal read command to cause a signal to propagate in the ANN 125 when the external output is generated. Alternatively, the host processor 133 may control the propagation of signals in the ANN 125 by selectively reading the outputs of neurons; and the data storage device 101 may actively buffer the data that may be needed in the buffer 152 to accelerate the ANN calculation.

[0138] Fig.11 The configuration of the namespace 183 for an artificial neural network (ANN) model according to one embodiment is described. For example, Fig.11 The configuration can be implemented in Figure 7 and / or the data storage device 101 described in 10. For example, Fig.11 The setting 193 can be Fig.10 The namespace optimization settings are part of 192.

[0139] Fig.11 The configuration maps the ANN model namespace 183 to at least one storage media component A 161. Preferably, at least one storage media component A 161 is available to the controller 151 in parallel with storage media components (e.g., 163) of other namespaces (e.g., 185 and 187) hosting ANN data. For example, the storage media component A 161 may be in an integrated circuit package that is separate from the integrated circuit packages used for other namespaces (e.g., 185 and 187). Alternatively, the storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 may be formed on separate regions of the integrated circuit die, where the separate regions can be operated substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0140] exist Fig.11 In , setting 197 is optimized for a usage pattern of mostly reads and few updates.

[0141] Fig.12 The configuration of the namespace 185 for inputs sent to an artificial neuron according to one embodiment is described. For example, Fig.11 The configuration can be implemented in Figure 7 and / or the data storage device 101 described in 10. For example, Fig.11 The setting 195 can be Fig.10 The namespace optimization settings are part of 192.

[0142] Fig.12 The configuration maps the neuron input namespace 185 to at least one storage media component B 163. Preferably, at least one storage media component B 163 is available to the controller 151 in parallel with storage media components (e.g., 161) of other namespaces (e.g., 183 and 187) hosting ANN data. For example, storage media component B 163 may be in an integrated circuit package separate from the integrated circuit packages used for other namespaces (e.g., 183 and 187). Alternatively, storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, storage media components 161-163 may be formed on separate regions of the integrated circuit die, where the separate regions may be operated substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0143] exist Fig.12 In the example, setting 197 is optimized for use in a cyclic continuous overwrite mode that records a continuous input data stream sampled at fixed time intervals to enhance endurance.

[0144] Fig.13 The configuration of the namespace 187 for outputs from an artificial neuron according to one embodiment is illustrated. For example, Fig.11 The configuration can be implemented in Figure 7 and / or the data storage device 101 described in 10. For example, Fig.11 The setting 197 can be Fig.10 The namespace optimization settings are part of 192.

[0145] Fig.13The configuration maps the neuron output namespace 187 to at least one storage media component C 162. Preferably, at least one storage media component C 162 is available to the controller 151 in parallel with storage media components (e.g., 161 and 163) of other namespaces (e.g., 183 and 185) hosting ANN data. For example, the storage media component C 162 may be in an integrated circuit package that is separate from the integrated circuit packages used for the other namespaces (e.g., 183 and 185). Alternatively, the storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 may be formed on separate regions of the integrated circuit die, where the separate regions may be operated substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0146] exist Fig.13 In the example, the settings 197 are optimized to periodically overwrite the buffered data by random access. For example, the memory cells are configured to update / overwrite in the neuron output namespace 187 at a higher frequency than in the ANN model namespace 183, but less frequently than in the neuron input namespace 185 by optimizing the settings 193 to 197.

[0147] Fig.14 A method for predictive maintenance supported by model partitioning according to one embodiment is shown. For example, Fig.14 The method can be used Figure 7 or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.14 The method can be used with Figure 6 and / or the method in 9 are used in combination.

[0148] At block 241, the non-volatile memory of the data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) for storing data; and the same type of memory cells in different partitions (e.g., 183 to 187) may be configured in different ways to optimize their performance according to the usage patterns of the data stored in the different partitions (e.g., 183 to 187).

[0149] At box 243, the data storage device 101 stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125 for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include a model partition (e.g., 193) configured to store model data of the artificial neural network 125.

[0150] At block 245 , the data storage device 101 receives a sensor data stream (eg, 121 ) from at least one sensor 103 disposed on the vehicle 111 .

[0151] At block 247 , the controller 151 of the data storage device 101 operates the memory cells in the partitions 183 , 185 , 187 , . . . according to the set of memory operation settings (eg, 193 , 195 , 197 ) in response to the sensor data stream (eg, 121 ).

[0152] At box 249 , the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict maintenance service for the vehicle 111 based on the sensor data stream (eg, 121 ).

[0153] For example, the memory operation setting configures the model partition (e.g., 183) to store three or more bits per memory cell. The memory operation setting may include an address map 191 that maps between neurons in the ANN 125 and inputs to the neurons. When a first address to an input to a neuron in the artificial neural network 125 is received, the first address in the input partition (e.g., 185) separated from the model partition (e.g., 183) may be converted to at least one second address of model data associated with the neuron, so that the properties of the neuron and the identity of the neuron connected to the neuron can be retrieved from the model partition (e.g., 183) without an explicit command from the processor 133. The controller 151 may automatically retrieve the model data associated with the neuron from the model partition (e.g., 183) using the at least one second address in response to receiving the first address. The neural network accelerator 159 may generate an output of the neuron from the input to the neuron and the model data associated with the neuron. In general, the input to the neuron may include outputs from multiple neurons connected to the neuron in the ANN 125. Controller 151 may store the output of the neuron in buffer 152 in data storage device 101 to facilitate accelerated access to the output by host processor 133 and / or neural network accelerator 159.

[0154] Typically, the model data does not change during calculations used to predict maintenance services. For example, the model data may include neuron connectivity data of an artificial neural network and static properties of neurons in the artificial neural network. The memory operation settings (e.g., 192) may configure the model partition (e.g., 183) to store more than one bit per memory cell in the non-volatile memory based on a usage pattern that mostly reads and rarely updates the model data.

[0155] For example, partitions (e.g., 183, 185, 187, ...) in a data storage device may be implemented as namespaces in which logical addresses are defined; and an address mapping 191 in the data storage device is configured to map the namespaces 183, 185, 187, ... to separate storage media components (e.g., 161, 163, 162, ...).

[0156] The model data in the model namespace 183 may be updated during training via machine learning 171 or 175 or during an over-the-air update to the ANN 125 from the server 119 .

[0157] In some embodiments, the controller 151 is configured to retrieve model data associated with a neuron in the artificial neural network from the model partition via the address map 191 in response to an input to a neuron or an output from the neuron being addressed in a partition separate from the model partition 183. In addition, the controller 151 can retrieve model data associated with a neuron from the model partition 183 in parallel with storing input to a neuron in a partition (e.g., 185) separate from the model partition 183.

[0158] Fig.15 A method for predictive maintenance supported by input partitioning according to one embodiment is shown. For example, Fig.15 The method can be used Figure 7 or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.15 The method can be used with Figure 6 , 9 and / or the method in 14 are used in combination.

[0159] At block 261, the non-volatile memory of the data storage device 101 is configured into multiple partitions (e.g., 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) implemented in multiple storage media components (e.g., 161-163).

[0160] At box 263, the data storage device 101 stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125 for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include an input partition (e.g., 185) configured to cyclically store input data for the artificial neural network 125.

[0161] For example, the input partition 185 can be configured to store external inputs rather than internal inputs for the artificial neural network 125. The input data stored in the input partition 185 is independent of the outputs from the neurons in the artificial neural network 125.

[0162] For example, the input data stored in input partition 185 may include a portion of a sensor data stream (e.g., 121). In some embodiments, the input data stored in input partition 185 is calculated from a sensor data stream (e.g., 121) for a subset of neurons in artificial neural network 125.

[0163] For example, the memory operation settings (eg, 195) configure the input partition 185 to store one bit per NAND memory cell in non-volatile memory for enhanced endurance to repeated data erasure and data programming.

[0164] For example, the memory operation setting (eg, 195 ) configures the controller to write input data sequentially into the input partition 185 and overwrite the oldest input data in the input partition 185 with the newest input data received in the data storage device 101 .

[0165] At block 265 , the data storage device 101 receives a sensor data stream (eg, 121 ) from at least one sensor 103 disposed on the vehicle 111 .

[0166] At box 267, the controller 151 of the data storage device 101 operates the memory cells in the partitions 183, 185, 187, ... according to the set of memory operation settings (eg, 193, 195, 197) in response to the sensor data stream (eg, 121).

[0167] At box 269 , the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict maintenance service for the vehicle 111 based on the sensor data stream (eg, 121 ).

[0168] Fig.16 A method for predictive maintenance supported by input partitioning according to one embodiment is shown. For example, Fig.16 The method can be used Figure 7or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.16 The method can be used with Figure 6 , 9 , 14 and / or 15 are used in combination.

[0169] At block 281, non-volatile memory of the data storage device 101 is configured into multiple partitions (eg, 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (eg, NAND flash memory cells) for storing data.

[0170] At box 283, the data storage device 101 stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125 for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include an output partition (e.g., 187) configured to store output data for the artificial neural network 125.

[0171] For example, the output data stored in the output partition (e.g., 187) may include state data of a neuron in the artificial neural network 125. For example, the state data of a neuron in the artificial neural network may identify an activation level of the neuron for generating a spike in the spiking neural network. The activation level may be controlled via a differential equation. Thus, the activation level may change in response to input to the artificial neural network 125 and / or in response to the passage of time.

[0172] For example, the output data may include predictions or classifications generated by the artificial neural network 125 in response to the sensor data stream.

[0173] For example, the memory operation setting configures the output partition to store no more than two bits per memory cell in the non-volatile memory.

[0174] At block 285 , the data storage device 101 receives a sensor data stream (eg, 121 ) from at least one sensor 103 disposed on the vehicle 111 .

[0175] At box 287 , the controller 151 of the data storage device 101 operates the memory cells in the partitions 183 , 185 , 187 , . . . according to the set of memory operation settings (eg, 193 , 195 , 197 ) in response to the sensor data stream (eg, 121 ).

[0176] At box 289 , the computer system 131 having the data storage device 101 uses the artificial neural network 125 to predict maintenance service for the vehicle 111 based on the sensor data stream (eg, 121 ).

[0177] For example, the data storage device 101 may include a buffer 152. The buffer 152 may be implemented via a volatile memory (e.g., SRAM or DRAM) to obtain faster access performance compared to a non-volatile memory (e.g., NAND flash memory) of the data storage device 101. The memory operation setting configures the controller 151 to store the output data in the buffer 152 for access by the processor (e.g., 133) via the host interface 157 during or after storing the output data in the output partition 187.

[0178] For example, the data storage device 101 may include a neural network accelerator 159 coupled to the controller 151. The neural network accelerator is configured to apply inputs provided to neurons in the artificial neural network 125 to model data of the artificial neural network 125 to generate output data through one or more output neurons in the artificial neural network 125. In response to the neural network accelerator 159 completing the calculation of the output data, the controller is configured to provide an indication of the availability of the output data generated by the artificial neural network 125 to the processor (e.g., 133), so that the processor (e.g., 133) can request the data storage device 101 to transmit the output data.

[0179] Optionally, the controller 151 is configured to provide the output data to the processor in parallel with storing the output data in the output partition. For example, the controller 151 may be configured to automatically discard the output data calculated for the previous segment of the sensor data stream if the processor (e.g., 133) does not request that the output data be transmitted to the processor (e.g., 133) within a predetermined time period or before the next version of the output data is available. Optionally, after reporting the availability of the output data to the processor (e.g., 133), the controller 151 may be configured to selectively discard the output data calculated for the previous segment of the sensor data stream based on the response of the processor (e.g., 133) to the output data sent to the processor (e.g., 133). For example, in some cases, a processor (e.g., 133) may request that output data be transmitted to the processor (e.g., 133) without saving the output data to the output partition (e.g., 187); and in other cases, a processor (e.g., 133) may request that output data be transmitted to the processor (e.g., 133) and that the output data be stored to the output partition (e.g., 187).

[0180] Optionally, output data from the artificial neural network 125 may also be stored in output partitions in a round-robin manner (eg, for segments of output data within a time period selected by the processor (eg, 133)).

[0181] For example, external inputs to the artificial neural network 125 may be continuously recorded in the input namespace 185 over the last time period T1. When the sensor data is sampled at a predetermined time interval T2, the input namespace 185 may save the latest T1 / T2 input data set. In contrast, external outputs from the artificial neural network 125 may be selectively recorded in the output namespace 187 (e.g., once per predetermined time period T3, where T3 is a multiple of T2). Output data may be recorded in the output namespace 187 at a lower frequency; and the output namespace 187 may be allocated to store a predetermined number of output data sets (e.g., via continuous writing and writing that occurs in a cyclic manner to keep the last output data set).

[0182] Server 119 , computer system 131 , and / or data storage device 101 may each be implemented as one or more data processing systems.

[0183] The present disclosure includes methods and apparatus for performing the methods described above, including data processing systems for performing these methods, and computer-readable media containing data that, when executed on a data processing system, causes the system to perform these methods.

[0184] A typical data processing system may include an interconnect (eg, a bus and system core logic) that interconnects a microprocessor and memory. A microprocessor is typically coupled to a cache memory.

[0185] The interconnect interconnects the microprocessor and memory together and also interconnects them to input / output (I / O) devices via I / O controllers. The I / O devices may include a display device and / or peripheral devices such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (such as a printer, scanner, mouse, and / or keyboard) are optional.

[0186] The interconnect may include one or more buses connected to each other through various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling USB (Universal Serial Bus) peripheral devices, and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripheral devices.

[0187] The memory may include one or more of: read-only memory (ROM), volatile random access memory (RAM), and non-volatile memory such as a hard drive, flash memory, and the like.

[0188] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires a constant supply of power to refresh or maintain the data in the memory. Nonvolatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other types of memory systems that maintain data even after power is removed from the system. Nonvolatile memory can also be random access memory.

[0189] The non-volatile memory may be a local device directly coupled to the rest of the components in the data processing system. Non-volatile memory that is remote from the system may also be used, such as a network storage device coupled to the data processing system through a network interface such as a modem or Ethernet interface.

[0190] In the present disclosure, some functions and operations are described as being performed by software code or caused by software code to simplify the description. However, this expression is also used to specify that the functions are generated by a processor such as a microprocessor executing code / instructions.

[0191] Alternatively or in combination, the functions and operations as described herein may be implemented using dedicated circuitry with or without software instructions, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments may be implemented using hard-wired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the data processing system.

[0192] While one embodiment may be implemented on a fully functional computer and computer system, various embodiments are capable of being distributed as a computing product in many forms and are applicable regardless of the particular type of machine or computer readable media actually used to achieve the distribution.

[0193] At least some aspects disclosed may be embodied at least in part in software. That is, these techniques may be implemented in a computer system or other data processing system in response to its processor, such as a microprocessor, executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0194] The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a “computer program.” A computer program typically includes one or more sets of instructions at various times in various memories and storage devices in a computer, and when the instruction sets are read and executed by one or more processors in a computer, causes the computer to perform the necessary operations to perform elements related to various aspects.

[0195] Machine-readable media can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various locations including, for example, ROM, volatile RAM, nonvolatile memory, and / or cache memory. Portions of this software and / or data can be stored in any of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or the same communication session. All data and instructions can be obtained before executing the application. Alternatively, portions of data and instructions can be obtained dynamically and in a timely manner when execution is required. Therefore, it is not required that data and instructions are all on machine-readable media at a specific time.

[0196] Examples of computer-readable media include, but are not limited to, non-transitory recordable and non-recordable types of media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CDROM), digital versatile disk (DVD), etc.), etc. Computer-readable media can store instructions.

[0197] The instructions may also be embodied in digital and analog communication links for electrical, optical, acoustical or other forms of propagated signals such as carrier waves, infrared signals, digital signals, etc. However, propagated signals such as carrier waves, infrared signals, digital signals, etc. are not tangible machine-readable media and cannot be configured to store instructions.

[0198] Generally speaking, a machine-readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device with one or more processors, etc.).

[0199] In various embodiments, hard-wired circuits may be used in combination with software instructions to implement the technology.Thus, the technology is neither limited to any specific combination of hardware circuitry and software nor to any specific source for the instructions executed by the data processing system.

[0200] The above description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described to avoid obscuring the description. References to one or an embodiment in the present disclosure do not necessarily refer to the same embodiment; and such references mean at least one.

[0201] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made without departing from the broader spirit and scope as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A data storage device for a vehicle, comprising: a host interface configured to receive a sensor data stream from at least one sensor configured on the vehicle; a non-volatile memory configured into a plurality of partitions, the partitions respectively having different sets of memory operation settings set based on different usage patterns of data related to the artificial neural network, wherein the partitions include a model partition configured to store model data of the artificial neural network; and a controller configured to operate memory cells in the partition according to the memory operation setting set; Wherein the artificial neural network is configured to predict maintenance service for the vehicle based on the sensor data stream. 2 . The data storage device according to claim 1 , wherein the model data does not change during a calculation period used to predict the maintenance service.

3. The data storage device of claim 2, wherein the model data comprises neuron connectivity data of the artificial neural network.

4. A data storage device according to claim 3, wherein the model data further comprises static properties of neurons in the artificial neural network.

5. The data storage device of claim 2, wherein the memory operation settings configure the model partition to store more than one bit per memory cell in the non-volatile memory.

6. The data storage device according to claim 5, further comprising: A neural network accelerator is coupled to the controller, the neural network accelerator being configured to apply inputs provided to neurons in the artificial neural network to the model data to generate outputs from neurons in the artificial neural network.

7. The data storage device of claim 6, wherein the artificial neural network comprises spiking neurons.

8. The data storage device of claim 7, wherein the partition is a namespace in which logical addresses are defined; and an address mapping in the data storage device is configured to map the namespace to a separate storage media component.

9. A data storage device according to claim 8, wherein the model partition is a model namespace; the controller is configured to retrieve model data associated with neurons in the artificial neural network from the model namespace, and the retrieval is in response to inputs to the neurons or outputs from the neurons being addressed in a namespace separate from the model namespace.

10. A data storage device according to claim 9, wherein the controller is configured to retrieve the model data associated with the neuron from the model namespace in parallel with storing the input sent to the neuron in the namespace separate from the model namespace.

11. The data storage device of claim 10, wherein the model data associated with the neuron includes static properties of the neuron and identification of neurons connected to the neuron.

12. A vehicle comprising: at least one processor; at least one sensor configured to generate a sensor data stream; and A data storage device comprising: a host interface configured to receive the sensor data stream; a non-volatile memory configured into a plurality of partitions, the partitions respectively having different sets of memory operation settings set based on different usage patterns of data related to the artificial neural network, wherein the partitions include a model partition configured to store model data of the artificial neural network; and a controller configured to operate memory cells in the partition according to the memory operation setting set; Wherein the artificial neural network is configured to predict maintenance service for the vehicle based on the sensor data stream.

13. The vehicle of claim 12, wherein the model data is updateable in the model partition during training via machine learning.

14. The vehicle of claim 13, wherein the model data comprises neuron connectivity data of the artificial neural network and static properties of neurons in the artificial neural network.

15. The vehicle of claim 13, wherein the memory operation settings configure the model partitions to store three or more bits per memory cell.

16. A vehicle according to claim 15, wherein the controller is configured to retrieve model data associated with neurons in the artificial neural network from the model partition, the retrieval being in response to inputs to or outputs from the neurons being addressed in a partition separate from the model partition.

17. The vehicle of claim 16, wherein the controller is configured to retrieve the model data associated with the neuron from the model partition in parallel with storing inputs to the neuron in the partition separate from the model partition.

18. A method for predicting maintenance service of a vehicle, comprising: configuring a non-volatile memory of the data storage device into a plurality of partitions; storing, for each of the partitions, different memory operation setting sets that are set based on different usage modes of data related to the artificial neural network, wherein the partitions include a model partition configured to store model data of the artificial neural network; In the data storage device, a sensor data stream is received from at least one sensor disposed on the vehicle; In response to the sensor data stream, operating memory cells in the partition according to the memory operation setting set; and Using the artificial neural network, maintenance service for the vehicle is predicted based on the sensor data stream.

19. The method of claim 18, wherein the memory operation settings configure the model partition to store three or more bits per memory cell; and the method further comprises: receiving a first address of an input to a neuron in the artificial neural network, the first address being in a partition separate from the model partition; converting the first address into at least one second address of model data associated with the neuron; and In response to the receiving the first address, the model data associated with the neuron is retrieved from the model partition using the at least one second address.

20. The method of claim 19, further comprising: generating an output of the neuron from the input to the neuron and the model data associated with the neuron; and The output of the neuron is buffered in the data storage device.

Images