Bandwidth optimization for different types of operations scheduled in a data storage device

Abstract

Systems, methods, and apparatuses are provided for bandwidth optimization of different types of operations scheduled in a data storage device. For example, a data storage device of a vehicle includes a storage media component, a controller configured to store data into and retrieve data from the storage media component according to commands received in the data storage device, and an artificial neural network configured to receive, as input, operation parameters indicative of a data access pattern and changing over time, and based on the input, generate a prediction to determine an optimized bandwidth allocation scheme for controlling access to the storage media component by different types of operations in the data storage device. The controller is configured to schedule the operations of the different types accessing the one or more storage media components according to the optimized bandwidth allocation scheme.

Description

TECHNICAL FIELD

[0001] At least some embodiments disclosed herein relate generally to data storage devices, and more particularly, but not by way of limitation, to intelligent bandwidth allocation to different types of operations that access storage media in a data storage device. BACKGROUND

[0002] Recent developments in the field of autonomous driving technology allow computing systems to operate control elements of a motor vehicle at least under some conditions without the assistance of a human operator of the vehicle.

[0003] For example, sensors (e.g., cameras and radar) can be installed on a motor vehicle to detect the surroundings of a vehicle traveling on a roadway. A computing system installed on the vehicle analyzes the sensor inputs to identify conditions and generate control signals or commands for autonomous adjustments of the direction and / or speed of the vehicle, with or without any input from a human operator of the vehicle.

[0004] In some arrangements, when the computing system recognizes a situation in which the computing system can not be able to continue to operate the vehicle in a safe manner, the computing system alerts a human operator of the vehicle and requests the human operator to take over control of the vehicle and drive manually instead of allowing the computing system to drive the vehicle autonomously.

[0005] Autonomous driving and / or advanced driver assistance systems (ADAS) often involve artificial neural networks (ANNs) for recognizing events and / or objects captured in sensor inputs.

[0006] Generally, artificial neural networks (ANNs) use networks of neurons to process inputs to the network and generate outputs from the network.

[0007] For example, each neuron in the network receives a set of inputs. Some of the inputs to a neuron can be outputs of certain neurons in the network; and some of the inputs to a neuron can be inputs provided to the neural network. The input / output relationships among the neurons in the network represent the connectivity of the neurons in the network.

[0008] For example, each neuron can have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function can be in the form of a step function, a linear function, a log-sigmoid (log-sigmoid) function, etc. Different neurons in the network can have different activation functions.

[0009] For example, each neuron can generate a weighted sum of its inputs and its bias and then generate an output that is a function of the weighted sum, calculated using the activation function of the neuron.

[0010] The relationship between the inputs and outputs of an ANN is generally defined by an ANN model, which includes data representing the connectivity of the neurons in the network, as well as the bias, activation function, and synaptic weights of each neuron. Using a given ANN model, a computing device computes the output of the network from a given set of inputs to the network.

[0011] For example, inputs to an ANN network can be generated based on camera inputs; and the output from the ANN network can be the identification of an item, such as an event or an object.

[0012] A spiking neural network (SNN) is a type of ANN that closely mimics natural neural networks. SNN neurons generate spikes as output when their activation level is high enough. The activation level of an SNN neuron mimics the membrane potential of a natural neuron. The output / spikes of an SNN neuron can change the activation level of other neurons that receive the output. The current activation level of an SNN neuron over time is typically modeled using a differential equation and is considered the state of the SNN neuron. An incoming spike from other neurons can push the activation level of a neuron to be higher to reach the threshold for a spike. Once a neuron generates a spike, its activation level is reset. The activation level of an SNN neuron can decay over time, as governed by a differential equation, before a spike is generated. The time element in the behavior of SNN neurons makes SNNs suitable for processing spatio-temporal data. The connectivity of SNNs is typically sparse, which is advantageous for reducing computational workload.

[0013] Generally, an ANN can be trained using a supervised method in which parameters in the ANN are adjusted to reduce the error between known outputs produced by respective inputs and computed outputs produced from applying the inputs to the ANN. Examples of supervised learning / training methods include reinforcement learning, as well as learning that corrects for errors.

[0014] Alternatively or in combination, an ANN can be trained using an unsupervised method in which the exact outputs produced by a given set of inputs are not known prior to completion of training. An ANN can be trained to classify items into a plurality of categories, or to classify data points into clusters.

[0015] A plurality of training algorithms can be employed for complex machine learning / training paradigms. SUMMARY

[0016] One aspect of the present disclosure provides a data storage device, wherein the data storage device comprises: one or more storage media components; a controller configured to store data into and retrieve data from the one or more storage media components according to commands received in the data storage device; an artificial neural network configured to receive as input and over time operating parameters indicative of a data access pattern in the data storage device, and based on the input, generate a prediction to determine an optimized bandwidth allocation scheme for controlling access to the one or more storage media components by different types of operations in the data storage device; wherein the controller is configured to schedule the different types of the operations that access the one or more storage media components according to the optimized bandwidth allocation scheme.

[0017] Another aspect of the present disclosure provides a method, wherein the method comprises: storing data into and retrieving data from one or more storage media components of a data storage device according to commands received in the data storage device; in an artificial neural network, receiving as input and over time operating parameters indicative of a data access pattern in the data storage device; using the artificial neural network, generating a prediction based on the input to determine an optimized bandwidth allocation scheme for controlling access to the one or more storage media components by different types of operations in the data storage device; and scheduling, by a controller of the data storage device, the different types of the operations that access the one or more storage media components according to the optimized bandwidth allocation scheme.

[0018] Another aspect of the present disclosure provides a vehicle, wherein the vehicle comprises: a computer system configured to generate operating parameters of the vehicle over time; and a data storage device configured to identify operating parameters of the data storage device, wherein the operating parameters of the vehicle and the operating parameters of the data storage device are indicative of a data access pattern in the data storage device; wherein the data storage device is configured to generate a prediction based on the operating parameters of the vehicle and the operating parameters of the data storage device as input to an artificial neural network to determine an optimized bandwidth allocation scheme for controlling access to one or more storage media components by different types of operations in the data storage device; and wherein data storage device is configured to schedule the different types of the operations that access the one or more storage media components according to the optimized bandwidth allocation scheme. BRIEF DESCRIPTION OF DRAWINGS

[0019] Embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. The following description with reference to the figures is provided to assist in understanding various aspects of the disclosure.

[0020] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data, according to some embodiments.

[0021] Figure 2 An autonomous vehicle with a data storage device is shown, according to one embodiment.

[0022] Figures 3-5 Training of an artificial neural network for maintenance service prediction is illustrated, according to some embodiments.

[0023] Figure 6 A method of predictive maintenance is shown, according to one embodiment.

[0024] Figure 7 A data storage device to accelerate neural network computation is shown, according to one embodiment.

[0025] Figure 8 A storage media component to accelerate neural network computation is shown, according to one embodiment.

[0026] Figure 9 A method to accelerate neural network computation in an automotive vehicle is shown, according to one embodiment.

[0027] Figure 10 A data storage device configured to support neural network computation is shown, according to one embodiment.

[0028] Figure 11 Configuration of a namespace for an artificial neural network (ANN) model is illustrated, according to one embodiment.

[0029] Figure 12 Configuration of a namespace for input to an artificial neuron is illustrated, according to one embodiment.

[0030] Figure 13 Configuration of a namespace for output from an artificial neuron is illustrated, according to one embodiment.

[0031] Figures 14-16 A method of predictive maintenance with model partition, input partition, and output partition support is shown, according to one embodiment.

[0032] Figure 17 Communicating with a data storage device to implement neural network computation is shown, according to one embodiment.

[0033] Figure 18 Communicating with a data storage device to implement neural network computation is shown, according to one embodiment.

[0034] Figure 19 A method of communicating with a data storage device to implement neural network computation is shown, according to one embodiment.

[0035] Figure 20 A data storage device configured to optimize bandwidth allocation for different types of operations accessing a storage media is shown in accordance with one embodiment.

[0036] Figure 21 A method of bandwidth optimization for different types of operations accessing a storage media in a data storage device is shown in accordance with one embodiment. DETAILED DESCRIPTION

[0037] At least some embodiments disclosed herein provide systems, methods, and apparatuses to process sensor data generated in an automotive vehicle, or another vehicle with or without an advanced driver assistance system (ADAS).

[0038] There can be indications of whether a component of an automotive vehicle needs replacement or maintenance before the component fails or malfunctions during operation of the vehicle. Such indications can not be noticeable to a typical driver or passenger. However, sensor data can be collected and analyzed to predict a probability of component failure. The prediction can be used to schedule a maintenance service, which can reduce or eliminate the chances of an accident where a component of the vehicle fails or malfunctions during operation of the vehicle on a roadway. Furthermore, the prediction allows scheduling a service trip at a convenient time.

[0039] For example, sensors can be installed in an automotive system to collect data during its routine operation; and the sensor data can be used to predict whether and how soon a component needs replacement or maintenance. The sensor data can be provided as input to an artificial neural network (ANN), such as a spiking neural network (SNN), of an artificial intelligence (AI) system to train itself (e.g., using unsupervised machine learning techniques) over a period of time where the vehicle is expected to operate normally. The training customizes the neural network for the specific operating environment of the driver, passenger, or user of the vehicle and the individualized operating habits of the vehicle passenger. Subsequently, when the operating data deviates from the normal pattern, the artificial neural network can detect an abnormal condition. The AI system can be used to recommend a maintenance service and / or identify a component that can need replacement or maintenance.

[0040] Figure 1 A system in which a vehicle is configured with a data storage device to collect and process sensor data is shown in accordance with some embodiments.

[0041] Figure 1The system includes a vehicle 111 having a data storage device 101. Optionally, the vehicle 111 has an advanced driver assistance system (ADAS) 105 and one or more sensors 103 that provide sensor data input to the ADAS 105 and / or the data storage device 101. The data storage device 101 is configured to use an artificial neural network (ANN) 125 to predict / identify a need for maintenance service based on data collected by the sensors 103. The ADAS 105 can be omitted without affecting the predictive maintenance feature. In some implementations, at least a portion of the data produced by the sensors 103 is used in the ADAS 105 for driving assistance and in the ANN 125 for maintenance prediction. Optionally, the output of the ANN 124 is used in the data storage device 101 and in the ADAS 105.

[0042] The sensors 103 can include a digital camera, a lidar, a radar, an ultrasonic sonar, a brake sensor, a speed sensor, an acceleration sensor, an airbag sensor, a global positioning system (GPS) receiver, an audio sensor / microphone, a vibration sensor, a force / stress sensor, a deformation sensor, a motion sensor, a temperature sensor, etc. Some of the sensors 103 can be primarily configured to monitor the environment of the vehicle 111; and other sensors 103 can be primarily configured to monitor operating conditions of one or more components of the vehicle 111 (e.g., an internal combustion engine, an exhaust system, an electric motor, brake pads, tires, a battery, etc.).

[0043] The output of the sensors 103 over time is provided as a sensor data stream to the ADAS 105 and / or the ANN 125 to provide driving assistance (e.g., autonomous driving) and maintenance prediction.

[0044] For example, the vehicle 111 can have a wireless communication device that communicates with a remote server 119 via wireless signals 113 and a communication network 117. The remote server 119 is typically configured at a location remote from the road 102 on which the vehicle 111 is operating. For example, the vehicle 111 can provide some sensor data 121 to the server 119 and receive updates of the ANN 125 from the server 119.

[0045] One example of the communication network 117 is a cellular telephone network having one or more base stations (e.g., 115) that receive wireless signals (e.g., 113). Another example of the communication network 117 is the Internet, in which a wireless local area network signal (e.g., 113) transmitted by the vehicle 111 is received in an access point (e.g., 115) for further transmission to the server 119. In some implementations, the vehicle 111 uses a communication link 107 to a satellite 109 or a communication balloon that communicates with the server 119.

[0046] The server 119 can also communicate with one or more maintenance service facilities (e.g., 127) to receive maintenance service data 123 for the vehicle (e.g., 111). The maintenance service data 123 can include inspection records and / or service records for components of the vehicle (e.g., 111). For example, the inspection records and / or service records can indicate a degree of wear of the components inspected at the maintenance service facility (e.g., 127) during service of the components, identification of malfunctioning or failing components, etc. The sensor data 121 for the vehicle (e.g., 111) for a time period prior to service and the maintenance service data 123 can be used to train the ANN 125 to predict a probability that a component requires maintenance service. The updated ANN 125 can be used to predict and recommend maintenance service for the vehicle 111 based on sensor data 121 received for a recent time period. Alternatively, the updated ANN 125 can be transmitted to the vehicle 111; and the vehicle 111 can use data generated from the sensors 103 during routine operation of the vehicle 111 to predict and recommend maintenance service.

[0047] The data storage device 101 of the vehicle 111 can be configured to record sensor data for a time period that can be used in an ANN for predictive maintenance. Maintenance predictions are typically for relatively long time periods (e.g., days, weeks, and / or months). In contrast, recorded sensor data for review of an accident, collision, or near collision involving an autonomous vehicle is typically for a short time period (e.g., 30 seconds to a few minutes). Thus, a typical black box data recorder configured to record sensor data for review / analysis of an accident or collision is insufficient for predictive maintenance.

[0048] Optionally, the data storage device 101 stores sensor data for a time period leading up to a trip to a maintenance service facility (e.g., 127). The maintenance service facility (e.g., 127) can download the sensor data 121 from the data storage device 101 and provide the sensor data 121 and corresponding maintenance service data 123 to the server 119 to facilitate training of the ANN 125.

[0049] Optionally, or in combination, the data storage device 101 is configured with a machine learning module to customize and / or train the ANN 125 installed in the vehicle 111 for predictive maintenance.

[0050] For example, the machine learning module of the data storage device 101 can be used to calibrate the ANN 125 to account for typical / daily environments in which the vehicle 111 is operated and / or driving preferences / habits of the driver of the vehicle 111.

[0051] For example, during time periods when the vehicle is expected to operate with healthy components in typical / routine environments, sensor data produced by the sensors 103 can be used to train the ANN 125 to recognize patterns of sensor data indicative of unimpeded operation. Such patterns can differ for different vehicles (e.g., 111) based on their routine operating environments and their drivers' driving habits / characteristics. The training allows the ANN 125 to detect deviations from the recognized normal patterns and report maintenance predictions anomalies.

[0052] For example, the ANN 125 can include an SNN configured to classify time-based changes in sensor data and / or detect deviations from known sensor data patterns of a vehicle 111 operating in normal / healthy conditions but in a personalized environment (e.g., a driver's / passenger's daily route) and / or operating in personalized driving habits / patterns.

[0053] Figure 2 An autonomous vehicle 111 having a data storage device 101 is shown in accordance with one embodiment. For example, the autonomous vehicle 111 can be implemented using Figure 2 the vehicle 111 in the system of Figure 1

[0054] Figure 2 The vehicle 111 is configured to have an advanced driver assistance system (ADAS) 105. The ADAS 105 of the vehicle 111 can have an artificial neural network (ANN) 125 for object detection, recognition, identification, and / or classification. The ANN 125 and / or another neural network (e.g., configured in the data storage device 101) can be used to predict a probability that a component of the vehicle 111 needs maintenance service (e.g., repair, replacement, or adjustment).

[0055] Preferably, the data storage device 101 is configured to process sensor data at least partially for predictive maintenance, reducing the computational burden on the processor 133 responsible for operating the ADAS 105 and / or other components (e.g., infotainment system 149).

[0056] The vehicle 111 typically includes an infotainment system 149, a communication device 139, one or more sensors 103, and a computer system 131 connected to some controls of the vehicle 111 (e.g., a steering control 141 for a direction of the vehicle 111, a brake control 143 for stopping the vehicle 111, an acceleration control 145 for a speed of the vehicle 111, etc.). In some embodiments, the vehicle 111 in the system of Figure 1 The vehicle 111 in the system of

[0057] ​Operation of the ADAS 105 requires some of the sensors 103; and some of the sensors 103 are used to collect data related to the health of components of the vehicle 111 that can not be used in the ADAS 105. Optionally, the sensor data generated by the sensors 103 can also be used to predict the likelihood of a component failing soon. Such predictions can be used in the ADAS 105 to take emergency action to put the vehicle in a safe state (e.g., by reducing speed and / or stopping).

[0058] The computer system 131 of the vehicle 111 includes one or more processors 133, the data storage 101, and a memory 135 that stores firmware (or software) 147, including computer instructions and data models for the ADAS 105.

[0059] The one or more sensors 103 of the vehicle can include visible light cameras, infrared cameras, lidar, radar or sonar systems, peripheral device sensors, global positioning system (GPS) receivers, satellite positioning system receivers, brake sensors, and / or airbag sensors. In addition, the sensors 103 can include audio sensors (e.g., microphones) configured to monitor noise from various components and locations in the vehicle 111, vibration sensors pressure sensors, force sensors, stress sensors, and / or deformation sensors configured to measure load on components of the vehicle 111, accelerometers and / or gyroscope sensors that measure motion of some components of the vehicle 111, etc. Such sensors can be used to monitor the operational state and / or health of components for predictive maintenance.

[0060] The sensors 103 can provide real-time time sensor data streams to the computer system 131. The sensor data generated by the sensors 103 of the vehicle 111 can include capturing images of objects using cameras that capture the images using light visible to the human eye, or using infrared light. Image data obtained from at least one sensor of the vehicle is part of the collected sensor data recorded in the data storage 101 and / or as input to the ANN 125. For example, a camera can be used to obtain roadway information for the vehicle 111 to travel on, which the ANN 125 can process to generate control signals for the vehicle 111. For example, a camera can be used to monitor the operational state / health of components of the vehicle 111, which the ANN 125 can process to predict or schedule maintenance services.

[0061] The sensor data produced by the sensors 103 of the vehicle 111 can include audio streams that capture characteristics of sounds at locations on the vehicle 111 (e.g., locations proximate to engines, electric motors, transmission systems, wheels, doors, windows, etc.). The audio data obtained from at least one sensor 103 of the vehicle 111 can be part of the collected sensor data recorded in the data storage 101 and / or as input to the ANN 125. For example, the audio streams can be used to monitor the operational status / health of components of the vehicle 111 (e.g., internal combustion engines, exhaust systems, electric motors, brakes), which the ANN 125 can process to predict or schedule maintenance services.

[0062] The infotainment system 149 can be used to present the predicted or scheduled maintenance services. Optionally, the communication device 139 can establish a connection to a mobile device of a driver of the vehicle 111 to notify the driver of the recommended maintenance services and / or recommended service data, schedule an appointment on a calendar, etc.

[0063] When the vehicle 111 is configured with the ADAS 105, the outputs of the ADAS 105 can be used to control (e.g., 141, 143, 145) the acceleration of the vehicle 111, the speed of the vehicle 111, and / or the direction of the vehicle 111 during autonomous driving.

[0064] Figures 3-5 Training of an artificial neural network for maintenance service prediction is described in accordance with some embodiments.

[0065] In Figure 3 The supervised machine learning module 171 is used to train the artificial neural network 125 to minimize the difference between the service predictions 129 produced from the sensor data 121 and the maintenance service data 123.

[0066] For example, the maintenance service data 123 can identify measured wear of a component over time to predict when a recommended service is to be performed. The sensor data 121 can be used in the ANN 125 to produce a predicted time when the recommended service is to be performed. The supervised machine learning module 171 can adjust the artificial neural network 125 to reduce / minimize the difference between the predicted time based on the sensor data 121 and the time calculated from the wear measurements.

[0067] For example, the maintenance service data 123 can identify components that are replaced or repaired in the maintenance service facility 127. Sensor data 121 recorded over a period of time prior to the replacement or repair of the components can be used to calculate a time at which the replacement or repair will occur. Further, a segment of the sensor data stream over the period of time prior to the replacement or repair can be used in the ANN 125 to generate a prediction of the time at which the replacement or repair will occur. Supervised learning 171 can be used to adjust the ANN 125 to reduce the predicted time of the replacement or repair and the actual time of the replacement or repair.

[0068] Figure 2 Supervised learning 171 can be applied in the server 119 to generate a generic ANN for a population of vehicles based on sensor data of the population of vehicles and their maintenance service data 123.

[0069] Figure 2 Supervised learning 171 can be applied in the vehicle 111 to generate a customized / personalized ANN for a population of vehicles based on sensor data of the vehicle and their maintenance service data 123. For example, a generic ANN can be initially used in the vehicle 111; and the sensor data of the vehicle 111 and their maintenance service data 123 can be used to further train the ANN 125 of the vehicle for customization / personalization of the ANN 125 in the vehicle 111.

[0070] In Figure 4 In the non-supervised machine learning module 175 is used to train or optimize the artificial neural network 125 to facilitate anomaly detection 173. The non-supervised machine learning module 175 is configured to adjust the ANN (e.g., SNN) for classifications, clusters, or recognized patterns in the sensor data 121 such that a degree of deviation from the classifications, clusters, or recognized patterns in the sensor data 121 generated over a recent period of time can be used to signal detection of an anomaly 173. The anomaly detection 173 allows for scheduling the vehicle 111 for an inspection in the maintenance service facility 127. Optionally, after the inspection, the maintenance service data 123 can be used to apply supervised learning 171 to generate a more accurate prediction of the service. Figure 3

[0071] In general, it can be assumed that the vehicle 111 is operating under normal / healthy conditions over a certain period of time. For example, after a new vehicle 111 is initially delivered for use, it can be assumed that the vehicle 111 provides trouble-free service at least for a period of time (e.g., months). For example, after a replacement or repair of a component, it can be assumed that the component provides trouble-free service at least for a period of time (e.g., months or a year). Thus, sensor data 121 obtained during this period of time can be pre-classified as “normal” to use unsupervised learning 175 as in Figure 4 Figure 5 ​​The supervised learning 171 trains the ANN 125.

[0072] For example, sensor data 121 collected during a "normal" service period of the vehicle 111 or component can be classified into several clusters via unsupervised learning 175. Different clusters can correspond to different types of normal conditions (e.g., driving on different routes, driving on roads with different road conditions, driving in different weather conditions, driving in different time periods of the day, driving in different days of the week, driving in different moods of the driver's driving habits). An anomaly is detected when continuous sensor data 121 is classified outside of the "normal" clusters.

[0073] Optionally, as explained in Figure 5 The supervised machine learning 171 can be used to train the ANN 125. During a "normal" service period of the vehicle 111 or component, the expected classification 177 can be used to label the sensor data 121. The supervised learning 171 can be used to minimize the classification difference between the predictions 179 made using the ANN 125 from the sensor data 121 and the expected classification 177. Further, when the sensor data 121 is known to be "abnormal" (e.g., after a diagnosis by a user, driver, or passenger of the vehicle 111 or in a maintenance service facility 127), the expected classification 177 can be changed to "abnormal" for further training of the ANN 125 to directly recognize abnormalities (e.g., instead of relying on deviation from known "normal" clusters to infer abnormalities).

[0074] Thus, the ANN 125 can be trained to recognize abnormal sensor data and schedule maintenance services by estimating the severity of the abnormality.

[0075] Figure 6 A method of predictive maintenance is shown in accordance with one embodiment. For example, Figure 6 The method of Figure 1 or the computer system 131 in the vehicle 111 of Figure 2 of FIG. 1.

[0076] At block 201, a sensor (e.g., 103) installed in a vehicle 111 generates a sensor data stream (e.g., 121) during operation of the vehicle 111 on a road 102.

[0077] At block 203, the sensor data stream (e.g., 121) is provided into an artificial neural network (ANN) 125. For example, the ANN 125 can include a spiking neural network (SNN).

[0078] At block 205, an artificial neural network (ANN) 125 generates a prediction of a maintenance service based on the sensor data stream (e.g., 121).

[0079] At block 207, a data storage device 101 configured on the vehicle stores at least a portion of the sensor data stream (e.g., 121).

[0080] At block 209, the artificial neural network (ANN) is trained using the sensor data stream (e.g., 121) collected over a predetermined period of time from when the vehicle left the factory or a maintenance service facility 127.

[0081] For example, the artificial neural network (ANN) can be configured to identify in a maintenance service a component of the vehicle 111 that needs repair or replacement and / or identify a predicted period of time to component failure or malfunction, or a recommended period of time to a recommended maintenance service of the component before the component failure or malfunction. Thus, the predicted maintenance service can avoid an accident of the component failure or malfunction when the vehicle 111 is running on the road 102.

[0082] For example, the sensor 103 can be a microphone installed near the component, a vibration sensor attached to the component, a pressure sensor installed in the component, a force or stress sensor installed to or attached to the component, a deformation sensor attached to the component, an accelerometer configured to measure a motion parameter of the component.

[0083] Optionally, the data storage device 101, the vehicle 111 of the computer system 131 and / or the server 119 remote from the vehicle can have a machine learning module configured to train the artificial neural network (ANN) 125 during a period of time when the vehicle 111 is assumed to be in a healthy state, for example a predetermined period of time from when the vehicle 111 left the factory or a maintenance service facility 127.

[0084] For example, as explained in Figure 4 the machine learning module can use unsupervised machine learning 175 to train the ANN 125 to recognize / classify normal patterns of the sensor data 121 and thus have the ability to detect anomalies based on deviations from the normal patterns. Alternatively, as explained in Figure 3 or 5, supervised machine learning 171 can be used.

[0085] For example, the unsupervised machine learning 175 can be applied by the data storage device 101 or the computer system 131 of the vehicle 111 during a predetermined period of time when the vehicle and / or the component is known to operate without obstacles or degradation.

[0086] Alternatively or in combination, some of the sensor data 121 stored in the data storage 101 of the vehicle 111 can be uploaded to the server 119 for training the ANN 125.

[0087] In at least some embodiments disclosed herein, the data storage 101 is configured to accelerate the computation of an artificial neural network (ANN) 125 of the vehicle 111.

[0088] For example, in addition to typical operations supporting data access and storage, the data storage 101 can be further configured to perform at least a portion of the computation involving the artificial neural network (ANN) 125, such as generating a prediction (e.g., 129 or 173) or classification (e.g., 179) from the sensor data 121 and / or adjusting the ANN 125 by unsupervised machine learning 175 (e.g., as explained in Figure 4 ) and / or supervised machine learning 171 (e.g., as explained in Figure 3 or 5).

[0089] For example, the computations configured in the data storage 101 can be used to reduce the amount of data transferred to the processor 133 for use or application by the ANN 125 and / or reduce the computational tasks of the processor 133 in evaluating the output of the ANN 125 and / or in training the ANN 125. Such an arrangement can result in faster output from the data storage 101 and / or reduced energy usage, since data moving in and out of memory does not have to be moved to a dedicated, standalone neural network accelerator. The computational capability of the data storage 101 in processing data related to the ANN 125 allows the computer system 131 of the motor vehicle 111 to monitor the health of the automotive components (e.g., in a non-real-time manner, or pseudo-real-time manner) with reduced impact on, or no impact on, the processing of mission-critical tasks (e.g., autonomous driving by the ADAS 105). Furthermore, the computational capability of the data storage 101 can be used to accelerate the processing of sensor data for the ADAS 105 and thus improve the processing of mission-critical tasks.

[0090] Figure 7 A data storage 101 accelerating neural network computation is shown according to one embodiment. For example, the data storage 101 of Figure 7 may be used to implement the data storage 101 of the vehicle 111 as explained in Figure 1 or 2.

[0091] In Figure 7In particular embodiments, the data storage device 101 has a host interface 157 configured to communicate with a processor (e.g., 133). For example, communication between the processor (e.g., 133) and the host interface 157 can be at least partially in accordance with a communication protocol for a Peripheral Component Interconnect Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Universal Serial Bus (USB) bus, and / or a Storage Area Network (SAN).

[0092] For example, the host interface 157 can be used to receive sensor data 121 generated by sensors 103 of the vehicle 111, to optionally store a portion of the sensor data 121 in the storage media components 161-163.

[0093] For example, each of the storage media components 161-163 can be a memory integrated circuit configured to store data. For example, a media component 161 or 163 can include one or more integrated circuit dies embedded in an integrated circuit package. The integrated circuit dies can have a plurality of memory cells formed thereon to store data.

[0094] Generally, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and can retain stored data even when not powered.

[0095] Examples of non-volatile memory include flash memory, memory cells formed based on “and not” (NAND) logic gates, “or not” (NOR) logic gates, phase change memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point storage devices, and memory devices. Cross-point memory devices use memory elements with fewer transistors, each of which has memory cells and selectors stacked together in columns. The columns of memory elements are connected via two perpendicular layers of wires, with one layer above the columns of memory elements and the other layer below the columns of memory elements. Each memory element can be individually selected at the intersection of one wire on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), and electrically erasable programmable read only memory (EEPROM) memory, etc. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0096] The data storage device 101 can have a controller 151 that includes a volatile local memory 153 and at least one processing device 155.

[0097] The local memory of the controller 151 can be an embedded memory configured to store instructions for performing various processes, operations, logic flows, and routines of the operation of the processing device 155, including handling communications between the data storage device 101 and the processor (e.g., 133) of the vehicle 111, as well as other functions described herein. The local memory 153 of the controller 151 can include read-only memory (ROM) for storing microcode, and / or memory registers to store, for example, memory pointers, fetched data, etc., and / or volatile memory, such as dynamic random access memory (DRAM) and static random access memory (SRAM).

[0098] In Figure 7 , the data storage device 101 includes a neural network accelerator 159 coupled to the controller 151 and / or the storage media components 161-163.

[0099] For example, the neural network accelerator 159 can be configured to perform matrix arithmetic computations. The computations involve the ANN 125 having matrix multiplication and accumulation operations, which can be computationally intensive for a general-purpose processor (e.g., 133). Performing the matrix arithmetic computations using the neural network accelerator 159 can reduce the data to be transferred to the processor 133 of the vehicle 111 and reduce the computational workload of the processor 133.

[0100] For example, when the ANN 125 includes a spiking neural network (SNN), the simulation of differential equations for controlling the activation levels of SNN neurons can be computationally intensive for a general-purpose processor (e.g., 133). The neural network accelerator 159 can simulate the differential equations using specialized hardware and thus improve the computational efficiency of the computer system 131 as a whole.

[0101] In some implementations, the neural network accelerator 159 is an integrated circuit device separate from the controller 151 and / or the storage media components 161-163. Alternatively or in combination, the neural network accelerator 159 is integrated with the controller 151 in an integrated circuit package. Further, as Figure 8 explained in , the neural network accelerator 159 can be integrated in at least one of the storage media components 161-163.

[0102] Figure 8 A storage media component 160 that accelerates neural network computations according to one embodiment is shown. For example, the storage media component 160 of Figure 8 may be used to implement each or some of the storage media components 161-163 in Figure 7 .

[0103] In Figure 8In particular embodiments, the storage media component 160 can be housed within an integrated circuit package. The input / output (I / O) interface 172 of the storage media component 160 is configured to handle input / output signals in the pins of the integrated circuit package. For example, the input / output signals can include address signals that specify locations in the media cells 176, and data signals that represent data written in the media cells 176 at the locations specified via the address signals, or data retrieved from the locations in the media cells 176.

[0104] In particular embodiments, the neural network accelerator 159 is coupled with the control logic 174 and / or the media cells 176 to perform computations used in evaluating the output of the ANN 125 and / or in training the ANN 125. Figure 8

[0105] For example, the input / output interface 172 can receive an address that identifies a matrix stored in the media cells and operated on via the neural network accelerator 159. The storage media component 160 can provide the results of the computations of the neural network accelerator 159 as output data responsive to the address, store the output data in a buffer for further operations, store the output data into a location in the media cells 176 specified via the address signals. Thus, the computations performed by the neural network accelerator 159 can be within the storage media component 160, which is proximate to the media cells 176 in which the matrix data is stored. For example, each of the media cells 176 can be an integrated circuit die of memory cells formed with non-volatile memory thereon.

[0106] For example, the state data of the SNN neurons can be stored in the media cells 176 according to a predetermined pattern. The neural network accelerator 159 can automatically update the states of the SNN neurons over time according to differential equations used to control the activation levels of the SNN neurons. Optionally, the neural network accelerator 159 is configured to handle the spikes of the neurons in the neural network. Alternatively, the neural network accelerator 159 and / or the processor 133 of the data storage device 101 can be configured to handle the accumulation of the spikes of the neurons and / or the inputs to the SNN.

[0107] Figure 9 A method of accelerating neural network computations in a motor vehicle is shown, according to one embodiment. For example, Figure 9 The method of Figure 7 The data storage device 101 of Figure 8 The storage media component 160 of Figure 1 Or the vehicle 111 of 2. For example, Figure 9 The method of Figure 6 The method of

[0108] ​At block 221, the data storage device 101 of the vehicle 111 receives a sensor data stream from at least one sensor (e.g., 103) configured on the vehicle 111.

[0109] At block 223, the data storage device 101 stores at least a portion of the sensor data stream.

[0110] At block 225, the neural network accelerator 159 configured within the data storage device 101 performs at least a portion of the computations based on the artificial neural network 125 and the sensor data stream.

[0111] At block 227, a maintenance service for the vehicle 111 is predicted based at least in part on the computations performed by the neural network accelerator 159 configured within the data storage device 101.

[0112] Optionally, at block 229, the artificial neural network (ANN) is trained in the vehicle 111 using at least in part the neural network accelerator and using sensor data streams collected over a predetermined time period, such as a time period after the new vehicle 111 is manufactured or after a component is replaced in the maintenance service facility 127.

[0113] For example, the neural network accelerator 159 can be configured on an integrated circuit device that is separate from the controller 151 of the data storage device and / or separate from the storage media components 161-163.

[0114] For example, the neural network accelerator 159 can be configured on an integrated circuit device that includes the controller 151 of the data storage device 101, or on an integrated circuit device that includes the storage media components 160, 161, or 163 of the data storage device 101.

[0115] For example, the neural network accelerator 159 can be configured to perform computations using data stored in the data storage device 101, such as matrix arithmetic computations for ANNs and / or differential equation simulations for SNNs.

[0116] Examples of matrix arithmetic computations include matrix multiplication and accumulation operations. After performing computations using data stored in the data storage device 101 to produce results of the matrix arithmetic computations, the neural network accelerator 159 can provide the results as output of the data storage device 111 when data is retrieved (e.g., in response to a read command). Alternatively or in combination, the results of the matrix arithmetic computations can be buffered in the data storage device 101 as operands for a next matrix computation performed in conjunction with data matrices retrieved from the non-volatile memory via a read command received in the host interface 157.

[0117] When the artificial neural network (ANN) 125 comprises a spiking neural network (SNN), the neural network accelerator can be configured to simulate a differential equation that controls an activation level of a neuron in the spiking neural network (SNN). Optionally, the storage media component is configured to store a state of a neuron in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to automatically update the state of the neuron over time according to the differential equation. For example, the neural network accelerator 159 can be configured to train a spiking neural network (SNN) to detect anomalies via unsupervised machine learning.

[0118] The computations performed by the neural network accelerator 159 according to the artificial neural network (ANN) 125 involve different data types with different usage patterns of the data storage 101.

[0119] For example, using the artificial neural network (ANN) 125 for prediction involves data specifying a model of the artificial neural network (ANN) 125, input data provided to artificial neurons, and output data produced by the artificial neurons.

[0120] The storage capacity of the data storage 101 can be partitioned into different portions of the different types of ANN-related data. The different portions can be individually configured to optimize access and storage of the corresponding data according to usage patterns of the neural network accelerator 159 and / or the processor 133 of the computer system 131 in which the data storage 101 is configured.

[0121] The model of the artificial neural network (ANN) 125 can include parameters that specify the static properties of individual artificial neurons in the ANN 125 and the connectivity of neurons in the ANN 125. The model data of the ANN 125 is static and does not change during the prediction computations performed using the ANN 125. Thus, it is generally a read- dominated usage pattern of the model data. However, the model data of the ANN 125 can change when an updated ANN 125 is installed. For example, the vehicle 111 can download an updated ANN 125 from the server 119 to the data storage 101 of the vehicle 111 to update its prediction capabilities. The model data of the ANN 125 can also change during or after training the ANN 125 using machine learning techniques (e.g., 171 or 175). It is preferable to configure a separate partition or namespace of the data storage 101 to store the model data, wherein the partition or namespace operates with configuration parameters of the memory cells that are optimized for the specific usage pattern (e.g., mostly read, less update) of the model data. For example, when the memory cells are implemented using NAND logic gate based flash memory, the memory cells in the ANN model partition / namespace can be configured to operate in a multi-level cell (MLC) mode, a triple-level cell (TLC) mode, or a quad-level cell (QLC) mode, wherein each memory cell stores two, three, or four bits for increased storage capacity.

[0122] The input data provided to the artificial neurons in the ANN 125 can include external inputs and internal inputs. The external inputs are typically generated by the sensors 103 of the vehicle 111 and not by the artificial neurons in the ANN 125. The external inputs can be saved in a circular fashion such that the input data for the most recent time period of a predetermined driving length can be found in the data storage 101. Thus, it is preferable to configure a separate partition or namespace of the data storage 101 to store the external input data, wherein the partition or namespace operates with configuration parameters of the memory cells (e.g., enhanced endurance for circular overwrite) that are optimized for the storage pattern of the external input data. For example, when the memory cells are implemented using NAND logic gate based flash memory, the memory cells in the ANN input partition / namespace can be configured to operate in a single-level cell (SLC) mode, wherein each memory cell stores one bit of data for improved endurance in circular overwrite operations.

[0123] In some implementations, artificial neurons can have state variables that change over time in response to inputs during prediction computations. For example, the activation level of a spiking neuron can change over time and be considered a dynamic state variable of the spiking neuron. In some implementations, such state variable data of artificial neurons has similar storage usage patterns as external input data; and thus, the state variable data can be stored in the partition or namespace configured for external input data. In other implementations, state variable data of artificial neurons are kept in buffers and are stored less frequently than external inputs; and thus, another partition / namespace can be configured for storing dynamic state variable data of artificial neurons.

[0124] Output data generated by artificial neurons in ANN 125 can be buffered for further access by neural network accelerator 159 and / or processor 133 of computer system 131. The output data can include external outputs and internal outputs. The external inputs are generated by artificial neurons as output from ANN 125, for example, as a result of classification or prediction made by ANN 125. The output of ANN 125 is typically further processed by processor 133 of computer system 131. The external inputs can be saved periodically (e.g., in a manner similar to storing state variable data). Some of the internal outputs and / or external outputs can be internal inputs to artificial neurons in ANN 125. In general, it can not be necessary to store internal outputs from buffers of data storage device to storage media components. In some implementations, when the buffering capacity of data storage device 101 is insufficient to save the entire state variable data and / or internal outputs, data storage device 101 can use swap partitions / namespaces to extend the capacity of the buffers. The swap partitions / namespaces can be configured for optimizing random access and for improving durability.

[0125] The external outputs and / or dynamic states of the neurons can be saved in a circular fashion in a separate output partition or namespace, such that the external output data and / or dynamic states of the neurons can be stored periodically, and the most recent set of external outputs and / or dynamic states can be found in the data storage 101. The external outputs and / or dynamic states of the neurons can be selectively stored, as some of such data can be regenerated by the ANN from the external inputs stored in the input partition or namespace. Preferably, the output partition or namespace is configured to store one or more sets of external outputs and / or dynamic states that cannot be regenerated from the external inputs stored in the input partition or namespace. In storing data in the input / output partitions or namespaces in a circular fashion, the oldest set of stored data is erased to make room for the newest set of data. The ANN input / output partitions / namespaces can be configured for optimized continuous write streams to copy data from the buffers of the data storage to the memory cells in the storage media components of the data storage.

[0126] Figure 10 A data storage 101 configured to support neural network computations is shown in accordance with one embodiment. For example, the data storage 101 can be used in a vehicle 111 in FIG. 1 or 2 to facilitate predictive maintenance and / or support ADAS 105. Figure 1

[0127] Similar to the data storage 101 of FIG. 1, Figure 7 The data storage 101 of FIG. 1 includes a host interface 157 and a controller 151. Figure 10

[0128] Similar to the storage media components 161-163 in the data storage 101 of FIG. 1, Figure 7 The storage capacity 181 of the data storage 101 of FIG. 1 can be implemented using a set of storage media components. Figure 10

[0129] A set of namespaces 183, 185, 187,... can be created on the storage capacity 181 of the data storage 101. Each of the namespaces (e.g., 183, 185, or 187) corresponds to a named portion of the storage capacity 181. Logical addresses are defined within each namespace. An address map 191 is configured to map between the logical addresses defined in the namespaces 183, 185, 187,... to the physical addresses of the memory cells in the storage media components (e.g., 161-163) as explained in FIG. 1. Figure 7

[0130] The address map 191 can include namespace optimization settings 192 for the namespaces 183, 185, and 187.

[0131] ​​​​For example, ANN model namespace 183 can be a memory / storage partition configured for model data of an artificial neural network (ANN) 125. Namespace optimization settings 192 optimize memory operations in ANN model namespace 183 according to data usage patterns of the ANN model (e.g., mostly read, less update-centric).

[0132] For example, neuron input namespace 185 can be a memory / storage partition configured for external input data to an artificial neural network (ANN) 125. Namespace optimization settings 192 optimize memory operations in neuron input namespace 185 according to data usage patterns of the external input data (e.g., enhanced durability to support circular overwrite of a continuous input data stream for continuous writes).

[0133] For example, neuron output namespace 187 can be a memory / storage partition configured for external output data provided from an artificial neural network (ANN) 125. Namespace optimization settings 192 optimize memory operations in neuron output namespace 187 according to data usage patterns of the external output data (e.g., improve durability to periodically overwrite data through random read / write access).

[0134] Data storage device 101 includes buffer 152 configured to store temporary / intermediate data of artificial neural network (ANN) 125, such as internal input / output of artificial neurons in ANN 125.

[0135] Optionally, a swap namespace can be configured in storage capacity 181 to extend the capacity of buffer 152.

[0136] Optionally, address mapping 191 includes a mapping between logical memory addresses received in host interface 157 to access data of artificial neurons and identities of artificial neurons. Thus, a read or write command to access one type of data of an artificial neuron in one namespace can cause controller 151 to access another type of data of the artificial neuron in another namespace.

[0137] For example, in response to a request to write external input data for a neuron into the storage capacity 181 of the data storage 185, the address map 191 can be used to compute the address of the model parameters of the neuron in the ANN model namespace 183 and read the model parameters into the buffer 152 to allow the neural network accelerator 159 to perform the computation of the output of the neuron. The output of the neuron can be saved in the buffer 152 as internal input to other neurons (e.g., to reduce write amplification). Further, the identity of other neurons connected to the neuron can also be retrieved from the ANN model namespace 183 into the buffer 152, which allows the neural network accelerator 159 and / or the processor to further process the propagation of the output in the ANN 125. The retrieval of the model data from the ANN model namespace 183 can be performed in parallel with the storage of the external input data into the neuron input namespace 185. Thus, the processor 133 of the computer system 131 of the vehicle 111 does not have to explicitly send a read command to retrieve the model data from the ANN model namespace 183.

[0138] Similarly, in response to reading the output data of a neuron, the address map 191 can be used to compute the address of the model parameters of the neuron stored in the ANN model namespace 183 and read the model parameters into the buffer 152 to allow the neural network accelerator 159 to apply the internal input in the buffer 152 to perform the computation of the output of the neuron. The computed output can be provided as a response to reading the output data of the neuron without having to cause the data storage 101 to store the output data in a storage media component (e.g., 161-163). Thus, the processor 133 and / or the neural network accelerator 159 can control the computation of a neuron via writing input to the neuron and / or reading output from the neuron.

[0139] Generally, the external input data incoming to the ANN 125 can be raw sensor data 121 generated directly by the sensors 103 without processing by the processor 133 and / or the neural network accelerator 159. Alternatively, indirect sensor data 121 that has been processed by the processor 133 of the ANN 125 from signals from the sensors 103 can be provided as the external input data. The incoming external input data can be accepted in the host interface 157 and written in a round-robin fashion to the neuron input namespace 185 and automatically buffered in the buffers 152 of the neural network accelerator 159 to generate neuron outputs using the models stored in the ANN model namespace 183. The outputs generated by the neural network accelerator 159 can be further buffered as internal inputs to further apply the models in the ANN model namespace 183. When the external outputs become available, the data storage 101 can report completion of the write request by an indication of availability of the external outputs. Optionally, the controller 151 and / or the neural network accelerator 159 can generate internal read commands to propagate signals in the ANN 125 when the external outputs are generated. Alternatively, the host processor 133 can control the propagation of signals in the ANN 125 by selectively reading the outputs of neurons; and the data storage 101 can proactively buffer data that can be needed in the buffers 152 to speed up ANN computations.

[0140] Figure 11 The configuration of the namespace 183 for an artificial neural network (ANN) model according to one embodiment is described. For example, Figure 11 The configuration of the namespace 183 for an artificial neural network (ANN) model according to one embodiment is described. For example, Figure 7 and / or the data storage 101 described in 10. For example, Figure 11 The settings 193 of the namespace 183 for an artificial neural network (ANN) model according to one embodiment are described. For example, Figure 10 The namespace optimization settings 192 of the namespace 183 for an artificial neural network (ANN) model according to one embodiment are described. For example,

[0141] Figure 11 The configuration of the namespace 183 for an artificial neural network (ANN) model according to one embodiment is described. For example,

[0142] InFigure 11 In one embodiment, the settings 197 are optimized for a use pattern that is mostly reads and few updates.

[0143] Figure 12 The configuration of the namespace 185 for inputs to artificial neurons is illustrated according to one embodiment. For example, Figure 11 The configuration of the namespace 185 for inputs to artificial neurons can be implemented in Figure 7 and / or the data storage 101 illustrated in 10. For example, Figure 11 The settings 195 of the namespace 185 for inputs to artificial neurons can be part of the namespace optimization settings 192. Figure 10

[0144] The configuration of the namespace 185 for inputs to artificial neurons maps the neuron input namespace 185 to at least one storage media component B 163. Preferably, the at least one storage media component B 163 is available to the controller 151 in parallel with the storage media components (e.g., 161) hosting other namespaces (e.g., 183 and 187) of ANN data. For example, the storage media component B 163 can be in an integrated circuit package separate from the integrated circuit package used for the other namespaces (e.g., 183 and 187). Alternatively, the storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 can be formed on separate regions of an integrated circuit die, where the separate regions can be operated substantially in parallel (e.g., for reads, for erases, and / or for writes). Figure 12 In one embodiment, the settings 197 are optimized for a use pattern that is mostly reads and few updates.

[0145] Figure 12 The configuration of the namespace 187 for outputs from artificial neurons is illustrated according to one embodiment. For example,

[0146] The configuration of the namespace 187 for outputs from artificial neurons can be implemented in Figure 13 and / or the data storage 101 illustrated in 10. For example, Figure 11 The settings 197 of the namespace 187 for outputs from artificial neurons can be part of the namespace optimization settings 192. Figure 7 Figure 11 Figure 10

[0147] Figure 13 ​​​The configuration of the at least one storage media component C 162 maps the neuron output namespace 187 to the at least one storage media component C 162. Preferably, the at least one storage media component C 162 is available to the controller 151 in parallel with the storage media components (e.g., 161 and 163) hosting other namespaces (e.g., 183 and 185) of ANN data. For example, the storage media component C 162 can be in an integrated circuit package that is separate from the integrated circuit package used for the other namespaces (e.g., 183 and 185). Alternatively, the storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 can be formed on separate regions of an integrated circuit die, where the separate regions can operate substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0148] In Figure 13 , the setting 197 is optimized for a use pattern that periodically overwrites buffered data through random access. For example, the memory cells are configured via the optimized settings 193-197 to update / overwrite in the neuron output namespace 187 at a higher frequency than in the ANN model namespace 183, but at a lower frequency than in the neuron input namespace 185.

[0149] Figure 14 A method of predictive maintenance is shown that is supported by model partitioning according to one embodiment. For example, Figure 14 The method of Figure 7 or 10 can use the data storage 101 and / or Figure 8 The storage media component 160 of Figure 1 or 2 can be implemented in the vehicle 111 of Figure 14 The method of Figure 6 may be used in combination with the method of

[0150] At block 241, the non-volatile memory of the data storage 101 is configured into a plurality of partitions (e.g., 183, 185, 187,...). For example, the non-volatile memory can have the same type of memory cells (e.g., NAND flash memory cells) for storing data; and the same type of memory cells in different partitions (e.g., 183-187) can be configured differently to optimize their performance according to the use pattern of the data stored in the different partitions (e.g., 183-187).

[0151] At block 243, the data storage device 101 stores, for each partition (e.g., 183, 185, 187,...), a different set of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125, where the partition (e.g., 183, 185, 187,...) contains a model partition (e.g., 193) configured to store model data of the artificial neural network 125.

[0152] At block 245, the data storage device 101 receives a sensor data stream (e.g., 121) from at least one sensor 103 configured on the vehicle 111.

[0153] At block 247, the controller 151 of the data storage device 101 operates the memory cells in the partition 183, 185, 187,... according to the set of memory operation settings (e.g., 193, 195, 197) in response to the sensor data stream (e.g., 121).

[0154] At block 249, the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict a maintenance service for the vehicle 111 based on the sensor data stream (e.g., 121).

[0155] For example, the memory operation settings configure the model partition (e.g., 183) to store three or more bits per memory cell. The memory operation settings can include an address mapping 191 that maps between neurons in the ANN 125 and inputs to the neurons. When a first address is received that is directed to an input to a neuron in the artificial neural network 125, a first address in an input partition (e.g., 185) that is separate from the model partition (e.g., 183) can be translated to at least one second address of model data associated with the neuron such that properties of the neuron and identities of neurons connected to the neuron can be retrieved from the model partition (e.g., 183) without an explicit command from the processor 133. The controller 151 can automatically retrieve the model data associated with the neuron from the model partition (e.g., 183) using the at least one second address in response to receiving the first address. The neural network accelerator 159 can generate an output of the neuron from the input to the neuron and the model data associated with the neuron. In general, the input to the neuron can include outputs from multiple neurons connected to the neuron in the ANN 125. The controller 151 can save the output of the neuron in a buffer 152 in the data storage device 101 to facilitate accelerated access to the output by the host processor 133 and / or the neural network accelerator 159.

[0156] Generally, model data does not change during computation to predict maintenance service. For example, model data can include neuron connectivity data of an artificial neural network and static attributes of neurons in the artificial neural network. The memory operation settings (e.g., 192) can configure the model partition (e.g., 183) to store more than one bit per memory cell in the non-volatile memory based on a usage pattern that there are mostly reads and few updates to the model data.

[0157] For example, the partitions (e.g., 183, 185, 187,...) in the data storage device can be implemented as namespaces in which logical addresses are defined; and the address mapping 191 in the data storage device is configured to map the namespaces 183, 185, 187,... to separate storage media components (e.g., 161, 163, 162,...).

[0158] The model data in the model namespace 183 can be updated during training via the machine learning 171 or 175 or during over-the-air updates of the ANN 125 from the server 119.

[0159] In some implementations, the controller 151 is configured to retrieve, responsive to an input addressed to a neuron in a partition that is separate from the model partition 183 or an output from the neuron, model data associated with the neuron in the artificial neural network from the model partition via the address mapping 191. Further, the controller 151 can retrieve model data associated with a neuron from the model partition 183 in parallel with storing an input to the neuron in a partition (e.g., 185) that is separate from the model partition 183.

[0160] Figure 15 A method of predictive maintenance supported by an input partition is shown in accordance with one embodiment. For example, Figure 15 The method of Figure 7 or 10 can use the data storage device 101 and / or Figure 8 the storage media components 160 of Figure 1 or 2. For example, Figure 15 The method of Figure 6 , 9 and / or 14 can be used in combination with the method of

[0161] At block 261, the non-volatile memory of the data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187,...). For example, the non-volatile memory can have the same type of memory cells (e.g., NAND flash memory cells) implemented in a plurality of storage media components (e.g., 161-163).

[0162] At block 263, the data storage device 101 stores, for each partition (e.g., 183, 185, 187,...), a different set of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125, where the partition (e.g., 183, 185, 187,...) contains an input partition (e.g., 185) configured to cyclically store input data for the artificial neural network 125.

[0163] For example, the input partition 185 can be configured to store external input, but not internal input, for the artificial neural network 125. The input data stored in the input partition 185 is independent of the output from neurons in the artificial neural network 125.

[0164] For example, the input data stored in the input partition 185 can include a portion of a sensor data stream (e.g., 121). In some embodiments, the input data stored in the input partition 185 is computed from the sensor data stream (e.g., 121) for a subset of neurons in the artificial neural network 125.

[0165] For example, the memory operation settings (e.g., 195) configure the input partition 185 to store one bit per NAND memory cell in the non-volatile memory for enhanced durability of repeated data erasure and data programming.

[0166] For example, the memory operation settings (e.g., 195) configure the controller to sequentially write input data into the input partition 185 and overwrite the oldest input data in the input partition 185 with the newest input data received in the data storage device 101.

[0167] At block 265, the data storage device 101 receives a sensor data stream (e.g., 121) from at least one sensor 103 configured on the vehicle 111.

[0168] At block 267, the controller 151 of the data storage device 101 operates the memory cells in the partitions 183, 185, 187,... according to the set of memory operation settings (e.g., 193, 195, 197) in response to the sensor data stream (e.g., 121).

[0169] At block 269, the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict a maintenance service for the vehicle 111 based on the sensor data stream (e.g., 121).

[0170] Figure 16 A method of predictive maintenance supported by an input partition is shown in accordance with one embodiment. For example, Figure 16 The method of Figure 7or the data storage device 101 of 10 and / or Figure 8 The storage media assembly 160 of 10 is implemented in Figure 1 For example, the method of 10 can be used in combination with Figure 16 the method of 11, 12, 13, 14, and / or 15. Figure 6 9

[0171] At block 281, the non-volatile memory of the data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187,...). For example, the non-volatile memory can have the same type of memory cells (e.g., NAND flash memory cells) for storing data.

[0172] At block 283, the data storage device 101 stores, for the partitions (e.g., 183, 185, 187,...) respectively, different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125, wherein the partitions (e.g., 183, 185, 187,...) include an output partition (e.g., 187) configured to store output data for the artificial neural network 125.

[0173] For example, the output data stored in the output partition (e.g., 187) can include state data of neurons in the artificial neural network 125. For example, the state data of a neuron in the artificial neural network can identify an activation level of the neuron for generating a spike in a spiking neural network. The activation level can be controlled via a differential equation. Thus, the activation level can change in response to inputs to the artificial neural network 125 and / or in response to the passage of time.

[0174] For example, the output data can include a prediction or classification generated by the artificial neural network 125 in response to the stream of sensor data.

[0175] For example, the memory operation settings configure the output partition to store no more than two bits per memory cell in the non-volatile memory.

[0176] At block 285, the data storage device 101 receives a stream of sensor data (e.g., 121) from at least one sensor 103 configured on the vehicle 111.

[0177] At block 287, the controller 151 of the data storage device 101 operates the memory cells in the partitions 183, 185, 187,... in accordance with the sets of memory operation settings (e.g., 193, 195, 197) in response to the stream of sensor data (e.g., 121).

[0178] ​​At block 289, the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict a maintenance service for the vehicle 111 based on the sensor data stream (e.g., 121).

[0179] For example, the data storage device 101 can include a buffer 152. The buffer 152 can be implemented via volatile memory (e.g., SRAM or DRAM) to obtain faster access performance compared to non-volatile memory (e.g., NAND flash memory) of the data storage device 101. The memory operation setting configures the controller 151 to store the output data in the buffer 152 for access by the processor (e.g., 133) via the host interface 157 during or after storing the output data into the output partition 187.

[0180] For example, the data storage device 101 can include a neural network accelerator 159 coupled to the controller 151. The neural network accelerator is configured to apply inputs provided to neurons in the artificial neural network 125 to model data of the artificial neural network 125 to generate output data by one or more output neurons in the artificial neural network 125. In response to the neural network accelerator 159 completing computation of the output data, the controller is configured to provide an indication of availability of the output data generated by the artificial neural network 125 to the processor (e.g., 133) so that the processor (e.g., 133) can request the data storage device 101 to transmit the output data.

[0181] Optionally, the controller 151 is configured to provide the output data to the processor in parallel with storing the output data into the output partition. For example, the controller 151 can be configured to automatically discard the output data computed for a previous segment of the sensor data stream if the processor (e.g., 133) does not request transmission of the output data to the processor (e.g., 133) within a predetermined time period or before a next version of the output data is available. Optionally, after reporting availability of the output data to the processor (e.g., 133), the controller 151 can be configured to selectively discard the output data computed for the previous segment of the sensor data stream based on a response of the processor (e.g., 133) to the output data transmitted to the processor (e.g., 133). For example, in some cases, the processor (e.g., 133) can request transmission of the output data to the processor (e.g., 133) without saving the output data into the output partition (e.g., 187); and in other cases, the processor (e.g., 133) can request transmission of the output data to the processor (e.g., 133) and storing the output data into the output partition (e.g., 187).

[0182] Optionally, the output data from the artificial neural network 125 can also be stored in a looped fashion into the output partition (e.g., for a segment of output data over a time period selected by the processor (e.g., 133)).

[0183] For example, the external input to the artificial neural network 125 can be continuously logged in the input namespace 185 over a previous time period T1. When the sensor data is sampled at a predetermined time interval T2, the input namespace 185 can hold the most recent T1 / T2 input data sets. In contrast, the external output from the artificial neural network 125 can be selectively logged into the output namespace 187 (e.g., once every predetermined time period T3, where T3 is a multiple of T2). The output data can be logged into the output namespace 187 at a lower frequency; and the output namespace 187 can be allocated to store a predetermined number of output data sets (e.g., the last output data set is maintained via continuous writing and writing in a looped fashion).

[0184] At least some embodiments disclosed herein include a communication protocol / interface to allow the data storage device to perform neural network acceleration in run-time with reduced data traffic to the host processor (e.g., central processing unit (CPU)).

[0185] For example, the host processor (e.g., 133) of the vehicle 111 can provide a write command to the data storage device 101 to store an artificial neural network model model in the model partition (e.g., 183). As the neural network accelerator 159 is configured to apply the model, the data communication of the model’s data of the ANN 125 back to the processor can be reduced or eliminated.

[0186] To use the ANN model in classification and / or prediction, the host processor of the vehicle 111 (e.g., 133) can stream input data for the ANN 125 into the neuron input partition (e.g., 185). The neural network accelerator 159 of the data storage 101 can automatically apply the input data to the model stored in the ANN model partition (e.g., 183) according to the address mapping 191. The data storage 101 makes the output of the computation available for propagation in the ANN 125. Preferably, the output of the computation is made available to the neural network accelerator 159 through the buffer 152 without the need to store the intermediate output into the storage media components (e.g., 161-163). Thus, the data communications between the host processor (e.g., 133) and the data storage 101 for conveying the output of the neurons can be reduced. When the output has propagated to the output neurons in the ANN 125, the data storage 101 can provide a response to the write request associated with writing the input data set into the neuron input partition (e.g., 185). The response indicates that the external output from the neurons in the ANN 125 is available. In response, the host processor of the vehicle 111 (e.g., 133) can optionally issue a read command to retrieve the external output for further processing.

[0187] Figure 17 FIG. 1 shows a vehicle 111 in communication with a data storage 101 to implement neural network computations according to one embodiment. For example, as described above, the vehicle 111 can include a processor 133 that is configured to implement an ANN 125 for classification and / or prediction. The ANN 125 can be implemented in the data storage 101 as described above. The data storage 101 can include a neural network accelerator 159 that is configured to perform computations for the ANN 125. The data storage 101 can include a buffer 152 that is configured to store the output of the computations for the ANN 125. The data storage 101 can include an address mapping 191 that is configured to map the output of the computations to an output partition (e.g., 187) in the data storage 101. The data storage 101 can include a communication interface 195 that is configured to communicate with the processor 133 of the vehicle 111. The data storage 101 can include a power supply 197 that is configured to provide power to the data storage 101. Figure 17 Figure 7 Figure 1 The communication described in FIG. 1 can be implemented in the data storage 101 described in FIGS. 6-10.

[0188] In FIG. 1, the processor 133 can be configured with a simplified instruction set 301 to perform neural network computations because some of the computations involving the ANN 125 are performed by the neural network accelerator 159 within the data storage 101. Thus, it is not necessary to transfer the model data back to the processor 133 during prediction and / or classification using the ANN 125. Figure 17

[0189] The sensor 103 can produce a continuous stream of sensor data 121 based on the sampling rate of the data. The sensor data 121 can be sampled at fixed, predetermined time intervals (e.g., during operation of the vehicle 111). The processor 133 can execute the instructions 301 to convert the sensor data 121 into an input stream 303 for the input neurons in the ANN 125. The input neurons in the ANN 125 are configured to accept external input into the ANN 125; and the output neurons are configured to provide external output from the ANN 125.

[0190] ​​​Generally, the complete set of inputs for ANN 125 at a time instant contains the inputs for the entire set of input neurons of ANN 125. Input stream 303 contains a sequence of sets of inputs for a sequence of time instants spaced apart from each other according to a fixed, predetermined time interval.

[0191] Data storage 101 stores input stream 303 into neuron input namespace 185 in a circular fashion, where the oldest set of inputs corresponding to the earliest time instant of a data sample of a set of data currently stored in neuron input namespace 185 is erased to store the most recent set of inputs in input stream 303.

[0192] For each set of input data, neural network accelerator 159 applies the model of ANN 125 stored in ANN model namespace 183. Neural network accelerator 159 (or processor 133) can control the propagation of signals within the neural network. When an output neuron of ANN 125 generates its output in response to a set of input data, data storage 101 can provide an indication to processor 133 that the neuron output is ready for retrieval. The indication can be configured in a response to a request from processor 133 to write a set of input data into neuron input namespace 185. Processor 133 can optionally retrieve output data 305 (e.g., according to conditions and / or criteria programmed in instructions).

[0193] In some embodiments, a trigger parameter is configured in data storage 101. When the output parameter in external output 317 complexly triggers the requirement specified by the trigger parameter, data storage provides a response to a request from processor 133 to write a set of input data into neuron input namespace 185.

[0194] Figure 18 Figures 1-10 illustrate a communication with a data storage to implement a neural network computation according to one embodiment. For example, Figure 18 The communication can be implemented in conjunction with Figure 17 The communication can be implemented in conjunction with Figure 7 data storage 101 illustrated in Figures 1-10.

[0195] In Figure 18 model namespace 183 stores a model 313 of the entire ANN 125. In response to receiving an external input set 315 for a time instant from input stream 303 in buffer 152, data storage 101 can write external input 315 into input namespace 185 in parallel with retrieving neuron model 312 containing the portion of ANN model 313 corresponding to the parameters of the input neurons and / or the identities of the neurons connected to the input neurons. Buffer 152 allows neural network accelerator 159 to combine neuron model 312 and external input 325 to generate output 327 of the input neurons.

[0196] Generally, neuron output 327 may include a portion of internal output 316 for further propagation within ANN 125 and / or a portion of external output 317 for processor 133.

[0197] The internal output 316 is stored as internal input 316 in buffer 152 for further propagation in ANN 125 in a manner similar to generating neuron output 327 from external input 315. For example, a portion of the internal input 316 may cause controller 151 and / or neural network accelerator 159 to retrieve the corresponding neuron model 312 associated with the internal input, such that the internal input is applied in neural network accelerator 159 to the corresponding neuron model 312 to generate its neuron output 327.

[0198] When the complete external output set 317 is available in buffer 152, the external output 317 can be stored in output namespace 187.

[0199] Optionally, storage device 101 may not store each external output set 317 corresponding to a stored external input set 315 sampled at a given time. For example, storage device 101 may be configured to store one external output set 317 for every predetermined number of external input sets (e.g., 315). Alternatively or in combination, processor 133 may determine whether to store an external output 317. For example, storage device 101 may be configured to store an external output 317 for further processing in response to processor 133 retrieving the external output 317. For example, storage device 101 may be configured to store an external output 317 in response to a write command from processor 133 after processing the external output 317 in processor 133.

[0200] Figure 19 A method for communicating with a data storage device to perform neural network computation is illustrated according to one embodiment. For example, Figure 19 The method can be used Figure 7 Or 10 data storage devices 101 and / or Figure 8 The storage media component 160 is implemented in Figure 1 Or in vehicle 111 of vehicle 2. For example, Figure 19 The method can be with Figure 6 , 9 Methods 14, 15 and / or 16 can be used in combination.

[0201] At box 341, one or more processors 133 of vehicle 111 store model data (e.g., 313) of artificial neural network (e.g., 125) into data storage device 101.

[0202] At block 343, the one or more processors 133 of the vehicle 111 receive a set of sensor data from at least one sensor 103 configured on the vehicle 111.

[0203] At block 345, the one or more processors 133 of the vehicle 111 generate an input set to an artificial neural network (e.g., 125) based on the sensor data.

[0204] At block 347, the one or more processors 133 of the vehicle 111 provide the input set to the data storage device 101. In response to the input set, the data storage device 101 is configured to generate an output set using model data 313 of the artificial neural network (e.g., 125).

[0205] At block 349, the one or more processors 133 of the vehicle 111 retrieve the output set from the data storage device 101.

[0206] For example, the data storage device 101 generates the output set using at least a portion of the model data 183 stored in the data storage device without transmitting the portion of the model data 183 to the one or more processors 133 between receiving the input set and completing the computation of the output set.

[0207] For example, the portion of the model data 183 can include static properties of neurons in the artificial neural network (e.g., 125) and / or neuron connectivity data of the artificial neural network (e.g., 125).

[0208] For example, to provide the input set to the data storage device 101, the one or more processors 133 of the vehicle 111 can transmit one or more write commands to the data storage device 101. The one or more write commands are configured to instruct the data storage device 101 to store the input set in the data storage device 101. After completing the computation of the output set in the data storage device 101, the controller 151 of the data storage device 101 can transmit a response to the one or more write commands to the one or more processors 133. The response can include an indication that the output set is available for retrieval by the one or more processors 133.

[0209] In response to the indication, the one or more processors 133 can optionally retrieve the output set from the data storage device 101 by transmitting a read command to the data storage device 101 to obtain the output set (e.g., after determining to retrieve the output set from the data storage device 101 for processing).

[0210] Alternatively or in combination, the one or more processors 133 of the vehicle 111 can determine whether to store the output set in the non-volatile memory of the data storage device. In response to determining to store the output set in the non-volatile memory of the data storage device 101, the one or more processors 133 of the vehicle 111 can transmit a write command to the data storage device 101.

[0211] Because the output set is initially generated in the data storage device 101 and then buffered in the buffer 152 (e.g., volatile memory), the data storage device 101 can execute the write command to store the output set into the output namespace 187 without transmitting the output set to and / or receiving the output set from the one or more processors 133 in response to the write command.

[0212] For example, after receiving another set of sensor data 121 from at least one sensor 103 configured on the vehicle 111, the one or more processors 133 of the vehicle 111 generate another input set to the artificial neural network 125 based on the other set of sensor data.

[0213] The one or more processors 133 transmit another command to write another input set into the data storage device 101; and the data storage device 101 generates another output set using the model data 183 of the artificial neural network 125 and the other input set. After receiving a response to the other command to write another input set, the one or more processors 133 can determine to skip processing of the other output set and transmit a continuous write command to store the other output set to the data storage device 101. In response, the data storage device 101 can write the other output set buffered within the data storage device 101 into the output namespace 187 without transmitting the other output set from the one or more processors 133 of the vehicle 111 to the data storage device 101 and / or without transmitting the other output set from the data storage device 101 to the one or more processors 133 of the vehicle 111.

[0214] At least some of the neural network techniques discussed above can be used to optimize bandwidth allocation for different types of operations when accessing storage media in a data storage device.

[0215] For example, a data storage device can have a queue of operations to be performed and / or executed. The operations can be in accordance with commands received via a host interface, or in conjunction with background processes, such as wear leveling and garbage collection.

[0216] The operations can involve accessing storage media in one or more storage media components to store data or retrieve data. For example, a write operation involves transmitting data to a storage media component; and a read operation involves receiving data from a storage media component.

[0217] In some cases, the bandwidth to transfer data to and from a storage medium can limit the performance of a data storage device. A data storage device can have different ways of allocating communication bandwidth for different types of operations.

[0218] For example, when a read command is received in a data storage device, the response time between the receipt of the command and the provision of data requested by the read command represents the performance of the data storage device.

[0219] For example, when a write command is received in a data storage device, the data to be written via the write command can be buffered by the data storage device. The data storage device can schedule the execution of the write command when the execution of the write command does not delay the execution of a read command.

[0220] However, buffering write commands can reduce the buffer capacity for incoming read commands and / or data to be transmitted via read commands. Thus, different ways of prioritizing the execution of read commands and write commands can result in different levels of performance of the service provided by the data storage device.

[0221] Furthermore, different applications accessing the data storage device can have different requirements / preferences for the performance of the service of the data storage device.

[0222] An artificial neural network (ANN) (e.g., spiking neural network, convolutional neural network, recurrent neural network) can be configured to predict the identification of an optimized bandwidth scheme to improve the overall performance of the data storage device. The prediction can be based on data access patterns indicated by various factors, such as the operational state of the vehicle, the processing state / mileage / workload of the advanced driver assistance system (ADAS) of the vehicle, the activity of applications with data access requests, the type of access requests (e.g., read / write), the frequency of access requests, the address orientation of access requests, the chunk size of data involved in access requests, etc.

[0223] For example, an artificial neural network (ANN) can be trained to predict an optimized bandwidth scheme for the overall system performance for a current data access pattern.

[0224] Optimization for various scenarios can be found via self-organizing using unsupervised learning techniques, or predictive capacity is achieved via training using supervised learning techniques.

[0225] For example, an artificial neural network (ANN) (e.g., a spiking neural network, a convolutional neural network, a recurrent neural network) can be configured to predict parameters of a bandwidth allocation scheme such that when bandwidth is provided for different types of operations (e.g., reads, writes, wear leveling, garbage collection) to communicate with a storage media according to the scheme, performance of the data storage device is optimized. For example, performance of the data storage device can be an average response time to read commands received in the data storage device over a period of time. For example, the prediction of the artificial neural network (ANN) can be based on parameters indicative of data access patterns, such as operating state of the vehicle, processing state / mileage / workload of the vehicle’s advanced driver assistance system (ADAS), activity of applications with data access requests, type of access requests (e.g., read / write), frequency of access requests, address locality of access requests, chunk size of data involved in access requests, etc.

[0226] An artificial neural network (ANN) can be self-trained within a data storage device for determining an optimized bandwidth allocation scheme for optimizing performance of the data storage device. For example, unsupervised machine learning techniques can be used to train the artificial neural network (ANN) to self-organize input data to acquire prediction / classification capabilities.

[0227] Alternatively or in combination, supervised machine learning techniques can be used to train the artificial neural network (ANN) to optimize or establish a predictive model of an optimized bandwidth allocation scheme and its parameters.

[0228] The artificial neural network (ANN) can receive inputs of predictive parameters of data access patterns resulting from implementing bandwidth allocation schemes and measurements of performance of the data storage device. The artificial neural network (ANN) can self-organize to find optimization of bandwidth allocation under various scenarios using unsupervised learning techniques, or be trained to produce an optimized bandwidth allocation scheme using supervised learning techniques.

[0229] For example, during a training period, the data storage device can vary bandwidth allocation schemes and track / measure performance of the data storage device operating using the bandwidth allocation schemes for data access patterns. A training data set with different bandwidth allocation schemes and resulting performance levels of the data storage device implementing the respective bandwidth allocations can be used to train the artificial neural network (ANN) to predict a search for and / or determination of a bandwidth allocation scheme that optimizes performance level of the data storage device for current data access patterns.

[0230] For example, the data access patterns of a data storage device configured in a vehicle can be associated with operating parameters such as: the vehicle's current operating parameters, the current operating parameters of the application actively using the data storage device (e.g., a vehicle component or electronic control unit), and / or the current operating parameters of the data storage device. These operating parameters can be provided as input to an artificial neural network (ANN) to indicate the data usage patterns.

[0231] When a bandwidth allocation scheme is implemented in a data storage device, the performance level of the data storage device can be measured. For example, the performance level indicator of the data storage device may include the average response time to read commands over a predetermined time period, the average time that incoming commands are blocked during periods of insufficient buffer space, etc.

[0232] Unsupervised machine learning techniques can be used to train artificial neural networks (ANNs) using data collected during the training period to correlate data access patterns, bandwidth allocation schemes, and performance levels (e.g., via clustering, classification, etc.). This allows for the determination of preferred bandwidth allocation schemes for current data access patterns that can optimize the performance level of data storage devices.

[0233] Optionally or in combination, optimized bandwidth allocation schemes determined for various data access patterns can be used to train artificial neural networks (ANNs) using supervised machine learning techniques to minimize the difference between the bandwidth allocation schemes predicted by the ANNs and the optimized caching schemes determined for various data access patterns.

[0234] Therefore, when a data storage device operates in a data access mode represented by the current operating parameters, an artificial neural network (ANN) can be used to determine a bandwidth allocation scheme that optimizes the performance of the data storage device.

[0235] When measuring the performance of a data storage device using individual parameters (e.g., read response time, commands blocked by insufficient buffer space, etc.), a target performance indicator can be constructed based on a combination of these individual parameters. The target performance indicator can be configured to increase as any one performance parameter increases while the others remain constant, and decrease as any one performance parameter decreases while the others remain constant. Therefore, an artificial neural network (ANN) can be configured to predict bandwidth allocation optimization based on the target performance indicator.

[0236] For example, an artificial neural network (ANN) can be configured to predict target performance indicators of a data storage device when the device implements a selected bandwidth allocation scheme (and its associated parameters) for a specific data access pattern. The bandwidth application scheme predictions made by the ANN can be used to search for optimized bandwidth allocation schemes.

[0237] Optionally, when an artificial neural network (ANN) classifies a current data access pattern as a new pattern, the data storage device can attempt to change the bandwidth allocation scheme to generate training data for the new pattern to determine an optimal bandwidth allocation scheme for the new pattern.

[0238] Figure 20 A data storage device configured to optimize bandwidth allocation for different types of operations that access storage media is shown in accordance with one embodiment. For example, Figure 20 The data storage device of Figure 7 or 10 can be implemented in Figure 1 or 2.

[0239] In Figure 20 , the data storage device 101 includes an address mapping 191 that specifies a mapping between logical memory addresses used in the input stream 303 to identify storage locations and physical memory addresses of memory cells in the storage media components 161-163 in which the data of the logical memory addresses are physically stored. The logical memory addresses can be defined in a namespace (e.g., 183, 185,... 187) that is defined in the storage capacity provided by the storage media components 161-163 of the data storage device 101.

[0240] Because the same set of logical addresses in the input stream 303 can be mapped to the storage media components 161-163 in different ways, the same data access pattern in the logical address space can have different access patterns in the storage media components 161-163.

[0241] The controller 151 and / or the neural network accelerator 159 can have different operations that require access to the storage media in the components 161-163. The communication bandwidth for the media access communications 351 to and from the storage media components 161-163 can be a resource that limits performance.

[0242] An artificial neural network 125 (e.g., stored as model data 313 in the data storage device 101) can be used to make predictions to determine a bandwidth allocation scheme 355 that optimizes performance of the data storage device 101 as a whole.

[0243] In general, the communication bandwidth is available for the media access communications 351 of different operations for different data access patterns. The data access patterns can be recognized from operation parameters 353 generated by the controller 151 and / or received in the input stream 303.

[0244] Examples of the operating parameters 353 can include operating parameters of the vehicle 111 in which the data storage device 101 is installed. For example, the operating parameters of the vehicle 111 can include a speed of the vehicle 111, an orientation of the vehicle 111, a roadway on which the vehicle 111 is traveling, inputs from some of the sensors 103 of the vehicle 111, states of vehicle controls (e.g., 141, 143, 145), states of components of the vehicle 111 (e.g., infotainment system 149 and / or the communication device 139 of the vehicle 111). The operating parameters of the vehicle 111 can include operating states of an advanced driver assistance system (ADAS) 105 and / or other applications / components running in the vehicle 111.

[0245] Examples of the operating parameters 353 can include operating parameters of the data storage device 101, such as states and operations of the neural network accelerator 159, commands queued for execution in the data storage device 101, states of background operations to be executed and / or being executed in the data storage device 101, etc.

[0246] The neural network accelerator 159 can train the artificial neural network 125 using the operating parameters 353 (e.g., the model data 313 in the model namespace 183) stored in the data storage device 101. The training allows the artificial neural network 125 to recognize activity clusters corresponding to different data access patterns.

[0247] During a training period, the controller 151 can generate training data by trying different bandwidth allocation schemes 355 for controlling the bandwidth allocation of the media access communications 351 for different types of operations performed by the controller 151 and the neural network accelerator 159. The controller 151 measures the performance of the data storage device 101 under the respective bandwidth allocation schemes 355. The training data can include the operating parameters 353 indicative of the data access patterns, the different bandwidth allocation schemes 355 implemented in the data storage device 101, and the performance indicators 357 of the respective bandwidth allocation schemes 355.

[0248] The neural network accelerator 159 can train the artificial neural network 125 to recognize the data access patterns using the training data, and / or to predict the performance indicators 357 for various bandwidth allocation schemes 355.

[0249] Based on the predictions made by the neural network accelerator 159, the neural network accelerator 159 and / or the controller 151 can determine an optimized bandwidth allocation scheme 355 to control the priorities of different types of operations in the data storage device 101 when accessing the storage media components 161-163.

[0250] Optionally or alternatively, when the data storage device 111 is operating in a data access pattern recognized by the artificial neural network 125 based on the operating parameters 353, the controller 151 can vary the bandwidth allocation scheme 355 to search for a bandwidth allocation scheme 355 that optimizes the performance indicator 357.

[0251] The optimized bandwidth allocation scheme 355 set can be used to further train the neural network accelerator 159 to predict optimized bandwidth allocation schemes 355 for corresponding data access patterns.

[0252] Figure 21 A method of bandwidth optimization for different types of operations that access storage media in a data storage device is shown according to one embodiment. For example, Figure 21 The method of 1 can be used with Figure 7 , 10 the data storage device 101 of 20 and / or Figure 8 the storage media components 160 of 20. For example, Figure 1 The method of 1 can be used in combination with the methods of Figure 21 , Figure 6 , 9 14, 15, 16, and / or 19.

[0253] At block 361, the data storage device 101 stores data to and retrieves data from one or more storage media components of the data storage device 101 according to commands in the input stream 303 received in the data storage device 101.

[0254] At block 363, the controller 151 of the data storage device 101 provides as input to the artificial neural network 125 operating parameters 353 indicative of a time-varying data access pattern in the data storage device 101.

[0255] For example, the data storage device 101 can be configured in a vehicle 111 to support operation of an advanced driver assistance system (ADAS) 105 and / or an infotainment system 149. The operating parameters 353 can include operating parameters of the vehicle 111.

[0256] For example, the operational parameters 353 of the vehicle 111 can include a speed of the vehicle 111, an orientation of the vehicle 111, input from sensors 103 configured on the vehicle, a state of a vehicle control (e.g., 141, 143, or 145), a state of an infotainment system 149 of the vehicle 111, or a state of an advanced driver assistance system (ADAS) 105 of the vehicle 111, or any combination thereof.

[0257] The operational parameters 353 provided as input to the artificial neural network 125 can include operational parameters of the data processing system 101, such as an operational state of the neural network accelerator 159, a state of pending commands to be executed by the controller 151, or a state of ongoing operations in the storage media components 161-163, or any combination thereof.

[0258] For example, the artificial neural network 125 can include a spiking neural network trained in the data storage device 101 using a training data set generated in the data storage device.

[0259] At block 365, the artificial neural network 125 generates a prediction based on the input to determine an optimized bandwidth allocation scheme 355 for controlling access to one or more storage media components 161-163 in the data storage device 101 by different types of operations.

[0260] For example, the different types of operations can include operations in response to read commands and operations in response to write commands.

[0261] Optionally, the different types of operations can include operations generated by a background process in the data storage device. For example, the background process can have a wear leveling operation that relocates data from a low-wear memory region to a high-wear memory region. For example, the background process can have a garbage collection operation that relocates data from a block of memory cells that has portions with both valid and invalid data to allow the block to be erased. For example, the background process can have retrieving model data 313 of the artificial neural network 125 to apply input to the neural network accelerator of the artificial neural network 125.

[0262] In general, the prediction generated 367 by the artificial neural network 125 to determine the optimized bandwidth allocation scheme 355 can include an identification of a data access pattern, a performance indicator 357 for a bandwidth allocation scheme 355 for the data access pattern identified via the operational parameters, or the optimized bandwidth allocation scheme 355, or any combination thereof.

[0263] For example, the data storage device 101 can include a neural network accelerator 159 configured to generate a prediction using model data 313 of the artificial neural network 125 stored in the data storage device 101.

[0264] Optionally, the neural network accelerator 159 is further configured to train the artificial neural network 125 using training data generated in the data storage device 101 during a training period. For example, the training data can include operational parameters 353 identified in the training period, bandwidth allocation schemes 355 used in the training period, and performance indicators 357 measured during the training period for the bandwidth allocation schemes 355.

[0265] Optionally, the neural network accelerator 159 is further configured to train the artificial neural network 125 using training data generated for data access patterns. For example, the training data can include operational parameters 353 classified by the artificial neural network 125 as having a data access pattern in a time period; and the training data can additionally include optimized bandwidth allocation schemes 355 determined for the data access pattern recognized from the operational parameters 353 in the time period.

[0266] Optionally, the neural network accelerator 159 is configured to determine an optimized bandwidth allocation scheme 355 for training data based on a search facilitated by the artificial neural network 125 for different bandwidth allocation schemes 355 based on predictions of performance indicators 357 generated by the artificial neural network 125.

[0267] Optionally or in combination, the controller 151 is configured to determine an optimized bandwidth allocation scheme 355 for training data based on a search facilitated by measuring performance levels 357 of different bandwidth allocation schemes 355 used to control priorities of media access communications 351.

[0268] At block 367, the controller 151 of the data storage device 101 schedules different types of operations accessing the one or more storage media components 161-163 in accordance with an optimized bandwidth allocation scheme 355 determined based on predictions generated 367 by the artificial neural network 125.

[0269] The server 119, the computer system 131, and / or the data storage device 101 can each be implemented as one or more data processing systems.

[0270] The present disclosure includes methods and apparatuses that perform the methods described above, including data processing systems that perform these methods, and computer- readable media containing instructions that, when executed on data processing systems, cause the systems to perform these methods.

[0271] A typical data processing system can include interconnects (e.g., buses and system core logic) that interconnect microprocessors and memory. Microprocessors are typically coupled to cache memory.

[0272] The interconnect interconnects the microprocessor and the memory together and also interconnects them, via an I / O controller, to input / output (I / O) devices. The I / O devices can include a display device and / or peripheral devices, such as a mouse, a keyboard, a modem, a network interface, a printer, a scanner, a video camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices, such as a printer, a scanner, a mouse, and / or a keyboard, are optional.

[0273] The interconnect can include one or more buses that connect the various bridges, controllers and / or adapters to each other. In one embodiment, the I / O controller includes a USB adapter for controlling USB peripheral devices, and / or an IEEE- 1394 bus adapter for controlling IEEE- 1394 peripheral devices.

[0274] The memory can include one or more of read-only memory (ROM), volatile random access memory (RAM), and non-volatile memory such as a hard disk drive, a flash memory, or the like.

[0275] The volatile RAM typically is implemented as dynamic RAM (DRAM) which requires periodic refreshes to maintain the data in the memory. The non-volatile memory can be implemented as magnetic hard disk storage, magnetic tape, optical disk storage, or other types of memory that maintain data even after power is removed from the system.

[0276] The non-volatile memory can be a local device coupled directly to the rest of the components in the data processing system. A non-volatile memory in the system also can be implemented as a removable memory that is located, for example, in a drive bay of the data processing system. The removable memory can be implemented as a magnetic hard disk, magnetic tape, optical disk, or other type of memory that maintains data even after power is removed from the system.

[0277] In this disclosure, some functions and operations are described as being performed by or caused to be performed by a software code to simplify description. However, this expression is also used to designate that the functions are produced by execution of the code / instructions by a processor, such as a microprocessor.

[0278] Alternatively, or in combination, the functions and operations described herein can be implemented using special purpose circuitry having or not having software instructions, such as using an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Embodiments can be implemented using hard-wired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source for the instructions executed by the data processing system.

[0279] While one embodiment can be implemented in a fully functioning computer system and computer system, various embodiments can be distributed over a variety of forms and can be adapted to operate with other machines or computer readable media.

[0280] At least some aspects disclosed can be embodied, at least in part, in software. That is, the techniques can be performed in a computer system or other data processing system with a processor, such as a microprocessor, which performs a sequence of instructions contained in memory (e.g., ROM, volatile RAM, non-volatile memory, cache or a remote storage device).

[0281] Routines executed to implement an embodiment can be implemented as a part of an operating system or a specific application, component, program, object, module or sequence of instructions referred to as "computer programs." The computer programs typically comprise one or more instructions set at various times in various memory and storage devices in a computer, and that, when read and executed by one or more processors in a computer, cause the computer to perform operations necessary to execute elements involving the various aspects.

[0282] Machine readable media can be used to store software and data which when executed by a data processing system causes the system to perform various methods. The executable software and data can be stored in various places including for example ROM, volatile RAM, non-volatile memory and / or cache. Portions of this software and / or data can be stored in any one of these storage devices. Further, the data and instructions can be obtained from centralized servers or peer to peer networks. Different portions of the data and instructions can be obtained from different centralized servers and / or peer to peer networks at different times and in different communication sessions. The data and instructions can be obtained in entirety prior to the execution of the applications. Alternatively, portions of the data and instructions can be obtained dynamically, just in time, when needed, and / or in the middle of an execution of the applications. Thus, it should be appreciated that the data and instructions described herein can be expected to be obtained from a storage device, from a

[0283] Examples of computer readable media include but are not limited to non-transitory, recordable and non-recordable type media such as volatile and non-volatile memory devices, read only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, disk storage media, optical storage media (e.g., Compact Disk Read Only Memory (CD ROM), Digital Versatile Disk (DVD), etc.), among others. The computer readable media can store the instructions.

[0284] The instructions can also be embodied in digital and analog communication links, for example, carrier waves, infrared signals, digital signals, etc. However, the carrier waves, infrared signals, digital signals, etc. are non-tangible media that cannot be felt or touched and are not a tangible machine readable medium that can be configured to store instructions.

[0285] Generally, a machine-readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, manufacturing tool, any device with a set of one or more processors, etc.).

[0286] In various embodiments, hardwired circuitry can be used in combination with software instructions to implement the techniques. Thus, the techniques are neither limited to hardware circuitry nor software, nor to any particular source of the instructions executed to implement the techniques. Software refers to single or multi- threaded program code that is meant for execution on a processing device or devices, such as a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors, etc.

[0287] The above description and drawings are illustrative, and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in certain instances, well known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure are not necessarily references to the same embodiment; and, such references mean at least one.

[0288] In the foregoing specification, the disclosure has been described with reference to specific exemplary embodiments thereof. It will be evident that various modifications can be made to the specification without departing from the broader spirit and scope of the disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

1. A data storage device comprising: one or more storage media components; a controller configured to store data into and retrieve data from the one or more storage media components in accordance with commands received in the data storage device; a neural network accelerator; a model of an artificial neural network stored in a first namespace of the data storage device; wherein the controller is configured to receive, in response to a second namespace in the data storage device, time-varying operating parameters indicative of a data access pattern as input, enabling the neural network accelerator to run the model of the artificial neural network stored in the first namespace to generate a prediction based on the input to determine an optimized bandwidth allocation scheme for controlling access to the one or more storage media components by different types of operations in the data storage device; wherein the controller is configured to schedule the different types of the operations accessing the one or more storage media components in accordance with the optimized bandwidth allocation scheme.

2. The data storage device of claim 1, wherein the artificial neural network comprises a spiking neural network.

3. The data storage device of claim 2, wherein the different types of the operations comprise operations in response to read commands and operations in response to write commands.

4. The data storage device of claim 3, wherein the different types of the operations comprise operations generated by a background process in the data storage device.

5. The data storage device of claim 2, wherein the prediction comprises a performance indicator of the data storage device when the data storage device is controlled by a selected bandwidth allocation scheme.

6. The data storage device of claim 2, wherein the prediction comprises the optimized bandwidth allocation scheme.

7. The data storage device of claim 2, wherein the data storage device is configured in a vehicle; and the operating parameters comprise operating parameters of the vehicle.

8. The data storage device of claim 7, wherein the operating parameters of the vehicle comprise a speed of the vehicle, an orientation of the vehicle, input from a sensor configured on the vehicle, a state of a vehicle control, a state of an infotainment system of the vehicle, or a state of an advanced driver assistance system of the vehicle, or any combination thereof.

9. The data storage device of claim 8, further comprising: a neural network accelerator configured to generate the prediction using model data of the artificial neural network stored in the data storage device.

10. The data storage device of claim 9, wherein the neural network accelerator is further configured to train the artificial neural network using training data generated in a training period.

11. The data storage device of claim 10, wherein the training data includes the operational parameters in the training period, bandwidth allocation schemes implemented in the training period, and performance indicators for the bandwidth allocation schemes measured during the training period.

12. The data storage device of claim 9, wherein the neural network accelerator is further configured to train the artificial neural network using training data; and wherein the training data includes the operational parameters in a time period, and an optimized bandwidth allocation scheme determined for data access patterns recognized from the operational parameters in the time period and optimized for performance of the data storage device.

13. The data storage device of claim 12, wherein the neural network accelerator is configured to determine the optimized bandwidth allocation scheme based on predictions of performance indicators for different bandwidth allocation schemes.

14. The data storage device of claim 12, wherein the controller is configured to determine the optimized bandwidth allocation scheme based on measuring performance levels of different bandwidth allocation schemes used to control scheduling of the different types of operations accessing the one or more storage media components.

15. A method of operating a data storage device, comprising: storing data into and retrieving data from one or more storage media components of the data storage device in accordance with commands received in the data storage device; storing a model of an artificial neural network in a first namespace of the data storage device; receiving operational parameters indicative of data access patterns in the data storage device over time in a second namespace as inputs to the artificial neural network; enabling a neural network accelerator to run the model of the artificial neural network stored in the first namespace to generate predictions based on the inputs to determine an optimized bandwidth allocation scheme for controlling access to the one or more storage media components by different types of operations in the data storage device; and scheduling, by a controller of the data storage device, the different types of operations accessing the one or more storage media components in accordance with the optimized bandwidth allocation scheme.

16. The method of claim 15, wherein the artificial neural network includes a spiking neural network; and the method further comprising: training, in the data storage device, the artificial neural network using a training data set generated in the data storage device.

17. The method of claim 16, wherein the training data set includes an optimized bandwidth allocation scheme determined based on predictions of performance indicators for different bandwidth allocation schemes by the artificial neural network. ​ 18. The method of claim 16, wherein the training data set includes optimized bandwidth allocation schemes measured for different bandwidth allocation schemes used to control scheduling of the different types of operations to access the one or more storage media components.

19. A vehicle comprising: a computer system configured to generate time-varying operating parameters of the vehicle; and a data storage device configured to identify operating parameters of the data storage device, wherein the operating parameters of the vehicle and the operating parameters of the data storage device are indicative of data access patterns in the data storage device; a neural network accelerator; wherein the data storage device is configured to store a model of an artificial neural network in a first namespace of the data storage device; to receive, in response to a second namespace in the data storage device, the operating parameters of the vehicle and the operating parameters of the data storage device as inputs to the artificial neural network, so that the neural network accelerator can run the model of the artificial neural network stored in the first namespace to generate a prediction to determine an optimized bandwidth allocation scheme for controlling access to one or more storage media components by different types of operations in the data storage device; and wherein the data storage device is configured to schedule the different types of operations to access the one or more storage media components according to the optimized bandwidth allocation scheme.

20. The vehicle of claim 19, wherein the artificial neural network includes a spiking neural network; and the data storage device is further configured to generate training data and train the spiking neural network to generate the prediction.

Images