System, method and apparatus for communicating with data storage device in neural network computing
By storing artificial neural network model data in the data storage device of a mobile vehicle, using sensor data for predictive maintenance, identifying components that need maintenance, solving the problem of difficult component failure prediction in the prior art, and achieving safety and convenience Improvement of sex.
Patent Information
- Application Number
- CN202010805782.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-08-12
- Filing Date
- 2020-08-12
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-08-12
AI Technical Summary
The prior art is difficult to predict component failure or failure during operation of a motorized vehicle, resulting in safety hazards and inconvenient maintenance service arrangements.
Data storage devices are used to store artificial neural network model data, receive data through sensors and use artificial neural networks for predictive maintenance, identify the probability that components need to be maintained or replaced, and arrange services at convenient time.
Reduces the possibility of components failure or failure during operation of vehicles on the road, allows maintenance services to be scheduled at convenient times, and improves the reliability and safety of vehicles.
Smart Images

Figure CN112396157B_ABST
Abstract
Description
Technical Field
[0001] At least some embodiments disclosed herein relate generally to maintenance services for vehicles, and more particularly, but not limited to, communications with data storage devices for neural network computations in predictive maintenance of motor vehicles. Background Art
[0002] Conventionally, automobile maintenance is scheduled based on predetermined operational milestones. For example, a routine maintenance service may be scheduled every three or six months or after a predetermined driving distance (e.g., 3,000 miles, 6,000 miles, or 15,000 miles).
[0003] When a component of a motor vehicle breaks down or malfunctions during operation of the vehicle, this traffic event may be a safety hazard. After this traffic event occurs, travel for the vehicle's services will soon be scheduled at an inconvenient time.
[0004] Recent developments in the field of autonomous driving technology allow computing systems to operate control elements of a motor vehicle, at least under some conditions, without assistance from a human operator of the vehicle.
[0005] For example, sensors (e.g., cameras and radars) may be mounted on a motor vehicle to detect conditions surrounding the vehicle as it travels on a road. A computing system mounted on the vehicle analyzes the sensor inputs to identify the conditions and generate control signals or commands for autonomous adjustment of the vehicle's direction and / or speed, with or without any input from a human operator of the vehicle.
[0006] In some arrangements, when the computing system identifies a situation in which the computing system may not be able to continue operating the vehicle in a safe manner, the computing system alerts the human operator of the vehicle and requests the human operator to take over control of the vehicle and drive manually, rather than allowing the computing system to autonomously drive the vehicle.
[0007] U.S. Patent No. 9,533,579, entitled “Electronic Control Apparatus for Electrically-Driven Vehicle” and issued on January 3, 2017, discloses an electronic control apparatus for a vehicle having a self-diagnostic function.
[0008] Autonomous driving and / or advanced driver assistance systems (ADAS) often involve artificial neural networks (ANNs) for recognizing events and / or objects captured in sensor inputs.
[0009] Generally speaking, an artificial neural network (ANN) uses a network of neurons to process inputs to the network and produce outputs from the network.
[0010] For example, each neuron in the network receives a set of inputs. Some of the inputs to the neurons may be outputs of certain neurons in the network; and some of the inputs to the neurons may be inputs provided to the neural network. The input / output relationships among neurons in the network represent the neuron connectivity in the network.
[0011] For example, each neuron may have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function may be in the form of a step function, a linear function, a logarithmic sigmoid function, etc. Different neurons in the network may have different activation functions.
[0012] For example, each neuron may produce a weighted sum of its inputs and its bias, and then produce an output that is a function of the weighted sum computed using the neuron's activation function.
[0013] The relationship between the input and output of an ANN is typically defined by an ANN model, which includes data representing the connectivity of neurons in the network as well as biases, activation functions, and synaptic weights of each neuron. Using a given ANN model, a computing device calculates the output of the network from a given set of inputs to the network.
[0014] For example, input to the ANN network may be generated based on camera input; and output from the ANN network may be recognition of items, such as events or objects.
[0015] Spiking neural networks (SNNs) are a type of ANN that closely mimics natural neural networks. When the activation level of a neuron is high enough, the SNN neuron generates a spike as an output. The activation level of an SNN neuron mimics the membrane potential of a natural neuron. The output / spike of an SNN neuron can change the activation level of other neurons that receive the output. In general, the current activation level of an SNN neuron that changes over time is modeled using differential equations and is considered the state of the SNN neuron. Incoming spikes from other neurons can push the activation level of the neuron higher to reach a threshold for spikes. Once a neuron reaches a spike, its activation level is reset. Before the spike, the activation level of the SNN neuron decays over time, as controlled by a differential equation. The temporal element in the behavior of SNN neurons makes SNNs suitable for processing spatiotemporal data. The connectivity of SNNs is typically sparse, which has advantages in reducing computational workload.
[0016] In general, ANNs can be trained using supervised methods, where parameters in the ANN are adjusted to minimize or reduce the error between known outputs produced from corresponding inputs and the calculated outputs produced from applying the inputs to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.
[0017] Alternatively or in combination, the ANN may be trained using unsupervised methods, where the exact outputs produced from a given set of inputs are not known until training is complete. The ANN may be trained to classify items into multiple categories, or to classify data points into clusters.
[0018] Multiple training algorithms can be used for complex machine learning / training paradigms. Summary of the invention
[0019] In one aspect, the present invention relates to a method comprising: storing model data of an artificial neural network in a data storage device; receiving a set of sensor data from at least one sensor configured on a vehicle; generating a set of inputs to the artificial neural network based on the sensor data by one or more processors of the vehicle; providing the set of inputs to the data storage device, wherein the data storage device is configured to generate a set of outputs using the model data of the artificial neural network in response to the set of inputs; and retrieving the set of outputs from the data storage device by the one or more processors of the vehicle.
[0020] On the other hand, the present invention relates to a vehicle, comprising: at least one processor; at least one sensor configured to generate a sensor data stream; and a data storage device configured to store model data of an artificial neural network; wherein the at least one processor is configured to receive a set of sensor data from the at least one sensor, generate a set of inputs to the artificial neural network based on the sensor data, and request the data storage device to store the set of inputs; wherein the data storage device is configured to generate a set of outputs using the model data of the artificial neural network and the set of inputs received from the at least one processor; and wherein the at least one processor is configured to request the set of outputs from the data storage device.
[0021] On the other hand, the present invention relates to a data storage device of a vehicle, comprising: a host interface, which is configured to receive a set of inputs of an artificial neuron from at least one processor of the vehicle, the inputs being calculated based on sensor data generated by at least one sensor configured on the vehicle; a non-volatile memory, which is configured to store model data of an artificial neural network and to store the set of inputs; a neural network accelerator, which is configured to calculate a set of outputs using the model data of the artificial neural network stored in the non-volatile memory and the set of inputs received in the host interface; and a buffer for storing the set of outputs for access by the at least one processor via the host interface. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.
[0023] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.
[0024] Figure 2 An autonomous vehicle having a data storage device according to one embodiment is shown.
[0025] Figures 3 to 5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.
[0026] Figure 6 A method of predictive maintenance according to one embodiment is presented.
[0027] Figure 7 A data storage device for accelerating neural network computations according to one embodiment is shown.
[0028] Figure 8 A storage media component is shown for accelerating neural network computations according to one embodiment.
[0029] Fig. 9 A method for accelerating neural network computations in a motor vehicle is presented according to one embodiment.
[0030] Fig.10 A data storage device configured to support neural network computations is shown according to one embodiment.
[0031] Fig.11 The configuration of a namespace for an artificial neural network (ANN) model according to one embodiment is described.
[0032] Fig.12The configuration of a namespace for input to an artificial neuron according to one embodiment is described.
[0033] Fig.13 The configuration of a namespace for outputs from an artificial neuron according to one embodiment is described.
[0034] Figures 14 to 16 A method of predictive maintenance supported by model partitioning, input partitioning, and output partitioning according to one embodiment is presented.
[0035] Fig.17 Communication with a data storage device to implement neural network computations is shown according to one embodiment.
[0036] Fig.18 Communications within a data storage device to implement neural network computations are shown according to one embodiment.
[0037] Fig.19 A method of communicating with a data storage device for performing neural network computations is presented according to one embodiment. DETAILED DESCRIPTION
[0038] At least some embodiments disclosed herein provide systems, methods, and apparatus to process sensor data generated in a motor vehicle or another vehicle with or without an advanced driver assistance system (ADAS) to facilitate predictive maintenance.
[0039] Before a component of a motor vehicle breaks down or malfunctions during operation of the vehicle, it may be indicated whether the component needs replacement or maintenance. Such indications may not be obvious to a typical driver or passenger. However, sensor data may be collected and analyzed to predict the probability of component failure. This prediction may be used to schedule maintenance service, which may reduce or eliminate the likelihood of a traffic event in which a component of a vehicle breaks down or malfunctions during operation of the vehicle on the road. In addition, the prediction allows a service trip to be scheduled at a convenient time.
[0040] For example, sensors may be installed in an automotive system to collect data during its routine operation; and the sensor data may be used to predict whether and how soon a component will need to be replaced or maintained. The sensor data may be provided as input to an artificial neural network (ANN) (e.g., a spiking neural network (SNN)) of an artificial intelligence (AI) system to train itself (e.g., using unsupervised machine learning techniques) during a time period when normal operation of the vehicle is expected. The training is tailored to the specific operating environment of the driver, passenger, or user of the vehicle and the personalized operating habits of the vehicle occupant. Subsequently, the artificial neural network may detect abnormal conditions when the operating data deviates from the normal pattern. The AI system may be used to recommend maintenance services and / or identify components that may need to be replaced or maintained.
[0041] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.
[0042] Figure 1 The system includes a vehicle (111) having a data storage device (101). Optionally, the vehicle (111) has an advanced driver assistance system (ADAS) (105) and one or more sensors (103) that provide sensor data input to the ADAS (105) and / or the data storage device (101). The data storage device (101) is configured to use an artificial neural network (ANN) (125) to predict / identify the need for maintenance service based on data collected by the sensors (103). The ADAS (105) may be omitted without affecting the predictive maintenance feature. In some embodiments, at least a portion of the data generated by the sensors (103) is used in both the ADAS (105) for driver assistance and in the ANN (125) for maintenance prediction. Optionally, the output of the ANN (124) may be used in both the data storage device (101) and in the ADAS (105).
[0043] The sensors (103) may include digital cameras, lidars, radars, ultrasonic sonars, brake sensors, speed sensors, acceleration sensors, airbag sensors, GPS (Global Positioning System) receivers, audio sensors / microphones, vibration sensors, force / stress sensors, deformation sensors, motion sensors, temperature sensors, etc. Some of the sensors (103) may be configured primarily to monitor the environment of the vehicle (111); and other sensors (103) may be configured primarily to monitor the operating conditions of one or more components of the vehicle (111), such as the internal combustion engine, exhaust system, electric motor, brakes, tires, battery, etc.
[0044] The output of the time-varying sensor (103) is provided as a sensor data stream to an ADAS (105) and / or an ANN (125) to provide driver assistance (eg, autonomous driving) and maintenance prediction.
[0045] For example, the vehicle (111) may have a wireless communication device to communicate with a remote server (119) via a wireless signal (113) and a communication network (117). The remote server (119) is typically configured at a location remote from the highway (102) on which the vehicle (111) is serving. For example, the vehicle (111) may provide some sensor data (121) to the server (119) and receive updates of the ANN (125) from the server (119).
[0046] One example of a communication network (117) is a cellular telephone network having one or more base stations (e.g., 115) to receive wireless signals (e.g., 113). Another example of a communication network (117) is the Internet, where wireless local area network signals (e.g., 113) transmitted by the vehicle (111) are received at access points (e.g., 115) for further communication to a server (119). In some embodiments, the vehicle (111) uses a communication link (107) to a satellite (109) or a communication balloon to communicate with the server (119).
[0047] The server (119) may also communicate with one or more maintenance service facilities (e.g., 127) to receive maintenance service data (123) for a vehicle (e.g., 111). The maintenance service data (123) may include inspection records and / or service records of components of the vehicle (e.g., 111). For example, the inspection records and / or service records may indicate the degree of wear of components inspected during service at the maintenance service facility (e.g., 127), identification of failed or malfunctioning components, etc. The sensor data (121) and maintenance service data (123) of the vehicle (e.g., 111) in a time period before the service may be used to train the ANN (125) to predict the probability that the component requires maintenance service. The updated ANN (125) may be used to predict and recommend maintenance services for the vehicle (111) based on the sensor data (121) received in the most recent time period. Alternatively, the updated ANN (125) may be transmitted to the vehicle (111); and the vehicle (111) may use the data generated from the sensors (103) to predict and recommend maintenance services during routine operation of the vehicle (111).
[0048] The data storage device (101) of the vehicle (111) may be configured to record sensor data over a period of time that may be used in an ANN for predictive maintenance. Maintenance predictions typically last for a relatively long period of time (e.g., days, weeks, and / or months). In contrast, sensor data recorded for review of an accident, collision, or near-collision involving an autonomous vehicle typically lasts for a short period of time (e.g., 30 seconds to several minutes). Therefore, a typical black box data recorder configured to record sensor data for review / analysis of an accident or collision is not sufficient for predictive maintenance.
[0049] Optionally, the data storage device (101) stores sensor data for time periods leading up to a visit to a maintenance service facility (e.g., 127). The maintenance service facility (e.g., 127) can download the sensor data (121) from the data storage device (101) and provide the sensor data (121) and corresponding maintenance service data (123) to the server (119) to facilitate training of the ANN (125).
[0050] Optionally or in combination, the data storage device (101) is configured with a machine learning module to customize and / or train an ANN (125) installed in a vehicle (111) for predictive maintenance.
[0051] For example, the machine learning module of the data storage device (101) may be used to calibrate the ANN (125) to take into account the typical / daily environment in which the vehicle (111) operates and / or the preferences / habits of the driver of the vehicle (111).
[0052] For example, during a period of time when a vehicle is expected to operate with healthy components in a typical / daily environment, sensor data generated by sensors (103) can be used to train the ANN (125) to recognize sensor data patterns that represent trouble-free operation. Such patterns can vary for different vehicles (e.g., 111) based on their routine operating environment and the driving habits / characteristics of their drivers. Training allows the ANN (125) to detect deviations from the recognized normal patterns and report anomalies for maintenance prediction.
[0053] For example, the ANN (125) may include an SNN configured to classify time-based variations in sensor data and / or detect deviations from known patterns in sensor data of a vehicle (111) in normal / health conditions but operating in a personalized environment (e.g., a driver / passenger's daily route) and / or operating under personalized driving habits / patterns.
[0054] Figure 2 An autonomous vehicle (111) having a data storage device (101) is shown according to one embodiment. For example, Figure 2Autonomous vehicles (111) Figure 1 A vehicle (111) in a system.
[0055] Figure 2 The vehicle (111) is configured to have an advanced driver assistance system (ADAS) (105). The ADAS (105) of the vehicle (111) may have an artificial neural network (ANN) (125) for object detection, recognition, identification and / or classification. The ANN (125) and / or another neural network (e.g., configured in the data storage device (101)) may be used to predict the probability that a component of the vehicle (111) requires maintenance service (e.g., repair, replacement or adjustment).
[0056] Preferably, the data storage device (101) is configured to at least partially process the sensor data for predictive maintenance while reducing the computational burden of the processor (133) responsible for operating the ADAS (105) and / or other components, such as the infotainment system (149).
[0057] The vehicle (111) typically includes an infotainment system (149), a communication device (139), one or more sensors (103), and a computer system (131) connected to some controls of the vehicle (111), such as a steering control (141) for the direction of the vehicle (111), a brake control (143) for stopping the vehicle (111), an acceleration control (145) for the speed of the vehicle (111), etc. In some embodiments, Figure 1 The vehicle (111) in the system has a similar configuration and / or similar components.
[0058] Some of the sensors (103) are required for operation of the ADAS (105); and some of the sensors (103) are used to collect data related to the health of components of the vehicle (111), which may not be used in the ADAS (105). Optionally, the sensor data generated by the sensors (103) can also be used to predict the likelihood of an impending component failure. This prediction can be used in the ADAS (105) to take emergency action to put the vehicle in a safe state (e.g., by reducing speed and / or pulling into a parking space).
[0059] The computer system (131) of the vehicle (111) includes one or more processors (133), a data storage device (101), and a memory (135) storing firmware (or software) (147) including computer instructions and data models for an ADAS (105).
[0060] The one or more sensors (103) of the vehicle may include a visible light camera, an infrared camera, a lidar, a radar or sonar system, a peripheral sensor, a global positioning system (GPS) receiver, a satellite positioning system receiver, a brake sensor and / or an airbag sensor. In addition, the sensor (103) may include an audio sensor (e.g., a microphone) configured to monitor noise from various components and locations in the vehicle (111), a vibration sensor, a pressure sensor, a force sensor, a strain sensor and / or a deformation sensor configured to measure the load on the components of the vehicle (111), an accelerometer and / or a gyroscope sensor that measures the movement of some components of the vehicle (111), etc. Such sensors can be used to monitor the operating state and / or health of the components for predictive maintenance.
[0061] The sensor (103) may provide a real-time sensor data stream to the computer system (131). The sensor data generated by the sensor (103) of the vehicle (111) may include images of objects captured using a camera that uses light visible to the human eye for imaging, or using infrared light, or a sonar, radar, or LIDAR system for imaging. The image data obtained from at least one sensor of the vehicle is part of the collected sensor data for recording in the data storage device (101) and / or as an input (125) to the ANN. For example, the camera may be used to obtain road information for the vehicle (111) to travel, which may be processed by the ANN (125) to generate control signals for the vehicle (111). For example, the camera may be used to monitor the operating status / health of components of the vehicle (111), which may be processed by the ANN (125) to predict or schedule maintenance services.
[0062] Sensor data generated by sensors (103) of a vehicle (111) may include an audio stream that captures characteristics of sound at locations on the (111), such as locations near the engine, motor, drivetrain, wheels, doors, windows, etc. The audio data obtained from at least one sensor (103) of the vehicle (111) may be part of the collected sensor data for recording in a data storage device (101) and / or as input to an ANN (125). For example, the audio stream may be used to monitor the operating state / health of components of the vehicle (111) (e.g., internal combustion engine, exhaust system, motor, brakes), which may be processed by the ANN (125) to predict or schedule maintenance service.
[0063] The infotainment system (149) may be used to present a forecast or schedule maintenance service. Optionally, the communication device (139) may establish a connection to a mobile device of a driver of the vehicle (111) to notify the driver of recommended maintenance services and / or service recommendation data, to schedule an appointment, etc.
[0064] When the vehicle (111) is equipped with an ADAS (105), the output of the ADAS (105) can be used to control (e.g., (141), (143), (145)) the acceleration of the vehicle (111), the speed of the vehicle (111) and / or the direction of the vehicle (111) during autonomous driving.
[0065] Figures 3 to 5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.
[0066] exist Figure 3 In the invention, a supervised machine learning module (171) is used to train an artificial neural network (125) to minimize the difference between a service prediction (129) generated from sensor data (121) and maintenance service data (123).
[0067] For example, the maintenance service data (123) may identify measured wear of a component over time to predict a time to a recommended service. The sensor data (121) may be used in the ANN (125) to generate a predicted time to a recommended service. The supervised machine learning module (171) may adjust the artificial neural network (125) to reduce / minimize the difference between the time predicted based on the sensor data (121) and the time calculated from the measurement of wear.
[0068] For example, the maintenance service data (123) may identify a component that was replaced or repaired at the maintenance service facility (127). Sensor data (121) recorded during a period of time before the component was replaced or repaired may be used to calculate a time to replacement or repair. Additionally, segments of the sensor data stream during a period of time before the replacement or repair may be used in the ANN (125) to generate a prediction of the time to replacement or repair. Supervised learning (171) may be used to tune the ANN (125) to reduce the predicted time to replacement or repair and the actual time to replacement or repair.
[0069] Figure 2 Supervised learning (171) can be applied in the server (119) based on the sensor data of the vehicle population and its maintenance service data (123) to produce a general ANN for the vehicle population.
[0070] Figure 2The supervised learning (171) can be applied in the vehicle (111) based on the sensor data of the vehicle and its maintenance service data (123) to generate a customized / personalized ANN for a group of vehicles. For example, a general ANN can be initially used in the vehicle (111); the sensor data of the vehicle (111) and its maintenance service data (123) can be used to further train the vehicle's ANN (125) for customization / personalization of the ANN (125) in the vehicle (111).
[0071] exist Figure 4 In the embodiment of the present invention, a module (175) of unsupervised machine learning is used to train or improve an artificial neural network (125) to facilitate anomaly detection (173). The unsupervised machine learning module (175) is configured to adjust an ANN (e.g., SNN) classification, clustering, or recognition patterns in the sensor data (121) so that the degree of deviation from the classification, clustering, or recognition patterns in the sensor data (121) generated in a recent time period can be used to signal anomaly detection (173). The anomaly detection (173) allows the vehicle (111) to be scheduled for inspection at a maintenance service facility (127). Optionally, after inspection, the maintenance service data (123) can be used to apply supervised learning (171) to produce more accurate predictions for service, such as in Figure 3 middle.
[0072] In general, it may be assumed that the vehicle (111) operates in normal / healthy conditions for a certain period of time. For example, after initially delivering a new vehicle (111) for service, it may be assumed that the vehicle (111) provides trouble-free service for at least a period of time (e.g., several months). For example, after a period of time following a component replacement or repair, it may be assumed that the component provides trouble-free service for at least a period of time (e.g., several months or a year). Thus, sensor data (121) obtained during this period of time may be pre-classified as "normal" for use in, for example, Figure 4 Unsupervised learning in (175) or Figure 5 Supervised learning (171) in training ANN (125) is used.
[0073] For example, sensor data (121) collected during a "normal" service period of a vehicle (111) or component may be classified into several clusters via unsupervised learning (175). Different clusters may correspond to different types of normal conditions (e.g., driving on different routes, on roads with different surface conditions, on days with different weather conditions, during different times of the day, on different days of the week, driving in different moods of the driver's driving habits). When subsequent sensor data (121) is classified outside of a "normal" cluster, an anomaly is detected.
[0074] Optionally, supervised machine learning (171) may be used to train the ANN (125), such as Figure 5 . During a "normal" service cycle of a vehicle (111) or component, the expected classification (177) may be used to label the sensor data (121). Supervised learning (171) may be used to minimize classification differences between predictions (179) made using the ANN (125) based on the sensor data (121) and the expected classification (177). In addition, when the sensor data (121) is known to be "abnormal" (e.g., after a diagnosis is made in a maintenance service facility (127) or by a user, driver, or passenger of the vehicle (111)), the expected classification (177) may be changed to "abnormal" for further training of the ANN (125) to directly identify anomalies (e.g., instead of relying on deviations from known "normal" clusters to infer anomalies).
[0075] Thus, the ANN (125) may be trained to identify abnormal sensor data and estimate the severity of the anomaly to schedule maintenance service.
[0076] Figure 6 A method of predictive maintenance according to one embodiment is presented. For example, Figure 6 The method can be found in Figure 1 or 2 in a data storage device (101) in a vehicle (111) or in Figure 2 The invention is implemented in a computer system (131) in a vehicle (111).
[0077] At box 201 , sensors (eg, 103 ) installed in a vehicle ( 111 ) generate a sensor data stream (eg, 121 ) during operation of the vehicle ( 111 ) on a road ( 102 ).
[0078] At block 203, the sensor data stream (eg, 121) is provided to an artificial neural network (ANN) (125). For example, the ANN (125) may include a spiking neural network (SNN).
[0079] At block 205 , an artificial neural network (ANN) ( 125 ) generates a prediction of maintenance service based on the sensor data stream (eg, 121 ).
[0080] At block 207 , a data storage device ( 101 ) disposed onboard the vehicle stores at least a portion of the sensor data stream (eg, 121 ).
[0081] At block 209, an artificial neural network (ANN) is trained using a sensor data stream (eg, 121) collected over a predetermined period of time starting from the vehicle leaving the factory or maintenance service facility (127).
[0082] For example, the artificial neural network (ANN) may be configured to identify components of the vehicle (111) that require repair or replacement during maintenance service, and / or identify a predicted time period to failure or malfunction of the component, or a suggested time period to a recommended maintenance service for the component prior to failure or malfunction of the component. Thus, the performance of the predictive maintenance service may avoid the event of failure or malfunction of the component while the vehicle (111) is operating on the road (102).
[0083] For example, the sensor (103) may be a microphone mounted near the component, a vibration sensor attached to the component, a pressure sensor mounted in the component, a force or stress sensor mounted on or attached to the component, a deformation sensor attached to the component, an accelerometer configured to measure a motion parameter of the component.
[0084] Optionally, the data storage device (101), the computer system (131) of the vehicle (111), and / or a server (119) remote from the vehicle may have a machine learning module configured to train an artificial neural network (ANN) (125) during a time period in which the vehicle (111) is assumed to be in a healthy state, such as a predetermined time period starting from the time the vehicle (111) leaves a factory or maintenance service facility (127).
[0085] For example, the machine learning module may use unsupervised machine learning (175) to train the ANN (125) to recognize / classify normal patterns in the sensor data (121), and thus have the ability to detect anomalies based on deviations from normal patterns, such as Figure 4 Alternatively, supervised machine learning (171) can be used, such as Figure 3 or as described in 5.
[0086] For example, unsupervised machine learning (175) may be applied by a data storage device (101) or a computer system (131) of a vehicle (111) during a predetermined period of time during which the vehicle and / or components are known to operate without failure or degradation.
[0087] Alternatively or in combination, some of the sensor data (121) stored in the data storage device (101) of the vehicle (111) may be uploaded to the server (119) for use in training the ANN (125).
[0088] In at least some embodiments disclosed herein, a data storage device (101) is configured to accelerate computation of an artificial neural network (ANN) (125) of a vehicle (111).
[0089] For example, in addition to supporting typical operations of data access and storage, the data storage device (101) may be further configured to perform at least a portion of the computations involving an artificial neural network (ANN) (125), such as by unsupervised machine learning (175) (e.g., as described in Figure 4 ) and / or supervised machine learning (171) (e.g., as described in Figure 3 or 5) to generate predictions (e.g., 129 or 173) or classifications (e.g., 179) from sensor data (121) and / or adjustments to the ANN (125).
[0090] For example, the computations configured in the data storage device (101) can be used to reduce the amount of data to be transmitted to the processor (133) for use or application of the ANN (125) and / or reduce the computational tasks of the processor (133) in evaluating the output of the ANN (125) and / or in training the ANN (125). This arrangement can result in faster output and / or lower energy usage from the data storage device (101) because data does not have to be moved in and out of memory to a dedicated independent neural network accelerator. The computational power of the data storage device (101) in processing data related to the ANN (125) enables the computer system (131) of the motor vehicle (111) to monitor the health of the vehicle components (e.g., in a non-real-time or pseudo-real-time manner) with reduced or no impact on the processing of critical tasks (e.g., autonomous driving by ADAS (105)). In addition, the computational power of the data storage device (101) can be used to accelerate the processing of sensor data by the ADAS (105) and thereby improve the processing of critical tasks.
[0091] Figure 7 A data storage device (101) for accelerating neural network computations according to one embodiment is shown. For example, Figure 7 The data storage device (101) can be used to implement Figure 1 Or a data storage device (101) of a vehicle (111) described in 2.
[0092] exist Figure 7 In the embodiment of the present invention, the data storage device (101) has a host interface (157) configured to communicate with a processor (e.g., 133). For example, the communication between the processor (e.g., 133) and the host interface (157) may be based at least in part on a communication protocol for a peripheral component interconnect express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a universal serial bus (USB) bus, and / or a storage area network (SAN).
[0093] For example, the host interface (157) may be used to receive sensor data (121) generated by a sensor (103) of a vehicle (111) to optionally store a portion of the sensor data (121) in a storage media component (161-163).
[0094] For example, each of the storage media components (161-163) may be a memory integrated circuit configured to store data. For example, the media component (161 or 163) may include one or more integrated circuit dies embedded in an integrated circuit package. The integrated circuit die may have many memory cells formed thereon to store data.
[0095] Generally speaking, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and retain stored data even without power.
[0096] Examples of non-volatile memory include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase change memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point memory, and memory devices. Cross-point memory devices use transistor-free memory elements, each of which has memory cells and selectors stacked together as columns. The memory element columns are connected via two vertical wire layers, one of which is above the memory element columns and the other is below the memory element columns. Each memory element can be individually selected at the intersection of a wire on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electrically erasable programmable read-only memory (EEPROM) memory, etc. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).
[0097] The data storage device (101) may have a controller (151) including a volatile local memory (153) and at least one processing device (155).
[0098] The local memory of the controller (151) may be an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines for controlling the operation of the processing device (155), including handling communications between the data storage device (101) and the processor (e.g., 133) of the vehicle (111), as well as other functions described herein. The local memory (153) of the controller (151) may include a read-only memory (ROM) for storing microcode, and / or memory registers and / or volatile memory such as dynamic random access memory (DRAM) and static random access memory (SRAM) for storing, for example, memory pointers, fetch data, etc.
[0099] exist Figure 7 In the invention, the data storage device (101) includes a neural network accelerator (159) coupled to a controller (151) and / or storage media components (161 to 163).
[0100] For example, the neural network accelerator (159) may be configured to perform matrix arithmetic calculations. Calculations involving the ANN (125) have matrix multiplication and accumulation operations, which may be computationally intensive for a general purpose processor (e.g., 133). Using the neural network accelerator (159) to perform matrix arithmetic calculations may reduce the data to be transmitted to the processor (133) of the vehicle (111) and reduce the computational workload for the processor (133).
[0101] For example, when the ANN (125) includes a spiking neural network (SNN), simulation of the differential equations used to control the activation levels of the SNN neurons may be computationally intensive for a general purpose processor (e.g., 133). The neural network accelerator (159) may use special hardware to simulate the differential equations and thus improve the computational efficiency of the computer system (131) as a whole.
[0102] In some implementations, the neural network accelerator (159) is an integrated circuit device separate from the controller (151) and / or the storage media components (161-163). Alternatively or in combination, the neural network accelerator (159) is integrated with the controller (151) in an integrated circuit package. In addition, the neural network accelerator (159) may be integrated into at least one of the storage media components (161-163), such as Figure 8 As described in .
[0103] Figure 8 A storage media component (160) is shown for accelerating neural network computations according to one embodiment. For example, Figure 7 Each or some of the storage media components (161 to 163) in Figure 8The invention is implemented by a storage media component (160).
[0104] exist Figure 8 In the embodiment of the present invention, the storage media component (160) can be housed within an integrated circuit package. An input / output (I / O) interface (171) of the storage media component (160) is configured to process input / output signals in the pins of the integrated circuit package. For example, the input / output signals can include an address signal to specify a location in the media unit (175), and a data signal representing data to be written into the media unit (175) at the location specified by the address signal, or data to be retrieved from the location in the media unit (175).
[0105] exist Figure 8 In the embodiment of the present invention, the neural network accelerator (159) is coupled to the control logic (173) and / or the media unit (175) to perform calculations used in evaluating the output of the ANN (125) and / or in training the ANN (125).
[0106] For example, the input / output interface (171) may receive an address identifying a matrix stored in a media unit and to be operated on via the neural network accelerator (159). The storage media component (160) may provide the computational results of the neural network accelerator (159) as output data in response to the address, store the output data in a buffer for further operation, and store the output data to a location in the media unit (175) specified via the address signal. Thus, the computation performed by the neural network accelerator (159) may be within the storage media component (160), which is proximate to the media unit (175) storing the matrix data. For example, each of the media units (175) may be an integrated circuit die on which a memory cell of a non-volatile memory is formed.
[0107] For example, state data of SNN neurons may be stored in the media unit (175) according to a predetermined pattern. The neural network accelerator (159) may automatically update the state of the SNN neurons according to differential equations for controlling the activation level of the SNN neurons over time. Optionally, the neural network accelerator (159) is configured to process spikes of neurons in the neural network. Alternatively, the data storage device (101) and / or the neural network accelerator (159) of the processor (133) may be configured to process spikes of neurons and / or accumulation of inputs to the SNN.
[0108] Fig. 9 A method for accelerating neural network computation in a motor vehicle is presented according to one embodiment. For example, Fig. 9 The method can be used Figure 7 Data storage device (101) and / or Figure 8The storage media component (160) is Figure 1 or 2 is implemented in a vehicle (111). For example, Fig. 9 The method can be combined with Figure 6 The method used.
[0109] At block 221, a data storage device (101) of a vehicle (111) receives a sensor data stream from at least one sensor (eg, 103) disposed on the vehicle (111).
[0110] At block 223, the data storage device (101) stores at least a portion of the sensor data stream.
[0111] At block 225, a neural network accelerator (159) configured within the data storage device (101) performs at least a portion of the computation based on the artificial neural network (125) and the sensor data stream.
[0112] At box 227, maintenance service for the vehicle (111) is predicted based at least in part on calculations performed by a neural network accelerator (159) configured within the data storage device (101).
[0113] Optionally, at box 229, an artificial neural network (ANN) is trained in the vehicle (111) using at least a portion of the neural network accelerator and using sensor data streams collected within a predetermined time period, wherein the predetermined time period is, for example, a period after a new vehicle (111) is delivered from a factory or after a component is replaced in a maintenance service facility (127).
[0114] For example, the neural network accelerator (159) may be configured on an integrated circuit device separate from the controller (151) of the data storage device and / or separate from the storage media components (161-163).
[0115] For example, the neural network accelerator (159) may be configured on an integrated circuit device that includes a controller (151) of the data storage device (101), or on an integrated circuit device that includes a storage media component (160, 161, or 163) of the data storage device (101).
[0116] For example, the neural network accelerator (159) may be configured to perform computations using data stored in the data storage device (101), such as matrix arithmetic computations for an ANN and / or differential equation simulations for an SNN.
[0117] Examples of matrix arithmetic calculations include matrix multiplication and accumulation operations. After performing calculations using data stored in the data storage device (101) to produce the results of the matrix arithmetic calculations, the neural network accelerator (159) may provide the results as outputs of the data storage device (101) upon retrieval of the data (e.g., in response to a read command). Alternatively or in combination, the results of the matrix arithmetic calculations may be buffered in the data storage device (101) as operands for the next matrix calculation to be performed in combination with a matrix of data retrieved from the non-volatile memory via a read command received in the host interface (157).
[0118] When the artificial neural network (ANN) (125) includes a spiking neural network (SNN), the neural network accelerator can be configured to simulate differential equations that control activation levels of neurons in the spiking neural network (SNN). Optionally, the storage medium component is configured to store the states of the neurons in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to automatically update the states of the neurons over time according to the differential equation. For example, the neural network accelerator (159) can be configured to train the spiking neural network (SNN) to detect anomalies through unsupervised machine learning.
[0119] Computations performed by a neural network accelerator (159) based on an artificial neural network (ANN) (125) involve different types of data with different usage patterns of a data storage device (101).
[0120] For example, making predictions using an artificial neural network (ANN) (125) includes using data specifying a model of the artificial neural network (ANN) (125), input data provided to the artificial neuron, and output data generated by the artificial neuron.
[0121] The storage capacity of the data storage device (101) can be divided into different portions for different types of ANN-related data. The different portions can be individually configured to optimize access and storage of corresponding data according to the mode in which they are used by the neural network accelerator (159) and / or the processor (133) of the computer system (131) in which the data storage device (101) is configured.
[0122] The model of an artificial neural network (ANN) (125) may include parameters that specify static properties of individual artificial neurons in the ANN (125) and connectivity of neurons in the ANN (125). The model data of the ANN (125) is static and does not change during the use of the ANN (125) for predictive calculations. Therefore, the usage pattern of the model data is primarily read. However, the model data of the ANN (125) may change when an update to the ANN (125) is installed. For example, the vehicle (111) may download the update to the ANN (125) from the server (119) to the data storage device (101) of the vehicle (111) to update its predictive capabilities. The model data of the ANN (125) may also change during or after training the ANN (125) using machine learning techniques (e.g., 171 or 175). Preferably, a separate partition or namespace of the data storage device (101) is configured to store the model data, wherein the partition or namespace is operated according to configuration parameters of the memory cells optimized for a particular usage pattern of the model data (e.g., mostly read, infrequently updated). For example, when the memory cells are implemented using NAND logic gate-based flash memory, the memory cells in the ANN model partition / namespace may be configured to operate in a multi-level cell (MLC) mode, a three-level cell (TLC) mode, or a four-level cell (QLC) mode, wherein each memory cell stores two, three, or four bits for increased storage capacity.
[0123] Input data provided to the artificial neurons in the ANN (125) may include external inputs and internal inputs. The external inputs are typically generated by sensors (103) of the vehicle (111) rather than by the artificial neurons in the ANN (125). The external inputs may be stored in a cyclic manner so that input data for a recent time period having a predetermined driving length may be found in the data storage device (101). Therefore, it is preferred that a separate partition or namespace of the data storage device (101) is configured to store the external input data, wherein the partition or namespace is operated according to configuration parameters of the memory cells optimized for the storage mode of the external input data (e.g., enhanced endurance, cyclic overwrite). For example, when the memory cells are implemented using flash memory based on NAND logic gates, the memory cells in the ANN input partition / namespace may be configured to operate in a single level cell (SLC) mode, wherein each memory cell stores one data bit for improved endurance in cyclic overwrite operations.
[0124] In some embodiments, an artificial neuron may have state variables that change over time in response to input during a prediction computation. For example, the activation level of a spiking neuron may change over time and is considered a dynamic state variable of the spiking neuron. In some embodiments, this state variable data of the artificial neuron has a storage usage pattern similar to external input data; and therefore, the state variable data may be stored in a partition or namespace configured for external input data. In other embodiments, the state variable data of the artificial neuron is maintained in a buffer and is saved less frequently than the external input; and therefore, another partition / namespace may be configured for storing the dynamic state variable data of the artificial neuron.
[0125] Output data generated by artificial neurons in the ANN (125) may be buffered for further access by a neural network accelerator (159) and / or a processor (133) of a computer system (131). The output data may include external outputs and internal outputs. External inputs are generated by artificial neurons as outputs from the ANN (125), such as as a result of classification or prediction by the ANN (125). The outputs of the ANN (125) are typically further processed by the processor (133) of the computer system (131). External inputs may be periodically saved (e.g., in a manner similar to the storage of state variable data). Internal outputs and / or some external outputs may be internal inputs to artificial neurons in the ANN (125). Generally, it may not be necessary to store internal outputs from a buffer of a data storage device to a storage media component. In some embodiments, when the buffering capacity of the data storage device (101) is insufficient to hold the entire state variable data and / or internal outputs, the data storage device (101) may use a swap partition / namespace to expand the capacity of the buffer. The swap partition / namespace may be configured for optimizing random access and for improving durability.
[0126] The external outputs and / or dynamic states of the neurons may be stored in a separate output partition or namespace in a cyclic manner, such that the external output data and / or dynamic states of the neurons may be stored periodically and the latest set of external outputs and / or dynamic states may be found in the data storage device (101). The external outputs and / or dynamic states of the neurons may be selectively stored because some such data may be regenerated by the ANN from external inputs stored in the input partition or namespace. Preferably, the output partition or namespace is configured to store one or more sets of external outputs and / or dynamic states that cannot be created from external inputs stored in the input partition or namespace. When storing data in the input / output partition or namespace in a cyclic manner, the oldest stored data set is erased to make room for the latest data set. The ANN input / output partition / namespace may be configured for an optimized sequential write flow for copying data from a buffer of the data storage device to a memory cell in a storage media component of the data storage device.
[0127] Fig.10 A data storage device (101) configured to support neural network computations according to one embodiment is shown. For example, the data storage device (101) may be Figure 1 or 2 for use in a vehicle (111) to facilitate predictive maintenance and / or support ADAS (105).
[0128] Fig.10 The data storage device (101) includes a host interface (157) and a controller (151), which is similar to Figure 7 A data storage device (101).
[0129] Fig.10 The storage capacity (181) of the data storage device (101) may be implemented using a set of storage media components similar to Figure 7 A storage media component (161 to 163) in a data storage device (101).
[0130] A set of namespaces (183, 185, 187, ...) may be created on a storage capacity (181) of a data storage device (101). Each of the namespaces (e.g., 183, 185, or 187) corresponds to a named portion of the storage capacity (181). Logical addresses are defined in each namespace. An address mapping (191) is configured to associate logical addresses defined in the namespaces (183, 185, 187, ...) with storage media components (e.g., Figure 7 Mapping between the physical addresses of the memory cells in 161 to 163) described in .
[0131] The address map (191) may include namespace optimization settings (192) for the namespaces (183, 185, and 187).
[0132] For example, the ANN model namespace (183) can be a memory / storage partition configured for model data of an artificial neural network (ANN) (125). The namespace optimization settings (192) optimize memory operations in the ANN model namespace (183) based on the data usage pattern of the ANN model (e.g., centered around mostly reads, infrequent updates).
[0133] For example, the neuron input namespace (185) may be a memory / storage partition configured for external input data to the artificial neural network (ANN) (125). The namespace optimization settings (192) optimize memory operations in the neuron input namespace (185) according to data usage patterns of the external input data (e.g., enhanced durability to support cyclic overwrites for sequentially written continuous input data streams).
[0134] For example, the neuron output namespace (187) may be a memory / storage partition configured for external output data provided from an artificial neural network (ANN) (125). The namespace optimization settings (192) optimize memory operations in the neuron output namespace (187) according to the data usage pattern of the external output data (e.g., improved durability for periodically overwriting data with random read / write accesses).
[0135] The data storage device (101) includes a buffer (152) configured to store temporary / intermediate data of an artificial neural network (ANN) (125), such as internal inputs / outputs of artificial neurons in the ANN (125).
[0136] Optionally, a swap namespace may be configured in the storage capacity (181) to expand the capacity of the buffer (152).
[0137] Optionally, the address map (191) includes a mapping between logical memory addresses received in the host interface (157) to access data for an artificial neuron and the identity of the artificial neuron. Thus, a read or write command to access one type of data for an artificial neuron in one namespace may cause the controller 151 to access another type of data for an artificial neuron in another namespace.
[0138] For example, in response to a request to write external input data for a neuron into the storage capacity (181) of the data storage device (185), the address map (191) may be used to calculate the address of the model parameters of the neuron in the ANN model namespace (183) and read the model parameters into the buffer (152) to allow the neural network accelerator (159) to perform calculations of the output of the neuron. The output of the neuron may be saved in the buffer (152) as internal input to other neurons (e.g., to reduce write amplification). In addition, the identities of other neurons connected to the neuron may also be retrieved from the ANN model namespace (183) into the buffer (152), which allows the neural network accelerator (159) and / or processor to further process the propagation of the output in the ANN (125). The retrieval of model data from the ANN model namespace (183) may be performed in parallel with the storage of the external input data into the neuron input namespace (185). Therefore, the processor (133) of the computer system (131) of the vehicle (111) does not have to explicitly enter a read command for retrieving model data from the ANN model namespace (183).
[0139] Similarly, in response to reading the output data of the neuron, the address map (191) can be used to calculate the address of the model parameters of the neuron stored in the ANN model namespace (183) and read the model parameters into the buffer (152) to allow the neural network accelerator (159) to apply the internal input in the buffer (152) to perform the calculation of the output of the neuron. The calculated output can be provided in response to reading the output data for the neuron without the data storage device (101) having to store the output data in the storage media components (e.g., 161-163). Thus, the processor (133) and / or the neural network accelerator (159) can control the calculation of the neuron by writing input to the neuron and / or reading output from the neuron.
[0140] In general, the external input data passed into the ANN (125) may be raw sensor data (121) generated directly by the sensor (103) without being processed by the processor (133) and / or the neural network accelerator (159). Alternatively, indirect sensor data (121) that has been processed by the processor (133) of the ANN (125) based on the signal from the sensor (103) may be provided as the external input data. The incoming external input data may be accepted at the host interface (157) and written to the neuron input namespace (185) in a round-robin manner and automatically buffered in the buffer (152) for the neural network accelerator (159) to generate neuron output using the model stored in the ANN model namespace (183). The output generated by the neural network accelerator (159) may be further buffered as internal input for further application of the model in the ANN model namespace (183). When the external output becomes available, the data storage device (101) may report the completion of the write request with an indication of the availability of the external input. Optionally, the controller 151 and / or the neural network accelerator (159) may generate an internal read command to propagate a signal in the ANN (125) when an external output is generated. Alternatively, the host processor (133) may control the propagation of the signal in the ANN (125) by selectively reading the output of the neuron; and the data storage device (101) may actively buffer the data that may be needed in the buffer (152) to accelerate the ANN calculation.
[0141] Fig.11 The configuration of a namespace (183) for an artificial neural network (ANN) model according to one embodiment is described. For example, Fig.11 The configuration can be found in Figure 7 and / or the data storage device (101) described in 10. For example, Fig.11 The settings (193) can be Fig.10 Part of the Namespace Optimization Settings (192).
[0142] Fig.11The configuration maps the ANN model namespace (183) to at least one storage media component A (161). Preferably, the at least one storage media component A (161) can be used by the controller (151) in parallel with storage media components (e.g., 163) of other namespaces (e.g., 185 and 187) hosting ANN data. For example, the storage media component A (161) can be in an integrated circuit package that is separate from the integrated circuit packages used for the other namespaces (e.g., 185 and 187). Alternatively, the storage media components (161-163) are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components (161-163) can be formed on separate regions of the integrated circuit die, where the separate regions can be operated in substantially parallel (e.g., for reading, for erasing, and / or for writing).
[0143] exist Fig.11 In, setting (197) can be optimized for a usage pattern that is mostly read and infrequently updated.
[0144] Fig.12 The configuration of the namespace (185) for input to an artificial neuron according to one embodiment is described. For example, Fig.11 The configuration can be found in Figure 7 and / or the data storage device (101) described in 10. For example, Fig.11 The settings (195) can be Fig.10 Part of the Namespace Optimization Settings (192).
[0145] Fig.12 The configuration maps the neuron input namespace (185) to at least one storage media component B (163). Preferably, the at least one storage media component B (163) can be used by the controller (151) in parallel with storage media components (e.g., 161) of other namespaces (e.g., 183 and 187) hosting ANN data. For example, the storage media component B (163) can be in an integrated circuit package that is separate from the integrated circuit packages used for the other namespaces (e.g., 183 and 187). Alternatively, the storage media components (161-163) are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components (161-163) can be formed on separate regions of the integrated circuit die, where the separate regions can be operated in substantially parallel (e.g., for reading, for erasing, and / or for writing).
[0146] exist Fig.12 In the embodiment, setting (197) is optimized for use in a cyclic sequential overwrite mode for enhanced endurance when recording a continuous stream of input data sampled at fixed time intervals.
[0147] Fig.13 The configuration of the namespace (187) for outputs from an artificial neuron according to one embodiment is illustrated. For example, Fig.11 The configuration can be found in Figure 7 and / or the data storage device (101) described in 10. For example, Fig.11 The settings (197) can be Fig.10 Part of the Namespace Optimization Settings (192).
[0148] Fig.13 The configuration maps the neuron output namespace (187) to at least one storage media component C (162). Preferably, the at least one storage media component C (162) can be used by the controller (151) in parallel with storage media components (e.g., 161 and 163) of other namespaces (e.g., 183 and 185) hosting ANN data. For example, the storage media component C (162) can be in an integrated circuit package that is separate from the integrated circuit packages used for the other namespaces (e.g., 183 and 185). Alternatively, the storage media components (161-163) are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components (161-163) can be formed on separate regions of the integrated circuit die, where the separate regions can be operated in substantially parallel (e.g., for reading, for erasing, and / or for writing).
[0149] exist Fig.13 In the example, the settings (197) are optimized for a usage pattern of buffered data that is periodically overwritten with random access. For example, the memory cells are configured via the optimized settings (193-197) to update / overwrite in the neuron output namespace (187) at a higher frequency than in the ANN model namespace (183) but less frequently than in the neuron input namespace (185).
[0150] Fig.14 A method of predictive maintenance supported by model partitioning according to one embodiment is presented. For example, Fig.14 The method can be used Figure 7 or the data storage device (101) of 10 and / or Figure 8 The storage media component (160) is Figure 1 or 2 in a vehicle (111). For example, Figure 6 And / or method 9 uses Fig.14 method.
[0151] At block 241, a non-volatile memory of the data storage device (101) is configured into a plurality of partitions (e.g., 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) for storing data; and the same type of memory cells in different partitions (e.g., 183 to 187) may be configured differently to optimize their performance according to the usage pattern of the data stored in the different partitions (e.g., 183 to 187).
[0152] At box 243, the data storage device (101) stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network (125) for partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include a model partition (e.g., 193) configured to store model data of the artificial neural network (125).
[0153] At block 245, the data storage device (101) receives a sensor data stream (eg, 121) from at least one sensor (103) disposed on the vehicle (111).
[0154] At box 247, the controller (151) of the data storage device (101) operates memory cells in the partitions (183, 185, 187, ...) according to the multiple groups of memory operation settings (e.g., 193, 195, 197) in response to the sensor data stream (e.g., 121).
[0155] At box 249, a computer system (131) having a data storage device (101) uses an artificial neural network (125) to predict maintenance service for a vehicle (111) based on a sensor data stream (eg, 121).
[0156] For example, the memory operation setting configures the model partition (e.g., 183) to store three or more bits per memory cell. The memory operation setting may include an address map (191) to map between neurons in the ANN (125) and inputs to the neurons. When a first address of an input to a neuron in the artificial neural network (125) is received, the first address in an input partition (e.g., 185) separate from the model partition (e.g., 183) may be converted into at least one second address of model data associated with the neuron, so that the properties of the neuron and the identity of the neuron connected to the neuron can be retrieved from the model partition (e.g., 183) without an explicit command from the processor (133). In response to receiving the first address, the controller (151) may automatically retrieve the model data associated with the neuron from the model partition (e.g., 183) using the at least one second address. The neural network accelerator (159) may generate an output of the neuron from the input to the neuron and the model data associated with the neuron. In general, the input to the neuron may include outputs from a plurality of neurons connected to the neuron in the ANN (125). The controller (151) may store the output of the neuron in a buffer (152) in the data storage device (101) to facilitate accelerated access to the output by the host processor (133) and / or the neural network accelerator (159).
[0157] Typically, the model data does not change during calculations used to predict maintenance service. For example, the model data may include neuron connectivity data of an artificial neural network and static properties of neurons in the artificial neural network. The memory operation settings (e.g., 192) may configure the model partition (e.g., 183) to store more than one bit per memory cell in the non-volatile memory based on a mostly read, infrequently updated usage pattern of the model data.
[0158] For example, partitions (e.g., 183, 185, 187, ...) in a data storage device may be implemented as namespaces in which logical addresses are defined; and an address mapping (191) in the data storage device is configured to map the namespaces (183, 185, 187, ...) to separate storage media components (e.g., 161, 163, 162, ...).
[0159] The model data in the model namespace (183) is updateable during training via machine learning (171 or 175), or during over-the-air updates of the ANN (125) from the server (119).
[0160] In some embodiments, the controller (151) is configured to retrieve model data associated with a neuron in the artificial neural network from a model partition via an address map (191) in response to an input to the neuron or an output from the neuron being addressed in a partition separate from the model partition (183). In addition, the controller (151) may retrieve model data associated with a neuron from the model partition (183) in parallel with storing input to a neuron in a partition (e.g., 185) separate from the model partition (183).
[0161] Fig.15 A method for predictive maintenance supported by input partitioning according to one embodiment is presented. For example, Fig.15 The method can be used Figure 7 or the data storage device (101) of 10 and / or Figure 8 The storage media component (160) is Figure 1 or 2 in a vehicle (111). For example, Figure 6 , 9 and / or 14 ways to use Fig.15 method.
[0162] At block 261, the non-volatile memory of the data storage device (101) is configured into multiple partitions (e.g., 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) implemented in multiple storage media components (e.g., 161-163).
[0163] At box 263, the data storage device (101) stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network (125) for partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include an input partition (e.g., 185) configured to cyclically store input data for the artificial neural network (125).
[0164] For example, the input partition (185) may be configured to store external inputs for the artificial neural network (125), but not internal inputs. The input data stored in the input partition (185) is independent of the outputs from the neurons in the artificial neural network (125).
[0165] For example, the input data stored in the input partition (185) may include a portion of a sensor data stream (e.g., 121). In some embodiments, the input data stored in the input partition (185) is computed from the sensor data stream (e.g., 121) for a subset of neurons in the artificial neural network (125).
[0166] For example, the memory operation settings (eg, 195) configure the input partition (185) to store one bit per NAND memory cell in non-volatile memory to achieve enhanced endurance against repeated data erasure and data programming.
[0167] For example, the memory operation settings (eg, 195) configure the controller to sequentially write input data to the input partition (185) and overwrite the oldest input data in the input partition (185) with the newest input data received in the data storage device (101).
[0168] At block 265, the data storage device (101) receives a sensor data stream (eg, 121) from at least one sensor (103) disposed on the vehicle (111).
[0169] At box 267, the controller (151) of the data storage device (101) operates the memory cells in the partitions (183, 185, 187, ...) in response to the sensor data stream (e.g., 121) according to the multiple sets of memory operation settings (e.g., 193, 195, 197).
[0170] At box 269, a computer system (131) having a data storage device (101) uses an artificial neural network (125) to predict maintenance service for a vehicle (111) based on a sensor data stream (eg, 121).
[0171] Fig.16 A method for predictive maintenance supported by input partitioning according to one embodiment is presented. For example, Fig.16 The method can be used Figure 7 or the data storage device (101) of 10 and / or Figure 8 The storage media component (160) is Figure 1 or 2 in a vehicle (111). For example, Figure 6 , 9 , 14 and / or 15. Fig.16 method.
[0172] At block 281, the non-volatile memory of the data storage device (101) is configured into multiple partitions (eg, 183, 185, 187, ...). The non-volatile memory may have the same type of memory cells (eg, NAND flash memory cells) for storing data.
[0173] At box 283, the data storage device (101) stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network (125) for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include an output partition (e.g., 187) configured to store output data for the artificial neural network (125).
[0174] For example, the input partition (185) may be configured to store external inputs for the artificial neural network (125), but not internal inputs. The input data stored in the input partition (185) is independent of the outputs from the neurons in the artificial neural network (125).
[0175] For example, the output data stored in the output partition (e.g., 187) may include state data of a neuron in the artificial neural network (125). For example, the state data of a neuron in the artificial neural network may identify an activation level of the neuron for a spike in a spiking neural network. The activation level may be controlled via a differential equation. Thus, the activation level may change in response to input to the artificial neural network (125) and / or in response to the passage of time.
[0176] For example, the output data may include predictions or classifications generated by the artificial neural network (125) in response to the sensor data stream.
[0177] For example, a memory operation setting configures the output partition configuration to store no more than two bits per memory cell in the non-volatile memory.
[0178] At block 285, the data storage device (101) receives a sensor data stream (eg, 121) from at least one sensor (103) disposed on the vehicle (111).
[0179] At box 287, the controller (151) of the data storage device (101) operates the memory cells in the partitions (183, 185, 187, ...) in response to the sensor data stream (e.g., 121) according to the multiple sets of memory operation settings (e.g., 193, 195, 197).
[0180] At box 289, a computer system (131) having a data storage device (101) uses an artificial neural network (125) to predict maintenance service for a vehicle (111) based on a sensor data stream (eg, 121).
[0181] For example, the data storage device (101) may include a buffer (152). The buffer (152) may be implemented via a volatile memory (e.g., SRAM or DRAM) for faster access performance than a non-volatile memory (e.g., NAND flash memory) of the data storage device (101). The memory operation setting configures the controller (151) to store the output data in the buffer (152) for access by the processor (e.g., 133) via the host interface (157) during or after storing the output data in the output partition (187).
[0182] For example, the data storage device (101) may include a neural network accelerator (159) coupled to the controller (151). The neural network accelerator is configured to apply inputs provided to neurons in the artificial neural network (125) to a data model of the artificial neural network (125) to generate output data through one or more output neurons in the artificial neural network (125). In response to the neural network accelerator (159) completing the calculation of the output data, the controller is configured to provide an indication of the availability of the output data generated by the artificial neural network (125) to the processor (e.g., 133), so that the processor (e.g., 133) can request the data storage device (101) to transmit the output data.
[0183] Optionally, the controller (151) is configured to provide output data to the processor in parallel with storing the output data in the output partition. For example, the controller (151) may be configured to automatically discard output data calculated for a previous segment of the sensor data stream if the processor (e.g., 133) does not request transmission of the output data to the processor (e.g., 133) within a predetermined time period or before the next version of the output data is available. Optionally, after reporting the availability of the output data to the processor (e.g., 133), the controller (151) may be configured to selectively discard output data calculated for a previous segment of the sensor data stream based on the processor (e.g., 133)'s response to the output data to the processor (e.g., 133). For example, in some cases, a processor (e.g., 133) may request that output data be transferred to the processor (e.g., 133) without saving the output data to an output partition (e.g., 187); and in other cases, a processor (e.g., 133) may request that output data be transferred to the processor (e.g., 133) and that the output data be stored in an output partition (e.g., 187).
[0184] Optionally, output data from the artificial neural network (125) may also be stored in a round-robin fashion into output partitions (eg, segments of the output data for time periods selected by the processor (eg, 133)).
[0185] For example, external inputs to the artificial neural network (125) may be continuously recorded in the input namespace (185) during the last time period of T1. When the sensor data is sampled at a predetermined time interval T2, the input namespace (185) may maintain the latest T1 / T2 set of input data. In contrast, external outputs from the artificial neural network (125) may be selectively recorded in the output namespace (187) (e.g., once every predetermined time period T3, where T3 is a multiple of T2). Output data may be recorded in the output namespace (187) at a lower frequency; and the output namespace (187) may be allocated to store a predetermined number of sets of output data (e.g., via sequential writing and writing in a cyclic manner to maintain the latest set of output data).
[0186] At least some embodiments disclosed herein include a communication protocol / interface that allows a data storage device to perform neural network acceleration in real time with reduced data traffic to a host processor (e.g., a central processing unit (CPU)).
[0187] For example, a main processor (e.g., 133) of the vehicle (111) may provide a write command to the data storage device (101) to store the model of the artificial neural network in the model partition (e.g., 183). Since the neural network accelerator (159) is configured to apply the model, data communications for sending data of the model of the ANN (125) back to the processor may be reduced or eliminated.
[0188] To use the ANN model in classification and / or prediction, a main processor (e.g., 133) of a vehicle (111) may stream input data for the ANN (125) into a neuron input partition (e.g., 185). A neural network accelerator (159) of a storage device (101) may automatically apply the input data to a model stored in an ANN model partition (e.g., 183) according to an address map (191). The data storage device (101) makes the computation output available for propagation in the ANN (125). Preferably, the computation output is made available to the neural network accelerator (159) via a buffer (152) without storing intermediate outputs in storage media components (e.g., 161 to 163). Thus, data communications between a main processor (e.g., 133) and a data storage device (101) for transmission of neuron outputs may be reduced. When the output has been propagated to the output neuron in the ANN (125), the data storage device (101) may provide a response to a write request associated with writing a set of input data to the neuron input partition (e.g., 185). The response indicates that the external output from the neuron in the ANN (125) is available. In response, the main processor (e.g., 133) of the vehicle (111) may optionally issue a read command to retrieve the external output for further processing.
[0189] Fig.17 Communication with a data storage device (101) for performing neural network calculations is shown according to one embodiment. For example, Fig.17 The communication described in Figure 7 or the data storage device (101) described in 10 Figure 1 or 2 is implemented in a vehicle (111).
[0190] exist Fig.17 In the embodiment of the present invention, the processor (133) may be configured with a simplified set of instructions (301) to perform neural network calculations because some of the calculations involving the ANN (125) are performed by the neural network accelerator (159) within the data storage device (101). Therefore, it is not necessary to transmit model data back to the processor (133) during the use of the ANN (125) for prediction and / or classification.
[0191] The sensor (103) may generate a continuous stream of sensor data (121) based on a rate for sampling the data. The sensor data (121) may be sampled at fixed predetermined time intervals (e.g., during operation of the vehicle (111)). The processor (133) may execute instructions (301) to convert the sensor data (121) into an input stream (303) for input neurons in the ANN (125). The input neurons in the ANN (125) are configured to receive external inputs to the ANN (125); and the output neurons are configured to provide external outputs (125) from the ANN.
[0192] In general, a complete set of inputs for the ANN (125) at a time instance contains inputs for the entire set of input neurons of the ANN (125). The input stream (303) contains a sequence of sets of inputs for a sequence of time instances spaced apart from each other according to fixed predetermined time intervals.
[0193] The data storage device (101) stores the input stream (303) into the neuron input namespace (185) in a round-robin manner, wherein the oldest input group corresponding to the oldest time example of data sampling for the data group currently stored in the neuron input namespace (185) is erased to store the latest group input in the input stream (303).
[0194] For each input data set, the neural network accelerator (159) applies the model of the ANN (125) stored in the ANN model namespace (183). The neural network accelerator (159) (or the processor (133)) can control the propagation of signals within the neural network. When an output neuron of the ANN (125) produces its output in response to the input data set, the data storage device (101) can provide an indication to the processor (133) that the neuron output is ready for retrieval. The indication can be configured in response to a request from the processor (133) to write the input data set to the neuron input namespace (185). The processor (133) can optionally retrieve the output data (305) (e.g., according to conditions and / or criteria programmed in the instructions).
[0195] In some embodiments, the trigger parameters are configured in the data storage device (101). When the output parameters in the external output (317) meet the requirements specified by the trigger parameters, the data storage device provides a response to the request from the processor (133) to write the input data set into the neuron input namespace (185).
[0196] Fig.18 Communication within a data storage device for implementing neural network computations according to one embodiment is shown. Fig.17 communication, Fig.18 The communication can be found at Figure 7 Or implemented in the data storage device (101) described in 10.
[0197] exist Fig.18 In the ANN (125), the model namespace (183) stores the model (313) of the entire ANN (125). In response to receiving a set of external inputs (315) for a time instance from the input stream (303) in the buffer (152), the data storage device (101) can write the external inputs (315) to the input namespace (185) in parallel with retrieving a neuron model (312) that contains a portion of the ANN model (313) that corresponds to the parameters of the input neurons and / or the identities of neurons connected to the input neurons. The buffer (152) allows the neural network accelerator (159) to combine the neuron model (312) and the external inputs (325) to produce an output (327) for the input neurons.
[0198] In general, the neuron output (327) may include a portion of the internal output (316) for further propagation within the ANN (125) and / or a portion of the external output (317) for the processor (133).
[0199] The internal output (316) is stored in the buffer (152) as the internal input (316) for further propagation in the ANN (125) in a manner similar to the generation of the neuron output (327) from the external input (315). For example, a portion of the internal input (316) may cause the controller (151) and / or the neural network accelerator (159) to retrieve the corresponding neuron model (312) associated with the internal input, so that the internal input is applied to the corresponding neuron model (312) in the neural network accelerator (159) to generate its neuron output (327).
[0200] When a complete set of external outputs (317) is available in the buffer (152), the external outputs (317) may be stored in the output namespace (187).
[0201] Optionally, the storage device (101) does not store every set of external outputs (317) corresponding to a set of stored external inputs (315) sampled at the time instance. For example, the storage device (101) may be configured to store a set of external outputs (317) for every predetermined number of sets of external inputs (e.g., 315). Alternatively or in combination, the processor (133) may determine whether to store the external outputs (317). For example, the storage device (101) may be configured to store the external outputs (317) in response to the processor (133) retrieving the external outputs (317) for further processing. For example, the storage device (101) may be configured to store the external outputs (317) in response to a write command from the processor (133) after the external outputs (317) are processed in the processor (133).
[0202] Fig.19 A method of communicating with a data storage device for performing neural network calculations according to one embodiment is shown. For example, Fig.19 The method can be used Figure 7 or the data storage device (101) of 10 and / or Figure 8 The storage media component (160) Figure 1 or 2 in a vehicle (111). For example, Figure 6 , 9 , 14, 15 and / or 16 Fig.19 method.
[0203] At box 341, one or more processors (133) of the vehicle (111) stores model data (eg, 313) of the artificial neural network (eg, 125) in the data storage device (101).
[0204] At block 343, one or more processors (133) of the vehicle (111) receives a set of sensor data from at least one sensor (103) disposed on the vehicle (111).
[0205] At box 345, one or more processors (133) of the vehicle (111) generates a set of inputs to the artificial neural network (e.g., 125) based on the sensor data.
[0206] At box 347, one or more processors (133) of the vehicle (111) provide the set of inputs to the data storage device (101). The data storage device (101) is configured to generate a set of outputs using model data (313) of the artificial neural network (e.g., 125) in response to the set of inputs.
[0207] At box 349, the one or more processors (133) of the vehicle (111) retrieve the set of outputs from the data storage device (101).
[0208] For example, the data storage device (101) uses at least a portion of the model data (183) stored in the data storage device to generate the set of outputs without transmitting the portion of the model data (183) to one or more processors (133) between receipt of the set of inputs and completion of calculation of the set of outputs.
[0209] For example, portion of the model data (183) may include static properties of neurons in an artificial neural network (eg, 125) and / or neuron connectivity data of the artificial neural network (eg, 125).
[0210] For example, to provide the set of inputs to the data storage device (101), one or more processors (133) of the vehicle (111) may transmit one or more write commands to the data storage device (101). The one or more write commands are configured to instruct the data storage device (101) to store the set of inputs in the data storage device (101). After completing the calculation of the set of outputs in the data storage device (101), the controller (151) of the data storage device (101) may transmit a response to the one or more write commands to the one or more processors (133). The response may include an indication that the set of outputs can be retrieved by the one or more processors (133).
[0211] In response to the indication, one or more processors (133) may optionally retrieve the set of outputs from the data storage device (101) by transmitting a read command to the data storage device (101) for the set of outputs (e.g., after determining to retrieve the set of inputs from the data storage device (101) for processing).
[0212] Alternatively or in combination, the one or more processors (133) of the vehicle (111) may determine whether to store the set of outputs in a non-volatile memory of the data storage device. In response to determining to store the set of outputs in a non-volatile memory of the data storage device (101), the one or more processors (133) of the vehicle (111) may transmit a write command to the data storage device (101).
[0213] Because the set of outputs is initially generated in the data storage device (101) and then buffered in a buffer (152) (e.g., a volatile memory), the data storage device (101) may execute a write command to store the set of outputs in the output namespace (187) without transmitting the set of outputs to and / or receiving the set of outputs from the one or more processors (133) in response to the write command.
[0214] For example, after receiving another set of sensor data (121) from at least one sensor (103) configured on a vehicle (111), one or more processors (133) of the vehicle (111) generate another set of inputs to the artificial neural network (125) based on the another set of sensor data.
[0215] The one or more processors (133) transmit another command to write another set of inputs to the data storage device (101); and the data storage device (101) generates another set of outputs using the model data (183) of the artificial neural network (125) and the another set of inputs. After receiving a response to the another command to write the another set of inputs, the one or more processors (133) may determine to skip processing of the another set of outputs and transmit a subsequent write command to store the another set of outputs to the data storage device (101). In response, the data storage device (101) may write the another set of outputs buffered within the data storage device (101) to the output namespace (187) without transmitting the another set of outputs from the one or more processors (133) of the vehicle (111) to the data storage device (101), and / or without transmitting the another set of outputs from the data storage device (101) to the one or more processors (133) of the vehicle (111).
[0216] The server ( 119 ), computer system ( 131 ), and / or data storage device ( 101 ) may each be implemented as one or more data processing systems.
[0217] The present invention includes methods and apparatus for performing the methods described above, including a data processing system for performing these methods, and a computer-readable medium containing instructions which, when executed on a data processing system, cause the system to perform these methods.
[0218] A typical data processing system may include an interconnect (eg, a bus and system core logic) that interconnects a microprocessor and memory. A microprocessor is typically coupled to a cache memory.
[0219] The interconnect interconnects the microprocessor and memory together, and also interconnects them to input / output (I / O) devices via I / O controllers. The I / O devices may include display devices and / or peripheral devices such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some I / O devices, such as a printer, scanner, mouse, and / or keyboard, are optional.
[0220] The interconnect may include one or more buses connected to each other through various bridges, controllers and / or adapters. In one embodiment, the I / O controller includes a USB (Universal Serial Bus) adapter for controlling USB peripheral devices, and / or an IEEE-1394 bus adapter for controlling IEEE-1394 peripheral devices.
[0221] The memory may include one or more of: ROM (read only memory), volatile RAM (random access memory), and non-volatile memory such as a hard drive, flash memory, and the like.
[0222] Volatile RAM is typically implemented as dynamic RAM (DRAM), which continuously requires power in order to refresh or maintain the data in the memory. Nonvolatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other types of storage systems that maintain data even after power is removed from the system. Nonvolatile memory can also be random access memory.
[0223] The non-volatile memory may be a local device directly coupled to the rest of the components in the data processing system. Non-volatile memory remote from the system may also be used, such as a network storage device coupled to the data processing system through a network interface such as a modem or Ethernet interface.
[0224] In the present invention, some functions and operations are described as being executed by software codes or caused by software codes to simplify the description. However, such expressions are also used to specify that the functions are generated by a processor such as a microprocessor executing code / instructions.
[0225] Alternatively or in combination, dedicated circuitry with or without software instructions may be used to implement the functions and operations described herein, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments may be implemented using hardwired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any specific source for the instructions executed by the data processing system.
[0226] Although one embodiment may be implemented on fully functional computers and computer systems, various embodiments may be distributed as a computer product in various forms and may be used regardless of the specific type of machine or computer readable media used to actually effect the distribution.
[0227] At least some aspects disclosed may be embodied at least in part in software. That is, the techniques may be implemented in a computer system or other data processing system in response to its processor (e.g., a microprocessor) executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).
[0228] The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions called a "computer program". A computer program typically includes one or more instructions that are located at different times in various memories and storage devices in a computer, and when the instructions are read and executed by one or more processors in the computer, causes the computer to perform the operations necessary to perform the elements involved in the various aspects.
[0229] Machine-readable media can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various locations, including, for example, ROM, volatile RAM, non-volatile memory and / or cache. Portions of this software and / or data can be stored in any of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or in the same communication session. Data and instructions can be obtained as a whole before executing an application. Alternatively, portions of data and instructions can be obtained dynamically and in a timely manner when execution requires. Therefore, at a specific time instance, data and instructions do not need to be entirely located on machine-readable media.
[0230] Examples of computer-readable media include, but are not limited to, non-transitory, recordable and non-recordable types of media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy disks and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CDROM), digital versatile disk (DVD), etc.), etc. Computer-readable media may store instructions.
[0231] The instructions may also be embodied in digital and analog communication links using electrical, optical, acoustic or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). However, propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) are not tangible machine-readable media and are not configured to store instructions.
[0232] Generally speaking, a machine-readable medium includes any mechanism that provides (ie, stores and / or transmits) information in a form accessible by a machine (eg, a computer, a network device, a personal digital assistant, a manufacturing tool, any device with a set of one or more processors, etc.).
[0233] In various embodiments, hardwired circuitry may be used in combination with software instructions to implement the technology. Thus, the technology is neither limited to any specific combination of hardware circuitry and software nor to any specific source for the instructions executed by the data processing system.
[0234] The above description and drawings are illustrative and should not be construed as limiting. Numerous specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described to avoid making the description unclear. References to one or an embodiment in the present invention are not necessarily references to the same embodiment; and such references mean at least one.
[0235] In the foregoing specification, the invention has been described with reference to specific exemplary embodiments of the invention. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. Therefore, the description and drawings should be regarded as having an illustrative rather than a restrictive sense.
Claims
1. A method for communicating with a data storage device, wherein include: In the data storage device having a host interface and a storage capacity divided into at least a first namespace, a second namespace and a third namespace, storing model data of the artificial neural network in the first namespace; receiving a set of sensor data from at least one sensor disposed on the vehicle; generating, by one or more processors of the vehicle, a set of inputs to the artificial neural network based on the sensor data; writing, by the one or more processors connected to the host interface of the data storage device, the set of inputs to the second namespace in the data storage device, wherein the data storage device has a controller and a neural network accelerator, the neural network accelerator configured to perform computations using the model data of the artificial neural network stored in the first namespace in the data storage device and the set of inputs written to the second namespace via the host interface in response to the set of inputs being written to the second namespace to produce a set of outputs in the third namespace; and The set of outputs is retrieved from the third namespace of the data storage device via the host interface by the one or more processors of the vehicle.
2. A method according to claim 1, wherein the data storage device uses at least a portion of the model data stored in the data storage device to generate the set of outputs without transmitting the portion of the model data to the one or more processors between receiving the set of inputs and completing the calculation of the set of outputs.
3. The method of claim 2, wherein the portion of the model data comprises static properties of neurons in the artificial neural network.
4. The method of claim 3, wherein the portion of the model data further comprises neuron connectivity data of the artificial neural network.
5. The method of claim 2, wherein writing the set of inputs to the data storage device comprises transmitting one or more write commands from the one or more processors of the vehicle to the host interface to store the set of inputs in the second namespace in the data storage device; and the method further include: After the completion of the computation of the set of outputs in the data storage device, a response to the one or more write commands is received from the data storage device. 6 . The method of claim 5 , wherein the retrieval of the set of outputs from the data storage device comprises transmitting a read command to the data storage device for the set of outputs.
7. The method according to claim 6, further comprising: include: A determination is made, by the one or more processors of the vehicle, whether to retrieve the set of outputs from the data storage device, wherein the transmission of the read command is in response to a determination to retrieve the set of outputs from the data storage device for processing.
8. The method according to claim 6, further comprising: include: determining, by the one or more processors of the vehicle, whether to store the set of outputs in a non-volatile memory of the data storage device; and Responsive to a determination to store the set of outputs in the nonvolatile memory of the data storage device, a write command is transmitted to the data storage device.
9. The method according to claim 6, further comprising: include: receiving another set of sensor data from at least one sensor disposed on the vehicle; generating, by the one or more processors of the vehicle, another set of inputs to the artificial neural network based on the another set of sensor data; transmitting a command to write the another set of inputs to the data storage device, wherein the data storage device is configured to generate another set of outputs using the model data of the artificial neural network and the another set of inputs; receiving a response to the command to write the another set of inputs; and transmitting, from the one or more processors and in response to the response to the command to write the another set of inputs, a write command to store the another set of outputs; wherein said data storage device is buffered in said another set of outputs within said data storage device; and wherein the write command is executed without transmitting the another set of outputs from the one or more processors of the vehicle to the data storage device.
10. The method of claim 9, wherein the write command is executed in the data storage device without the another set of outputs being transmitted from the data storage device to the one or more processors of the vehicle.
11. A means of transport, wherein include: at least one processor; at least one sensor configured to generate a sensor data stream; and a data storage device having a controller, a neural network accelerator, a host interface connected to the at least one processor, and storage capacity divided into at least a first namespace configured to store model data of an artificial neural network, a second namespace configured to store inputs to the artificial neural network, and a third namespace configured to store outputs from the artificial neural network; wherein the at least one processor is configured to receive a set of sensor data from the at least one sensor, generate a set of inputs to the artificial neural network based on the sensor data, and write the set of inputs to the second namespace in the data storage device via the host interface; wherein the controller and the neural network accelerator in the data storage device are configured to perform computations using the model data of the artificial neural network in the first namespace stored in the data storage device and the set of inputs written to the second namespace via the host interface in response to the set of inputs being written into the second namespace to generate a set of outputs in the third namespace; and Wherein the at least one processor is configured to read the set of outputs from the third namespace of the data storage device via the host interface.
12. A vehicle according to claim 11, wherein the data storage device uses at least a portion of the model data stored in the data storage device to generate the set of outputs without transmitting the portion of the model data to the processor between receiving the set of inputs and completion of calculation of the set of outputs.
13. The vehicle of claim 12, wherein the portion of the model data comprises static properties of neurons in the artificial neural network and neuron connectivity data of the artificial neural network.
14. The vehicle of claim 12, wherein the at least one processor receives an indication of availability of the set of outputs in the data storage device from the data storage device after completion of the calculation of the set of outputs.
15. The vehicle of claim 14, wherein the at least one processor is configured to selectively request the set of outputs and selectively request the data storage device to store the set of outputs in non-volatile memory.
16. A data storage device for a vehicle, include: a host interface configured to receive a set of inputs to the artificial neuron from at least one processor of the vehicle, the inputs being calculated based on sensor data generated by at least one sensor configured on the vehicle; a non-volatile memory configured to store model data of the artificial neural network and to store the set of inputs; a neural network accelerator configured to compute a set of outputs using the model data of the artificial neural network stored in the non-volatile memory and the set of inputs received in the host interface; and A buffer is used to store the set of outputs for access by the at least one processor via the host interface.
17. A data storage device according to claim 16, wherein the data storage device uses at least a portion of the model data stored in the data storage device to generate the set of outputs without transmitting the portion of the model data to the processor between receiving the set of inputs and completing the calculation of the set of outputs.
18. The data storage device of claim 17, wherein the portion of the model data comprises static properties of neurons in the artificial neural network and neuron connectivity data of the artificial neural network.
19. The data storage device according to claim 17, further comprising: include: A controller wherein after completing computation of the set of outputs, the controller is configured to generate a response to a request from the at least one processor to store the set of inputs in the non-volatile memory, the response indicating availability of the set of outputs in the data storage device.
20. A data storage device according to claim 19, wherein the host interface is configured to receive a first command from the at least one processor to transmit the set of outputs to the at least one processor, or a second command to store the set of outputs in the non-volatile memory; wherein the controller is configured to use the set of outputs in the buffer to execute the first command or the second command without communicating the set of outputs from the at least one processor.
Citation Information
Patent Citations
Electronic control apparatus for electrically-driven vehicle
US9533579B2
Neural network computing apparatus and system, and method therefor
CN104145281A
Calculation apparatus and method for accelerator chip accelerating deep neural network algorithm
CN105488565A
Neural network processor based on weight compression, design method, and chip
CN106529670A
Collaborative system design of general processor and neural network processor
CN107657316A
