System, method and apparatus for controlling delivery of audio content into a vehicle cabin

Abstract

The present invention provides systems, methods and apparatus for controlling the delivery of audio content to a cabin of a vehicle in which occupants are located / seated. For example, a vehicle includes: at least one microphone configured to generate a signal representing audio content presented in the cabin of the vehicle; an infotainment system that accesses multiple audio content sources; and an artificial neural network configured to receive input parameters related to audio control in the vehicle and generate predictions of audio patterns based on the input parameters that vary over time. The input parameters may include data representing the signal from the at least one microphone, data from the infotainment system, and / or at least one operating parameter of the vehicle. The vehicle is configured to adjust settings of the infotainment system based at least in part on the predictions generated by the artificial neural network.

Description

Technical Field

[0001] At least some embodiments disclosed herein relate generally to vehicles, and more particularly, but not limited to, to smart audio control in a vehicle. Background Art

[0002] Recent developments in the field of autonomous driving technology allow computing systems to operate control elements of a motor vehicle without the assistance of a human operator of the vehicle, at least under some conditions.

[0003] For example, sensors (e.g., cameras and radars) may be mounted on a motor vehicle to detect the vehicle's surroundings as it travels on a roadway. A computing system mounted on the vehicle analyzes the sensor inputs to identify conditions and generates control signals or commands for autonomous adjustments to the vehicle's direction and / or speed, with or without any input from a human operator of the vehicle.

[0004] In some arrangements, when the computing system identifies a situation in which the computing system may be unable to continue operating the vehicle in a safe manner, the computing system alerts the human operator of the vehicle and requests the human operator to take over control of the vehicle and drive manually instead of allowing the computing system to autonomously drive the vehicle.

[0005] Autonomous driving and / or advanced driver assistance systems (ADAS) often involve artificial neural networks (ANNs) for recognizing events and / or objects captured in sensor inputs.

[0006] Generally speaking, an artificial neural network (ANN) uses a network of neurons to process inputs to the network and produce outputs from the network.

[0007] For example, each neuron in the network receives a set of inputs. Some of the inputs to the neurons may be outputs of certain neurons in the network; and some of the inputs to the neurons may be inputs provided to the neural network. The input / output relationships among neurons in the network represent the neuron connectivity in the network.

[0008] For example, each neuron may have a bias, an activation function, and a set of synaptic weights for its inputs, respectively. The activation function may be in the form of a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network may have different activation functions.

[0009] For example, each neuron may produce a weighted sum of its input and its bias and then produce an output that is a function of the weighted sum, computed using the neuron's activation function.

[0010] The relationship between the input and output of an ANN is generally defined by an ANN model, which contains data representing the connectivity of neurons in the network, as well as the bias, activation function, and synaptic weights of each neuron. Using a given ANN model, a computing device calculates the output of the network from a given set of inputs sent to the network.

[0011] For example, input to the ANN network may be generated based on camera input; and output from the ANN network may be recognition of items such as events or objects.

[0012] Spiking neural networks (SNNs) are a type of ANN that closely simulates natural neural networks. When the activation level of a neuron is high enough, the SNN neuron generates a pulse as an output. The activation level of an SNN neuron simulates the membrane potential of a natural neuron. The output / pulse of an SNN neuron can change the activation level of other neurons that receive the output. The current activation level of an SNN neuron over time is typically modeled using differential equations and is considered to be the state of the SNN neuron. Incoming pulses from other neurons can push the activation level of a neuron higher to reach the threshold of the pulse. Once a neuron generates a pulse, its activation level is reset. Before a pulse is generated, the activation level of an SNN neuron may decay over time, as controlled by a differential equation. The temporal element in the behavior of SNN neurons makes SNNs suitable for processing spatiotemporal data. The connectivity of SNNs is typically sparse, which helps reduce computational workload.

[0013] Generally speaking, ANNs can be trained using supervised methods in which parameters in the ANN are adjusted to cause the error between the known outputs produced by the corresponding inputs and the calculated outputs produced from applying the inputs to the ANN. Examples of supervised learning / training methods include reinforcement learning, and learning with error correction.

[0014] Alternatively or in combination, the ANN may be trained using unsupervised methods in which the exact output produced by a given set of inputs is not known until training is complete.The ANN may be trained to classify items into categories, or to classify data points into clusters.

[0015] Multiple training algorithms can be used for complex machine learning / training paradigms.

[0016] U.S. Patent No. 9,533,579, entitled “Electronic Control Apparatus for Electrically-Driven Vehicle” and published on January 3, 2017, discloses an electronic control apparatus for a vehicle having a self-diagnostic function.

[0017] Conventionally, automobile maintenance is scheduled based on predetermined operating milestones. For example, a routine maintenance service may be scheduled every three or six months, or after a predetermined distance (e.g., 3,000 miles, 6,000 miles, or 15,000 miles) has been traveled.

[0018] Such incidents can be safety threatening when components of a motor vehicle malfunction or fail during vehicle operation. After such incidents occur, it is important to schedule a trip to obtain vehicle service as soon as possible, even at an inconvenient time. Summary of the invention

[0019] In one aspect, the present disclosure relates to a vehicle comprising: at least one microphone configured to generate a signal representative of audio content presented in a cabin of the vehicle; an infotainment system that interfaces with multiple sources of audio content; and an artificial neural network configured to receive input parameters related to audio controls in the vehicle and generate predictions of audio patterns based on the input parameters as they vary over time, wherein the input parameters include data representative of the signal from the at least one microphone, and at least one operating parameter of the vehicle; wherein the vehicle is configured to adjust settings of the infotainment system based at least in part on the predictions generated by the artificial neural network.

[0020] In another aspect, the present disclosure relates to a method comprising: generating, by at least one microphone configured in a vehicle, a signal representing audio content presented in a cabin of the vehicle, wherein an infotainment system accesses multiple audio content sources; receiving, in an artificial neural network, input parameters related to audio control in the vehicle; generating, by the artificial neural network, predictions of audio patterns based on the input parameters that vary over time; and adjusting, by the vehicle, settings of the infotainment system based at least in part on the predictions generated by the artificial neural network.

[0021] In yet another aspect, the present disclosure relates to an infotainment system comprising: at least one microphone configured to generate a signal representing audio content presented in a cabin of a vehicle having multiple audio content sources; and an artificial neural network configured to receive input parameters related to audio controls in the vehicle and generate predictions of audio patterns based on the input parameters as they vary over time, wherein the input parameters include data representing the signal from the at least one microphone, and at least one operating parameter of the vehicle; wherein the infotainment system is configured to adjust settings to change the audio content presented in the cabin of the vehicle based at least in part on the predictions generated by the artificial neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Embodiments are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0023] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.

[0024] Figure 2 An autonomous vehicle with a data storage device is shown according to one embodiment.

[0025] Figure 3-5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.

[0026] Figure 6 A method of predictive maintenance according to one embodiment is shown.

[0027] Figure 7 A data storage device for accelerating neural network computations according to one embodiment is shown.

[0028] Figure 8 A storage media assembly for accelerating neural network computations is shown according to one embodiment.

[0029] Fig. 9 A method of accelerating neural network computations in a motor vehicle is shown according to one embodiment.

[0030] Fig.10 A data storage device configured to support neural network computations is shown according to one embodiment.

[0031] Fig.11 The configuration of a namespace for an artificial neural network (ANN) model according to one embodiment is described.

[0032] Fig.12 The configuration of a namespace for input sent to an artificial neuron according to one embodiment is described.

[0033] Fig.13 The configuration of a namespace for outputs from an artificial neuron according to one embodiment is described.

[0034] Figure 14-16 A method of predictive maintenance supported by model partitioning, input partitioning, and output partitioning according to one embodiment is shown.

[0035] Fig.17 Communication with a data storage device to perform neural network calculations is shown according to one embodiment.

[0036] Fig.18 Communication with a data storage device to perform neural network calculations is shown according to one embodiment.

[0037] Fig.19 A method of communicating with a data storage device to perform neural network calculations according to one embodiment is shown.

[0038] Fig. 20 A system for selectively presenting audio content in a vehicle is shown according to one embodiment.

[0039] Fig.21 A method of controlling presentation of audio content in a vehicle according to one embodiment is shown. DETAILED DESCRIPTION

[0040] At least some embodiments disclosed herein provide systems, methods, and apparatus for processing sensor data generated in a motor vehicle, or another vehicle with or without an advanced driver assistance system (ADAS).

[0041] Before a component of a motor vehicle fails or malfunctions during operation of the vehicle, there may be an indication whether the component requires replacement or maintenance. Such an indication may not be noticeable to a typical driver or passenger. However, sensor data may be collected and analyzed to predict the probability of component failure. The prediction may be used to schedule maintenance service, which may reduce or eliminate the chance of an incident where a component of the vehicle fails or malfunctions during operation of the vehicle on the roadway. Furthermore, the prediction allows a service trip to be scheduled at a convenient time.

[0042] For example, sensors may be installed in an automobile system to collect data during its routine operation; and the sensor data may be used to predict whether a component needs to be replaced or maintained and how soon it will be needed. The sensor data may be provided as input to an artificial neural network (ANN) (e.g., a spiking neural network (SNN)) sent to an artificial intelligence (AI) system to train itself (e.g., using unsupervised machine learning techniques) over a period of time in which the vehicle is expected to operate normally. The training is customized to the neural network for the specific operating environment of the driver, passenger, or user of the vehicle and the personalized operating habits of the vehicle's passengers. Subsequently, the artificial neural network may detect abnormal conditions when the operating data deviates from the normal pattern. The AI ​​system may be used to recommend maintenance services and / or identify components that may need to be replaced or maintained.

[0043] Figure 1 A system is shown in which a vehicle is configured with a data storage device to collect and process sensor data according to some embodiments.

[0044] Figure 1The system includes a vehicle 111 having a data storage device 101. Optionally, the vehicle 111 has an advanced driver assistance system (ADAS) 105 and one or more sensors 103 that provide sensor data input to the ADAS 105 and / or the data storage device 101. The data storage device 101 is configured to use an artificial neural network (ANN) 125 to predict / identify the need for maintenance service based on the data collected by the sensor 103. The ADAS 105 can be omitted without affecting the predictive maintenance feature. In some embodiments, at least a portion of the data generated by the sensor 103 is in the ADAS 105 for driving assistance and in the ANN 125 for maintenance prediction. Optionally, the output of the ANN 124 can be used in the data storage device 101 and in the ADAS 105.

[0045] Sensors 103 may include digital cameras, lidars, radars, ultrasonic sonars, brake sensors, speed sensors, acceleration sensors, airbag sensors, global positioning system (GPS) receivers, audio sensors / microphones, vibration sensors, force / strain sensors, deformation sensors, motion sensors, temperature sensors, etc. Some of sensors 103 may be configured primarily to monitor the environment of vehicle 111; and other sensors 103 may be configured primarily to monitor the operating conditions of one or more components of vehicle 111 (e.g., internal combustion engine, exhaust system, electric motor, brake pads, tires, battery, etc.).

[0046] The time-varying output of the sensors 103 is provided as a sensor data stream to the ADAS 105 and / or the ANN 125 to provide driving assistance (eg, autonomous driving) and maintenance prediction.

[0047] For example, the vehicle 111 may have a wireless communication device that communicates with a remote server 119 via wireless signals 113 and a communication network 117. The remote server 119 is typically configured at a location away from the road 102 on which the vehicle 111 is traveling. For example, the vehicle 111 may provide some sensor data 121 to the server 119 and receive updates of the ANN 125 from the server 119.

[0048] One example of a communication network 117 is a cellular telephone network having one or more base stations (e.g., 115) that receive wireless signals (e.g., 113). Another example of a communication network 117 is the Internet, where wireless local area network signals (e.g., 113) transmitted by the vehicle 113 are received in an access point (e.g., 115) for further transmission to a server 119. In some embodiments, the vehicle 111 uses a communication link 107 to a satellite 109 or a communication balloon that communicates with the server 119.

[0049] The server 119 may also communicate with one or more maintenance service facilities (e.g., 127) to receive maintenance service data 123 of a vehicle (e.g., 111). The maintenance service data 123 may include inspection records and / or service records of components of the vehicle (e.g., 111). For example, the inspection records and / or service records may indicate the degree of wear of the components inspected at the maintenance service facility (e.g., 127) during the service of the components, the identification of faulty or malfunctioning components, etc. The sensor data 121 and maintenance service data 123 of the vehicle (e.g., 111) in the time period before the service may be used to train the ANN 125 to predict the probability that the component needs maintenance service. The updated ANN 125 may be used to predict and recommend maintenance service for the vehicle 111 based on the sensor data 121 received in the most recent time period. Alternatively, the updated ANN 125 may be transmitted to the vehicle 111; and the vehicle 111 may use the data generated from the sensor 103 during the routine operation of the vehicle 111 to predict and recommend maintenance service.

[0050] The data storage device 101 of the vehicle 111 may be configured to record sensor data over a time period used in the ANN for predictive maintenance. Maintenance predictions are typically for relatively long time periods (e.g., days, weeks, and / or months). In contrast, sensor data recorded for reviewing an accident, collision, or near-collision involving an autonomous vehicle is typically for a short time period (e.g., 30 seconds to minutes). Therefore, a typical black box data recorder configured to record sensor data for reviewing / analyzing an accident or collision is insufficient for predictive maintenance.

[0051] Optionally, the data storage device 101 stores sensor data about time periods that cause trips to maintenance service facilities (e.g., 127). The maintenance service facility (e.g., 127) can download the sensor data 121 from the data storage device 101 and provide the sensor data 121 and corresponding maintenance service data 123 to the server 119 to facilitate training of the ANN 125.

[0052] Optionally, or in combination, the data storage device 101 is configured with a machine learning module to customize and / or train the ANN 125 installed in the vehicle 111 for predictive maintenance.

[0053] For example, the machine learning module of the data storage device 101 may be used to calibrate the ANN 125 to take into account the typical / daily environment in which the vehicle 111 is operated and / or the driving preferences / habits of the driver of the vehicle 111 .

[0054] For example, during a time period when the vehicle is expected to operate in a typical / daily environment with healthy components, the sensor data generated by the sensors 103 can be used to train the ANN 125 to recognize patterns of sensor data that represent trouble-free operation. Such patterns may be different for different vehicles (e.g., 111) based on their routine operating environments and the driving habits / characteristics of their drivers. The training allows the ANN 125 to detect deviations from the recognized normal patterns and report anomalies for maintenance predictions.

[0055] For example, the ANN 125 may include an SNN configured to classify time-based changes in sensor data and / or detect deviations from known sensor data patterns for the vehicle 111 operating under normal / healthy conditions but in a personalized environment (e.g., a driver / passenger's daily route) and / or under personalized driving habits / patterns.

[0056] Figure 2 An autonomous vehicle 111 with a data storage device 101 is shown according to one embodiment. For example, Figure 2 111 Implementation of Autonomous Vehicles Figure 1 Vehicle 111 in the system.

[0057] Figure 2 The vehicle 111 is configured with an advanced driver assistance system (ADAS) 105. The ADAS 105 of the vehicle 111 may have an artificial neural network (ANN) 125 for object detection, recognition, identification, and / or classification. The ANN 125 and / or another neural network (e.g., configured in the data storage device 101) may be used to predict the probability that a component of the vehicle 111 requires maintenance service (e.g., repair, replacement, or adjustment).

[0058] Preferably, the data storage device 101 is configured to process the sensor data, at least in part, for predictive maintenance, placing a reduced computational burden on the processor 133 responsible for operating the ADAS 105 and / or other components (eg, the infotainment system 149 ).

[0059] The vehicle 111 typically includes an infotainment system 149, a communication device 139, one or more sensors 103, and a computer system 131 connected to some controls of the vehicle 111 (e.g., a steering control 141 for the direction of the vehicle 111, a brake control 143 for stopping the vehicle 111, an acceleration control 145 for the speed of the vehicle 111, etc.). In some embodiments, Figure 1 The vehicle 111 in the system has a similar configuration and / or similar components.

[0060] Some of the sensors 103 are required for operation of the ADAS 105; and some of the sensors 103 are used to collect data related to the health of components of the vehicle 111, which may not be used in the ADAS 105. Optionally, the sensor data generated by the sensors 103 can also be used to predict the likelihood of an imminent failure of a component. Such predictions can be used in the ADAS 105 to take emergency actions to put the vehicle in a safe state (e.g., by reducing speed and / or stopping).

[0061] The computer system 131 of the vehicle 111 includes one or more processors 133 , the data storage device 101 , and a memory 135 storing firmware (or software) 147 (including computer instructions and data models for the ADAS 105 ).

[0062] The one or more sensors 103 of the vehicle may include a visible light camera, an infrared camera, a lidar, a radar or sonar system, a peripheral device sensor, a global positioning system (GPS) receiver, a satellite positioning system receiver, a brake sensor, and / or an airbag sensor. In addition, the sensors 103 may include an audio sensor (e.g., a microphone) configured to monitor noise from various components and locations in the vehicle 111, a vibration sensor, a pressure sensor, a force sensor, a strain sensor, and / or a deformation sensor configured to measure the load on the components of the vehicle 111, an accelerometer and / or a gyroscope sensor that measures the movement of some components of the vehicle 111, etc. Such sensors can be used to monitor the operating state and / or health of the components for predictive maintenance.

[0063] The sensor 103 may provide a real-time sensor data stream to the computer system 131. The sensor data generated by the sensor 103 of the vehicle 111 may include an image of an object captured using a camera, or a sonar, radar, or lidar system, wherein the camera is a camera that captures the image using light visible to the human eye, or a camera that captures the image using infrared light. The image data obtained from at least one sensor of the vehicle is part of the collected sensor data recorded in the data storage device 101 and / or as an input to the ANN 125. For example, the camera can be used to obtain roadway information for the vehicle 111 to travel, and the ANN 125 can process the roadway information to generate control signals for the vehicle 111. For example, the camera can be used to monitor the operating status / health of the components of the vehicle 111, and the ANN 125 can process the operating status / health to predict or schedule maintenance services.

[0064] The sensor data generated by the sensors 103 of the vehicle 111 may include an audio stream that captures characteristics of sounds at locations on the vehicle 111 (e.g., locations near the engine, motor, transmission system, wheels, doors, windows, etc.). The audio data obtained from at least one sensor 103 of the vehicle 111 may be part of the collected sensor data recorded in the data storage device 101 and / or as input to the ANN 125. For example, the audio stream may be used to monitor the operating status / health of components of the vehicle 111 (e.g., internal combustion engine, exhaust system, electric motor, brakes), which the ANN 125 may process to predict or schedule maintenance service.

[0065] The infotainment system 149 may be used to present the predicted or scheduled maintenance service. Optionally, the communication device 139 may establish a connection to a mobile device of the driver of the vehicle 111 to notify the driver of the recommended maintenance service and / or recommended service data, schedule the appointment, etc.

[0066] When the vehicle 111 is configured with the ADAS 105 , the output of the ADAS 105 may be used to control (eg, 141 , 143 , 145 ) the acceleration of the vehicle 111 , the speed of the vehicle 111 , and / or the direction of the vehicle 111 during autonomous driving.

[0067] Figure 3-5 The training of an artificial neural network for maintenance service prediction according to some embodiments is described.

[0068] exist Figure 3 In the example, the supervised machine learning module 171 is used to train the artificial neural network 125 to minimize the difference between the service prediction 129 generated from the sensor data 121 and the maintenance service data 123.

[0069] For example, the maintenance service data 123 may identify measured wear of a component over time to predict when a recommended service will be performed. The sensor data 121 may be used in the ANN 125 to generate a predicted time when the recommended service will be performed. The supervised machine learning module 171 may adjust the artificial neural network 125 to reduce / minimize the difference between the time predicted based on the sensor data 121 and the time calculated from the wear measurements.

[0070] For example, maintenance service data 123 may identify a component that was replaced or repaired in maintenance service facility 127. Sensor data 121 recorded during a period of time before the component was replaced or repaired may be used to calculate the time when the replacement or repair will be performed. In addition, segments of the sensor data stream during the period of time before the replacement or repair may be used in ANN 125 to generate a prediction of the time when the replacement or repair will be performed. Supervised learning 171 may be used to adjust ANN 125 to reduce the predicted time when the replacement or repair will be performed and the actual time when the replacement or repair is performed.

[0071] Figure 2 The supervised learning 171 may be applied in the server 119 to generate a general ANN for the vehicle population based on the sensor data of the vehicle population and its maintenance service data 123 .

[0072] Figure 2 The supervised learning 171 can be applied in the vehicle 111 to generate a customized / personalized ANN for a group of vehicles based on the vehicle's sensor data and its maintenance service data 123. For example, a general ANN can be initially used in the vehicle 111; and the sensor data of the vehicle 111 and its maintenance service data 123 can be used to further train the vehicle's ANN 125 for customization / personalization of the ANN 125 in the vehicle 111.

[0073] exist Figure 4 In the example of FIG. 1 , an unsupervised machine learning module 175 is used to train or optimize the artificial neural network 125 to facilitate anomaly detection 173. The unsupervised machine learning module 175 is configured to adjust the ANN (e.g., SNN) for classifications, clusters, or recognized patterns in the sensor data 121 so that the degree of deviation from the classifications, clusters, or recognized patterns in the sensor data 121 generated within a recent time period can be used to signal the detection 173 of an anomaly. The anomaly detection 173 allows the vehicle 111 to be scheduled for inspection at the maintenance service facility 127. Optionally, after the inspection, such as in Figure 3 In the example embodiment, the maintenance service data 123 may be used to apply supervised learning 171 to produce more accurate predictions of service.

[0074] Generally, it may be assumed that the vehicle 111 is operating in normal / healthy conditions for a certain period of time. For example, after a new vehicle 111 is initially put into service, it may be assumed that the vehicle 111 provides trouble-free service for at least a period of time (e.g., several months). For example, after a period of time after a component is replaced or repaired, it may be assumed that the component provides trouble-free service for at least a period of time (e.g., several months or a year). Therefore, the sensor data 121 obtained during this period of time may be pre-classified as "normal" for use in the following examples. Figure 4 Unsupervised learning in 175 or Figure 5Supervised learning in 171 training ANN 125.

[0075] For example, sensor data 121 collected during a "normal" service period of a vehicle 111 or component may be classified into several clusters via unsupervised learning 175. Different clusters may correspond to different types of normal conditions (e.g., driving on different routes, driving on roads with different road conditions, driving in different weather conditions, driving at different times of the day, driving on different days of the week, driving under different emotions of the driver's driving habits). When continuous sensor data 121 is classified outside of a "normal" cluster, an anomaly is detected.

[0076] Optionally, as Figure 5 , supervised machine learning 171 may be used to train the ANN 125. During a "normal" service cycle of a vehicle 111 or component, an expected classification 177 may be used to label the sensor data 121. Supervised learning 171 may be used to minimize the classification difference between a prediction 179 made from the sensor data 121 using the ANN 125 and the expected classification 177. In addition, when the sensor data 121 is known to be "abnormal" (e.g., after a diagnosis made in a maintenance service facility 127 or by a user, driver, or passenger of the vehicle 111), the expected classification 177 may be changed to "abnormal" for further training the ANN 125 to directly recognize abnormalities (e.g., instead of relying on deviations from a known "normal" cluster to infer abnormalities).

[0077] Thus, the ANN 125 may be trained to identify anomalous sensor data and schedule maintenance service by estimating the severity of the anomaly.

[0078] Figure 6 A method of predictive maintenance according to one embodiment is shown. For example, Figure 6 The method can be implemented in Figure 1 or 2 in the data storage device 101 in the vehicle 111 or Figure 2 A computer system 131 in the vehicle 111.

[0079] At box 201 , sensors (eg, 103 ) installed in a vehicle 111 generate a sensor data stream (eg, 121 ) during operation of the vehicle 111 on a road 102 .

[0080] At block 203, the sensor data stream (eg, 121) is provided to an artificial neural network (ANN) 125. For example, the ANN 125 may include a spiking neural network (SNN).

[0081] At block 205 , an artificial neural network (ANN) 125 generates a prediction of maintenance service based on the sensor data stream (eg, 121 ).

[0082] At block 207 , the data storage device 101 disposed on the vehicle stores at least a portion of the sensor data stream (eg, 121 ).

[0083] At box 209 , an artificial neural network (ANN) is trained using sensor data streams (eg, 121 ) collected over a predetermined period of time since the vehicle left the factory or maintenance service facility 127 .

[0084] For example, an artificial neural network (ANN) may be configured to identify components of vehicle 111 that require repair or replacement during maintenance service and / or identify a predicted time period until a component fails or malfunctions, or a suggested time period until a recommended maintenance service for a component prior to a component failure or malfunction. Thus, the performance of the predicted maintenance service may avoid an incident in which a component fails or malfunctions while vehicle 111 is operating on road 102.

[0085] For example, the sensor 103 may be a microphone mounted near the component, a vibration sensor attached to the component, a pressure sensor mounted in the component, a force or stress sensor mounted to or attached to the component, a deformation sensor attached to the component, or an accelerometer configured to measure a motion parameter of the component.

[0086] Optionally, the data storage device 101, the vehicle 111 of the computer system 131 and / or a server 119 remote from the vehicle may have a machine learning module that is configured to train an artificial neural network (ANN) 125 during a time period in which the vehicle 111 is assumed to be in a healthy state (e.g., a predetermined time period from when the vehicle 111 leaves a factory or maintenance service facility 127).

[0087] For example, if Figure 4 As described in , the machine learning module can use unsupervised machine learning 175 to train the ANN 125 to recognize / classify normal patterns of sensor data 121 and thus have the ability to detect anomalies based on deviations from normal patterns. Figure 3 As described in or 5, supervised machine learning 171 can be used.

[0088] For example, unsupervised machine learning 175 may be applied by the data storage device 101 or computer system 131 of the vehicle 111 during a predetermined time period during which the vehicle and / or components are known to operate without impairment or degradation.

[0089] Alternatively or in combination, some of the sensor data 121 stored in the data storage device 101 of the vehicle 111 may be uploaded to the server 119 for use in training the ANN 125 .

[0090] In at least some embodiments disclosed herein, the data storage device 101 is configured to accelerate computations of an artificial neural network (ANN) 125 of the vehicle 111 .

[0091] For example, in addition to supporting typical operations of data access and storage, the data storage device 101 may be further configured to perform at least a portion of computations involving an artificial neural network (ANN) 125, such as generating predictions (e.g., 129 or 173) or classifications (e.g., 179) from the sensor data 121 and / or performing unsupervised machine learning 175 (e.g., Figure 4 ) and / or supervised machine learning 171 (e.g., as described in Figure 3 or as described in 5) to adjust ANN 125.

[0092] For example, the calculations configured in the data storage device 101 can be used to reduce the amount of data transmitted to the processor 133 for use or application by the ANN 125 and / or reduce the computational tasks of the processor 133 when evaluating the output of the ANN 125 and / or when training the ANN 125. Such an arrangement can result in faster output from the data storage device 101 and / or reduced energy usage, because data in and out of the memory does not have to be moved to a dedicated independent neural network accelerator. The computing power of the data storage device 101 in processing data related to the ANN 125 allows the computer system 131 of the motor vehicle 111 to monitor the health of the vehicle components (e.g., in a non-real-time manner, or a pseudo-real-time manner), with reduced impact, or no impact, on the processing of mission-critical tasks (e.g., autonomous driving by the ADAS 105). In addition, the computing power of the data storage device 101 can be used to accelerate the processing of sensor data for the ADAS 105 and thus improve the processing of mission-critical tasks.

[0093] Figure 7 1 shows a data storage device 101 for accelerating neural network calculations according to one embodiment. For example, Figure 7 The data storage device 101 can be used to implement Figure 1 or the data storage device 101 of the vehicle 111 described in 2

[0094] exist Figure 7, the data storage device 101 has a host interface 157 configured to communicate with a processor (e.g., 133). For example, the communication between the processor (e.g., 133) and the host interface 157 may be based at least in part on a communication protocol for a peripheral component interconnect express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a universal serial bus (USB) bus, and / or a storage area network (SAN).

[0095] For example, the host interface 157 may be used to receive sensor data 121 generated by sensors 103 of the vehicle 111 to optionally store a portion of the sensor data 121 in the storage media components 161 - 163 .

[0096] For example, each of the storage media components 161 to 163 may be a memory integrated circuit configured to store data. For example, the media component 161 or 163 may include one or more integrated circuit dies embedded in an integrated circuit package. The integrated circuit die may have a plurality of memory cells formed thereon to store data.

[0097] Generally speaking, some memory integrated circuits are volatile and require power to maintain stored data; and some memory integrated circuits are non-volatile and can retain stored data even when not powered.

[0098] Examples of non-volatile memory include flash memory, memory cells formed based on "NAND" logic gates, "NOR" logic gates, phase change memory (PCM), magnetic memory (MRAM), resistive random access memory, cross-point memory devices, and memory devices. Cross-point memory devices use memory elements with fewer transistors, each of which has a memory cell and a selector stacked together in a column. The memory element columns are connected via two vertical wire layers, one of which is above the memory element columns and the other is below the memory element columns. Each memory element can be selected individually at the intersection of one wire on each of the two layers. Cross-point memory devices are fast and non-volatile, and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM) memory, etc. Examples of volatile memory include dynamic random access memory (DRAM) and static random access memory (SRAM).

[0099] The data storage device 101 may have a controller 151 including a volatile local memory 153 and at least one processing device 155 .

[0100] The local memory of the controller 151 may be an embedded memory configured to store instructions for executing various processes, operations, logic flows, and routines for controlling the operation of the processing device 155, including controlling the communication between the data storage device 101 and the processor (e.g., 133) of the vehicle 111, and other functions described herein. The local memory 151 of the controller 151 may include a read-only memory (ROM) for storing microcode and / or memory registers for storing, for example, memory pointers, extracted data, etc., and / or volatile memory, such as dynamic random access memory (DRAM) and static random access memory (SRAM).

[0101] exist Figure 7 , data storage device 101 includes a neural network accelerator 159 coupled to a controller 151 and / or storage media components 161 to 163.

[0102] For example, the neural network accelerator 159 may be configured to perform matrix arithmetic calculations. The calculations involve the ANN 125 having matrix multiplication and accumulation operations, which may be computationally intensive for a general purpose processor (e.g., 133). Using the neural network accelerator 159 to perform matrix arithmetic calculations may reduce the data to be transmitted to the processor 133 of the vehicle 111 and reduce the computational workload of the processor 133.

[0103] For example, when ANN 125 includes a spiking neural network (SNN), simulation of differential equations used to control activation levels of SNN neurons may be computationally intensive for a general purpose processor (e.g., 133). Neural network accelerator 159 may simulate differential equations using specialized hardware and thus improve the computational efficiency of computer system 131 overall.

[0104] In some embodiments, neural network accelerator 159 is an integrated circuit device separate from controller 151 and / or storage media components 161 to 163. Alternatively or in combination, neural network accelerator 159 is integrated with controller 151 in an integrated circuit package. Figure 8 As illustrated in FIG. 1 , the neural network accelerator 159 may be integrated in at least one of the storage media components 161 to 163 .

[0105] Figure 8 A storage media component 160 for accelerating neural network computations according to one embodiment is shown. For example, Figure 8 The storage media component 160 implements Figure 7 Each or some of the storage media components 161 to 163 in.

[0106] exist Figure 8In the embodiment of the present invention, the storage media component 160 can be housed in an integrated circuit package. The input / output (I / O) interface 171 of the storage media component 160 is configured to process input / output signals in the pins of the integrated circuit package. For example, the input / output signals can include address signals that specify locations in the media unit 175, and data signals representing data written to the media unit 175 at the locations specified by the address signals, or data retrieved from the locations in the media unit 175.

[0107] exist Figure 8 , neural network accelerator 159 is coupled to control logic 173 and / or media unit 175 to perform calculations used in evaluating the output of ANN 125 and / or in training ANN 125.

[0108] For example, the input / output interface 171 may receive an address identifying a matrix stored in a media unit and operated via the neural network accelerator 159. The storage media component 160 may provide the computational results of the neural network accelerator 159 as output data in response to the address, store the output data in a buffer for further operation, and store the output data in a location specified via the address signal in the media unit 175. Thus, the computation performed by the neural network accelerator 159 may be within the storage media component 160, which is close to the media unit 175 in which the matrix data is stored. For example, each of the media units 175 may be an integrated circuit die on which a memory cell of a non-volatile memory is formed.

[0109] For example, state data of SNN neurons may be stored in the media unit 175 according to a predetermined pattern. The neural network accelerator 159 may automatically update the state of the SNN neurons over time according to the differential equations used to control the activation levels of the SNN neurons. Optionally, the neural network accelerator 159 is configured to process pulses of neurons in the neural network. Alternatively, the neural network accelerator 159 and / or the processor 133 of the data storage device 101 may be configured to process pulses of neurons and / or accumulation of inputs sent to the SNN.

[0110] Fig. 9 A method of accelerating neural network computation in a motor vehicle according to one embodiment is shown. For example, Fig. 9 The method can be used Figure 7 The data storage device 101 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig. 9 The method can be used with Figure 6 The methods are used in combination.

[0111] At block 221 , the data storage device 101 of the vehicle 111 receives a sensor data stream from at least one sensor (eg, 103 ) configured on the vehicle 111 .

[0112] At block 223 , the data storage device 101 stores at least a portion of the sensor data stream.

[0113] At box 225, the neural network accelerator 159 configured in the data storage device 101 performs at least a portion of the calculation based on the artificial neural network 125 and the sensor data stream.

[0114] At box 227, maintenance service for vehicle 111 is predicted based at least in part on calculations performed by a neural network accelerator 159 configured within the data storage device 101.

[0115] Optionally, at box 229, in the vehicle 111, an artificial neural network (ANN) is trained at least in part using a neural network accelerator and using sensor data streams collected over a predetermined time period (e.g., a period after a new vehicle 111 leaves the factory or after a component is replaced in a maintenance service facility 127).

[0116] For example, the neural network accelerator 159 may be configured on an integrated circuit device that is separate from the controller 151 of the data storage device and / or separate from the storage media components 161-163.

[0117] For example, the neural network accelerator 159 may be configured on an integrated circuit device including the controller 151 of the data storage device 101 , or configured on an integrated circuit device including the storage media component 160 , 161 , or 163 of the data storage device 101 .

[0118] For example, the neural network accelerator 159 may be configured to perform calculations using data stored in the data storage device 101, such as matrix arithmetic calculations for an ANN and / or differential equation simulations for an SNN.

[0119] Examples of matrix arithmetic calculations include matrix multiplication and accumulation operations. After performing calculations using the data stored in the data storage device 101 to produce the results of the matrix arithmetic calculations, the neural network accelerator 159 may provide the results as outputs of the data storage device 111 when the data is retrieved (e.g., in response to a read command). Alternatively or in combination, the results of the matrix arithmetic calculations may be buffered in the data storage device 101 as operands for the next matrix calculation performed in conjunction with the data matrix retrieved from the non-volatile memory via a read command received in the host interface 157.

[0120] When the artificial neural network (ANN) 125 includes a spiking neural network (SNN), the neural network accelerator may be configured to simulate differential equations that control activation levels of neurons in the spiking neural network (SNN). Optionally, the storage media component is configured to store states of neurons in the spiking neural network according to a predetermined pattern; and the neural network accelerator is configured to automatically update the states of the neurons over time according to the differential equation. For example, the neural network accelerator 159 may be configured to train a spiking neural network (SNN) to detect anomalies via unsupervised machine learning.

[0121] The computations performed by the neural network accelerator 159 based on the artificial neural network (ANN) 125 involve different data types with different usage patterns of the data storage device 101.

[0122] For example, making predictions using the artificial neural network (ANN) 125 includes using data specifying a model of the artificial neural network (ANN) 125, input data provided to the artificial neuron, and output data generated by the artificial neuron.

[0123] The storage capacity of the data storage device 101 can be divided into different portions of the different types of ANN-related data. The different portions can be individually configured to optimize access and storage of corresponding data based on the usage mode of the neural network accelerator 159 and / or the processor 133 of the computer system 131 in which the data storage device 101 is configured.

[0124] The model of the artificial neural network (ANN) 125 may include parameters that specify the static properties of individual artificial neurons in the ANN 125 and the connectivity of neurons in the ANN 125. The model data of the ANN 125 is static and does not change during the prediction calculations performed using the ANN 125. Therefore, the usage mode is usually to read the model data. However, when the updated ANN 125 is installed, the model data of the ANN 125 may change. For example, the vehicle 111 may download the updated ANN 125 from the server 119 to the data storage device 101 of the vehicle 111 to update its prediction capabilities. The model data of the ANN 125 may also change during or after training the ANN 125 using machine learning techniques (e.g., 171 or 175). It is preferred to configure a separate partition or namespace of the data storage device 101 to store the model data, wherein the partition or namespace optimizes the configuration parameters of the memory unit to operate according to a specific usage mode for the model data (e.g., mostly read, rarely updated). For example, when memory cells are implemented using NAND logic gate-based flash memory, the memory cells in the ANN model partition / namespace can be configured to operate in a multi-level cell (MLC) mode, a triple-level cell (TLC) mode, or a quad-level cell (QLC) mode, where each memory cell stores two, three, or four bits for increased storage capability.

[0125] The input data provided to the artificial neurons in the ANN 125 may include external inputs and internal inputs. The external inputs are usually generated by the sensors 103 of the vehicle 111 rather than by the artificial neurons in the ANN 125. The external inputs may be stored in a cyclic manner so that the input data for the latest time period of the predetermined driving length can be found in the data storage device 101. Therefore, it is preferred that a separate partition or namespace of the data storage device 101 is configured to store the external input data, wherein the partition or namespace operates according to configuration parameters that optimize the memory cells (e.g., enhance the durability of cyclic overwrite) for the storage mode of the external input data. For example, when the memory cells are implemented using flash memory based on NAND logic gates, the memory cells in the ANN input partition / namespace may be configured to operate in a single-level cell (SLC) mode, wherein each memory cell stores one bit of data for improved durability in cyclic overwrite operations.

[0126] In some embodiments, an artificial neuron may have state variables that change over time in response to inputs during a predictive computation. For example, the activation level of a spiking neuron may change over time and be considered a dynamic state variable of the spiking neuron. In some embodiments, such state variable data for an artificial neuron has a similar storage usage pattern as external input data; and therefore, the state variable data may be stored in a partition or namespace configured for external input data. In other embodiments, the state variable data for an artificial neuron is maintained in a buffer and is stored less frequently than external inputs; and therefore, another partition / namespace may be configured for storing dynamic state variable data for an artificial neuron.

[0127] The output data generated by the artificial neurons in the ANN 125 can be buffered for further access by the neural network accelerator 159 and / or the processor 133 of the computer system 131. The output data may include external outputs and internal outputs. The external input is generated by the artificial neuron as an output from the ANN 125, such as the result of classification or prediction performed by the ANN 125. The output of the ANN 125 is usually further processed by the processor 133 of the computer system 131. The external input can be saved periodically (e.g., in a manner similar to storing state variable data). Some of the internal outputs and / or external outputs may be internal inputs sent to the artificial neurons in the ANN 125. In general, it may not be necessary to store the internal outputs from the buffer of the data storage device to the storage media component. In some embodiments, when the buffering capacity of the data storage device 101 is insufficient to save the entire state variable data and / or internal output, the data storage device 101 can use the swap partition / namespace to expand the capacity of the buffer. The swap partition / namespace can be configured to optimize random access and to improve durability.

[0128] External outputs and / or dynamic states of neurons may be stored in a separate output partition or namespace in a cyclic manner, such that external output data and / or dynamic states of neurons may be stored periodically and the latest set of external outputs and / or dynamic states may be found in the data storage device 101. External outputs and / or dynamic states of neurons may be selectively stored since some of such data may be regenerated by the ANN from external inputs stored in the input partition or namespace. Preferably, the output partition or namespace is configured to store one or more sets of external outputs and / or dynamic states that cannot be generated from external inputs stored in the input partition or namespace. When storing data in an input / output partition or namespace in a cyclic manner, the oldest stored data set is erased to make room for the latest data set. The ANN input / output partition / namespace may be configured for an optimized continuous write stream to copy data from a buffer of the data storage device to a memory cell in a storage media component of the data storage device.

[0129] Fig.10 A data storage device 101 configured to support neural network computations according to one embodiment is shown. For example, the data storage device 101 may be used to Figure 1 or in a vehicle 111 in 2 to facilitate predictive maintenance and / or support ADAS 105 .

[0130] Similar to Figure 7 The data storage device 101, Fig.10 The data storage device 101 includes a host interface 157 and a controller 151 .

[0131] Similar to Figure 7 The storage media components 161 to 163 in the data storage device 101 can be implemented using a storage media component set Fig.10 The storage capacity 181 of the data storage device 101 is .

[0132] A set of namespaces 183, 185, 187, ... may be created on storage capacity 181 of data storage device 101. Each of the namespaces (e.g., 183, 185, or 187) corresponds to a named portion of storage capacity 181. Logical addresses are defined within each namespace. Address mapping 191 is configured to map logical addresses defined in namespaces 183, 185, 187, ... to storage media components (e.g., Figure 7 Mapping between the physical addresses of the memory cells in 161 to 163) described in .

[0133] Address map 191 may include namespace optimization settings 192 for namespaces 183 , 185 , and 187 .

[0134] For example, the ANN model namespace 183 may be a memory / storage partition configured for model data of the artificial neural network (ANN) 125. The namespace optimization settings 192 optimize memory operations in the ANN model namespace 183 based on the data usage pattern of the ANN model (e.g., mostly read, less update-centric).

[0135] For example, the neuron input namespace 185 may be a memory / storage partition configured for external input data sent to the artificial neural network (ANN) 125. The namespace optimization settings 192 optimize memory operations in the neuron input namespace 185 according to the data usage pattern of the external input data (e.g., enhanced durability to support cyclic overwrites for continuous input data streams for continuous writes).

[0136] For example, the neuron output namespace 187 may be a memory / storage partition / configured for external output data provided from the artificial neural network (ANN) 125. The namespace optimization settings 192 optimize memory operations in the neuron output namespace 187 based on the data usage pattern of the external output data (e.g., improving durability of data that is periodically overwritten by random read / write accesses).

[0137] The data storage device 101 includes a buffer 152 configured to store temporary / intermediate data of the artificial neural network (ANN) 125 , such as internal inputs / outputs of artificial neurons in the ANN 125 .

[0138] Optionally, a swap namespace may be configured in storage capacity 181 to expand the capacity of buffer 152 .

[0139] Optionally, address map 191 includes a mapping between logical memory addresses received in host interface 157 to access data of artificial neurons and identities of artificial neurons. Thus, a read or write command to access one type of data of an artificial neuron in one namespace may cause controller 151 to access another type of data of an artificial neuron in another namespace.

[0140] For example, in response to a request to write external input data for a neuron to the storage capacity 181 of the data storage device 185, the address map 191 may be used to calculate the address of the model parameters of the neuron in the ANN model namespace 183 and read the model parameters into the buffer 152 to allow the neural network accelerator 159 to perform calculations of the output of the neuron. The output of the neuron may be stored in the buffer 152 as an internal input to other neurons (e.g., to reduce write amplification). In addition, the identities of other neurons connected to the neuron may also be retrieved from the ANN model namespace 183 into the buffer 152, which allows the neural network accelerator 159 and / or processor to further process the propagation of the output in the ANN 125. The retrieval of the model data from the ANN model namespace 183 may be performed in parallel with the storage of the external input data into the neuron input namespace 185. Therefore, the processor 133 of the computer system 131 of the vehicle 111 does not have to explicitly send a read command to retrieve the model data from the ANN model namespace 183.

[0141] Similarly, in response to reading the output data of a neuron, address map 191 may be used to calculate the address of the model parameters of the neuron stored in ANN model namespace 183 and read the model parameters into buffer 152 to allow neural network accelerator 159 to apply the internal input in buffer 152 to perform the calculation of the output of the neuron. The calculated output may be provided as a response to reading the output data of the neuron without having to cause data storage device 101 to store the output data in storage media components (e.g., 161 to 163). Thus, processor 133 and / or neural network accelerator 159 may control the calculation of the neuron by writing input to the neuron and / or reading output from the neuron.

[0142] In general, the external input data passed into the ANN 125 may be raw sensor data 121 directly generated by the sensor 103 without being processed by the processor 133 and / or the neural network accelerator 159. Alternatively, indirect sensor data 121 that has been processed by the processor 133 of the ANN 125 from the signal from the sensor 103 may be provided as the external input data. The incoming external input data may be accepted in the host interface 157 and written to the neuron input namespace 185 in a cyclic manner, and automatically buffered in the buffer 152 of the neural network accelerator 159 to generate neuron output using the model stored in the ANN model namespace 183. The output generated by the neural network accelerator 159 may be further buffered as internal input to further apply the model in the ANN model namespace 183. When the external output becomes available, the data storage device 101 may report the completion of the write request by an indication of the availability of the external output. Optionally, the controller 151 and / or the neural network accelerator 159 may generate an internal read command to cause a signal to propagate in the ANN 125 when the external output is generated. Alternatively, the host processor 133 may control the propagation of signals in the ANN 125 by selectively reading the outputs of neurons; and the data storage device 101 may actively buffer the data that may be needed in the buffer 152 to accelerate the ANN calculation.

[0143] Fig.11 The configuration of the namespace 183 for an artificial neural network (ANN) model according to one embodiment is described. For example, Fig.11 The configuration can be implemented in Figure 7 and / or the data storage device 101 described in 10. For example, Fig.11 The setting 193 can be Fig.10 The namespace optimization settings are part of 192.

[0144] Fig.11 The configuration maps the ANN model namespace 183 to at least one storage media component A 161. Preferably, at least one storage media component A 161 is available to the controller 151 in parallel with storage media components (e.g., 163) of other namespaces (e.g., 185 and 187) hosting ANN data. For example, the storage media component A 161 may be in an integrated circuit package that is separate from the integrated circuit packages used for other namespaces (e.g., 185 and 187). Alternatively, the storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 may be formed on separate regions of the integrated circuit die, where the separate regions can be operated substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0145] exist Fig.11 In , setting 197 is optimized for a usage pattern of mostly reads and few updates.

[0146] Fig.12 The configuration of the namespace 185 for inputs sent to an artificial neuron according to one embodiment is described. For example, Fig.11 The configuration can be implemented in Figure 7 and / or the data storage device 101 described in 10. For example, Fig.11 The setting 195 can be Fig.10 The namespace optimization settings are part of 192.

[0147] Fig.12 The configuration maps the neuron input namespace 185 to at least one storage media component B 163. Preferably, at least one storage media component B 163 is available to the controller 151 in parallel with storage media components (e.g., 161) of other namespaces (e.g., 183 and 187) hosting ANN data. For example, storage media component B 163 may be in an integrated circuit package separate from the integrated circuit packages used for other namespaces (e.g., 183 and 187). Alternatively, storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, storage media components 161-163 may be formed on separate regions of the integrated circuit die, where the separate regions may be operated substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0148] exist Fig.12 In the example, setting 197 is optimized for use in a cyclic continuous overwrite mode that records a continuous input data stream sampled at fixed time intervals to enhance endurance.

[0149] Fig.13 The configuration of the namespace 187 for outputs from an artificial neuron according to one embodiment is illustrated. For example, Fig.11 The configuration can be implemented in Figure 7 and / or the data storage device 101 described in 10. For example, Fig.11 The setting 197 can be Fig.10 The namespace optimization settings are part of 192.

[0150] Fig.13The configuration maps the neuron output namespace 187 to at least one storage media component C 162. Preferably, at least one storage media component C 162 is available to the controller 151 in parallel with storage media components (e.g., 161 and 163) of other namespaces (e.g., 183 and 185) hosting ANN data. For example, the storage media component C 162 may be in an integrated circuit package that is separate from the integrated circuit packages used for the other namespaces (e.g., 183 and 185). Alternatively, the storage media components 161-163 are formed on separate integrated circuit dies embedded in the same integrated circuit package. Alternatively, the storage media components 161-163 may be formed on separate regions of the integrated circuit die, where the separate regions may be operated substantially in parallel (e.g., for reading, for erasing, and / or for writing).

[0151] exist Fig.13 In the example, the settings 197 are optimized to periodically overwrite the buffered data by random access. For example, the memory cells are configured to update / overwrite in the neuron output namespace 187 at a higher frequency than in the ANN model namespace 183, but less frequently than in the neuron input namespace 185 by optimizing the settings 193 to 197.

[0152] Fig.14 A method for predictive maintenance supported by model partitioning according to one embodiment is shown. For example, Fig.14 The method can be used Figure 7 or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.14 The method can be used with Figure 6 and / or the method in 9 are used in combination.

[0153] At block 241, the non-volatile memory of the data storage device 101 is configured into a plurality of partitions (e.g., 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) for storing data; and the same type of memory cells in different partitions (e.g., 183 to 187) may be configured in different ways to optimize their performance according to the usage patterns of the data stored in the different partitions (e.g., 183 to 187).

[0154] At box 243, the data storage device 101 stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125 for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include a model partition (e.g., 193) configured to store model data of the artificial neural network 125.

[0155] At block 245 , the data storage device 101 receives a sensor data stream (eg, 121 ) from at least one sensor 103 disposed on the vehicle 111 .

[0156] At block 247 , the controller 151 of the data storage device 101 operates the memory cells in the partitions 183 , 185 , 187 , . . . according to the set of memory operation settings (eg, 193 , 195 , 197 ) in response to the sensor data stream (eg, 121 ).

[0157] At box 249 , the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict maintenance service for the vehicle 111 based on the sensor data stream (eg, 121 ).

[0158] For example, the memory operation setting configures the model partition (e.g., 183) to store three or more bits per memory cell. The memory operation setting may include an address map 191 that maps between neurons in the ANN 125 and inputs to the neurons. When a first address to an input to a neuron in the artificial neural network 125 is received, the first address in the input partition (e.g., 185) separated from the model partition (e.g., 183) may be converted to at least one second address of model data associated with the neuron, so that the properties of the neuron and the identity of the neuron connected to the neuron can be retrieved from the model partition (e.g., 183) without an explicit command from the processor 133. The controller 151 may automatically retrieve the model data associated with the neuron from the model partition (e.g., 183) using the at least one second address in response to receiving the first address. The neural network accelerator 159 may generate an output of the neuron from the input to the neuron and the model data associated with the neuron. In general, the input to the neuron may include outputs from multiple neurons connected to the neuron in the ANN 125. Controller 151 may store the output of the neuron in buffer 152 in data storage device 101 to facilitate accelerated access to the output by host processor 133 and / or neural network accelerator 159.

[0159] Typically, the model data does not change during calculations used to predict maintenance services. For example, the model data may include neuron connectivity data of an artificial neural network and static properties of neurons in the artificial neural network. The memory operation settings (e.g., 192) may configure the model partition (e.g., 183) to store more than one bit per memory cell in the non-volatile memory based on a usage pattern that mostly reads and rarely updates the model data.

[0160] For example, partitions (e.g., 183, 185, 187, ...) in a data storage device may be implemented as namespaces in which logical addresses are defined; and an address mapping 191 in the data storage device is configured to map the namespaces 183, 185, 187, ... to separate storage media components (e.g., 161, 163, 162, ...).

[0161] The model data in the model namespace 183 may be updated during training via machine learning 171 or 175 or during an over-the-air update to the ANN 125 from the server 119 .

[0162] In some embodiments, the controller 151 is configured to retrieve model data associated with a neuron in the artificial neural network from the model partition via the address map 191 in response to an input to a neuron or an output from the neuron being addressed in a partition separate from the model partition 183. In addition, the controller 151 can retrieve model data associated with a neuron from the model partition 183 in parallel with storing input to a neuron in a partition (e.g., 185) separate from the model partition 183.

[0163] Fig.15 A method for predictive maintenance supported by input partitioning according to one embodiment is shown. For example, Fig.15 The method can be used Figure 7 or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.15 The method can be used with Figure 6 , 9 and / or the method in 14 are used in combination.

[0164] At block 261, the non-volatile memory of the data storage device 101 is configured into multiple partitions (e.g., 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (e.g., NAND flash memory cells) implemented in multiple storage media components (e.g., 161-163).

[0165] At box 263, the data storage device 101 stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125 for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include an input partition (e.g., 185) configured to cyclically store input data for the artificial neural network 125.

[0166] For example, the input partition 185 can be configured to store external inputs rather than internal inputs for the artificial neural network 125. The input data stored in the input partition 185 is independent of the outputs from the neurons in the artificial neural network 125.

[0167] For example, the input data stored in input partition 185 may include a portion of a sensor data stream (e.g., 121). In some embodiments, the input data stored in input partition 185 is calculated from the sensor data stream (e.g., 121) for a subset of neurons in artificial neural network 125.

[0168] For example, the memory operation settings (eg, 195) configure the input partition 185 to store one bit per NAND memory cell in non-volatile memory for enhanced endurance to repeated data erasure and data programming.

[0169] For example, the memory operation setting (eg, 195 ) configures the controller to write input data sequentially into the input partition 185 and overwrite the oldest input data in the input partition 185 with the newest input data received in the data storage device 101 .

[0170] At block 265 , the data storage device 101 receives a sensor data stream (eg, 121 ) from at least one sensor 103 disposed on the vehicle 111 .

[0171] At box 267, the controller 151 of the data storage device 101 operates the memory cells in the partitions 183, 185, 187, ... according to the set of memory operation settings (eg, 193, 195, 197) in response to the sensor data stream (eg, 121).

[0172] At box 269 , the computer system 131 with the data storage device 101 uses the artificial neural network 125 to predict maintenance service for the vehicle 111 based on the sensor data stream (eg, 121 ).

[0173] Fig.16 A method for predictive maintenance supported by input partitioning according to one embodiment is shown. For example, Fig.16 The method can be used Figure 7or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.16 The method can be used with Figure 6 , 9 , 14 and / or 15 are used in combination.

[0174] At block 281, non-volatile memory of the data storage device 101 is configured into multiple partitions (eg, 183, 185, 187, ...). For example, the non-volatile memory may have the same type of memory cells (eg, NAND flash memory cells) for storing data.

[0175] At box 283, the data storage device 101 stores different sets of memory operation settings (e.g., 193, 195, 197) for different types of data related to the artificial neural network 125 for the partitions (e.g., 183, 185, 187, ...), respectively, wherein the partitions (e.g., 183, 185, 187, ...) include an output partition (e.g., 187) configured to store output data for the artificial neural network 125.

[0176] For example, the output data stored in the output partition (e.g., 187) may include state data of a neuron in the artificial neural network 125. For example, the state data of a neuron in the artificial neural network may identify an activation level of the neuron for generating a spike in the spiking neural network. The activation level may be controlled via a differential equation. Thus, the activation level may change in response to input to the artificial neural network 125 and / or in response to the passage of time.

[0177] For example, the output data may include predictions or classifications generated by the artificial neural network 125 in response to the sensor data stream.

[0178] For example, the memory operation setting configures the output partition to store no more than two bits per memory cell in the non-volatile memory.

[0179] At block 285 , the data storage device 101 receives a sensor data stream (eg, 121 ) from at least one sensor 103 disposed on the vehicle 111 .

[0180] At box 287 , the controller 151 of the data storage device 101 operates the memory cells in the partitions 183 , 185 , 187 , . . . according to the set of memory operation settings (eg, 193 , 195 , 197 ) in response to the sensor data stream (eg, 121 ).

[0181] At box 289 , the computer system 131 having the data storage device 101 uses the artificial neural network 125 to predict maintenance service for the vehicle 111 based on the sensor data stream (eg, 121 ).

[0182] For example, the data storage device 101 may include a buffer 152. The buffer 152 may be implemented via a volatile memory (e.g., SRAM or DRAM) to obtain faster access performance compared to a non-volatile memory (e.g., NAND flash memory) of the data storage device 101. The memory operation setting configures the controller 151 to store the output data in the buffer 152 for access by the processor (e.g., 133) via the host interface 157 during or after storing the output data in the output partition 187.

[0183] For example, the data storage device 101 may include a neural network accelerator 159 coupled to the controller 151. The neural network accelerator is configured to apply inputs provided to neurons in the artificial neural network 125 to model data of the artificial neural network 125 to generate output data through one or more output neurons in the artificial neural network 125. In response to the neural network accelerator 159 completing the calculation of the output data, the controller is configured to provide an indication of the availability of the output data generated by the artificial neural network 125 to the processor (e.g., 133), so that the processor (e.g., 133) can request the data storage device 101 to transmit the output data.

[0184] Optionally, the controller 151 is configured to provide the output data to the processor in parallel with storing the output data in the output partition. For example, the controller 151 may be configured to automatically discard the output data calculated for the previous segment of the sensor data stream if the processor (e.g., 133) does not request that the output data be transmitted to the processor (e.g., 133) within a predetermined time period or before the next version of the output data is available. Optionally, after reporting the availability of the output data to the processor (e.g., 133), the controller 151 may be configured to selectively discard the output data calculated for the previous segment of the sensor data stream based on the response of the processor (e.g., 133) to the output data sent to the processor (e.g., 133). For example, in some cases, a processor (e.g., 133) may request that output data be transmitted to the processor (e.g., 133) without saving the output data to the output partition (e.g., 187); and in other cases, a processor (e.g., 133) may request that output data be transmitted to the processor (e.g., 133) and that the output data be stored to the output partition (e.g., 187).

[0185] Optionally, output data from the artificial neural network 125 may also be stored in output partitions in a round-robin manner (eg, for segments of output data within a time period selected by the processor (eg, 133)).

[0186] For example, external inputs to the artificial neural network 125 may be continuously recorded in the input namespace 185 over the last time period T1. When the sensor data is sampled at a predetermined time interval T2, the input namespace 185 may save the latest T1 / T2 input data set. In contrast, external outputs from the artificial neural network 125 may be selectively recorded in the output namespace 187 (e.g., once per predetermined time period T3, where T3 is a multiple of T2). Output data may be recorded in the output namespace 187 at a lower frequency; and the output namespace 187 may be allocated to store a predetermined number of output data sets (e.g., via continuous writing and writing that occurs in a loop to keep the last output data set).

[0187] At least some embodiments disclosed herein include a communication protocol / interface to allow a data storage device to perform on-the-fly neural network acceleration with reduced data traffic to a host processor (e.g., a central processing unit (CPU)).

[0188] For example, the host processor (e.g., 133) of the vehicle 111 may provide a write command to the data storage device 101 to store the artificial neural network model model in the model partition (e.g., 183). Since the neural network accelerator 159 is configured to apply the model, data communications for sending data of the model of the ANN 125 back to the processor can be reduced or eliminated.

[0189] To use the ANN model in classification and / or prediction, the host processor (e.g., 133) of the vehicle 111 may stream input data for the ANN 125 into the neuron input partition (e.g., 185). The neural network accelerator 159 of the storage device 101 may automatically apply the input data to the model stored in the ANN model partition (e.g., 183) according to the address map 191. The data storage device 101 makes the calculated output available for propagation in the ANN 125. Preferably, the calculated output is made available to the neural network accelerator 159 through the buffer 152 without the need to store the intermediate output in the storage media component (e.g., 161 to 163). Therefore, data communication between the host processor (e.g., 133) and the data storage device 101 for conveying the output of the neuron can be reduced. When the output has been propagated to the output neuron in the ANN 125, the data storage device 101 may provide a response to a write request associated with writing the input data set into the neuron input partition (e.g., 185). The response indicates that external outputs from neurons in ANN 125 are available. In response, a host processor (eg, 133) of vehicle 111 may optionally issue a read command to retrieve the external outputs for further processing.

[0190] Fig.17 101 to perform neural network calculations. Fig.17 The communication described in Figure 7 The data storage device 101 described in or 10 is implemented in Figure 1 Or 2 in vehicle 111.

[0191] exist Fig.17 In the embodiment of the present invention, the processor 133 may be configured with a simplified instruction set 301 for performing neural network calculations because some of the calculations involving the ANN 125 are performed by the neural network accelerator 159 within the data storage device 101. Therefore, it is not necessary to transfer model data back to the processor 133 during prediction and / or classification using the ANN 125.

[0192] The sensor 103 may generate a continuous stream of sensor data 121 based on a sampling rate of the data. The sensor data 121 may be sampled at fixed predetermined time intervals (e.g., during operation of the vehicle 111). The processor 133 may execute instructions 301 to convert the sensor data 121 into an input stream 303 for input neurons in the ANN 125. The input neurons in the ANN 125 are configured to accept external inputs sent to the ANN 125; and the output neurons are configured to provide external outputs from the ANN 125.

[0193] In general, a complete input set for the ANN 125 at a certain time contains inputs for the entire set of input neurons of the ANN 125. The input stream 303 contains a sequence of input sets for a sequence of time instances spaced apart from each other according to fixed predetermined time intervals.

[0194] The data storage device 101 stores the input stream 303 into the neuron input namespace 185 in a cyclic manner, wherein the oldest input set corresponding to the earliest time of data sampling of the data set currently stored in the neuron input namespace 185 is erased to store the latest input set in the input stream 303.

[0195] For each input data set, the neural network accelerator 159 applies the model of the ANN 125 stored in the ANN model namespace 183. The neural network accelerator 159 (or the processor 133) may control the propagation of signals within the neural network. When the output neuron of the ANN 125 generates its output in response to the input data set, the data storage device 101 may provide an indication to the processor 133 that the neuron output is ready for retrieval. The indication may be configured in response to a request from the processor 133 to write the input data set into the neuron input namespace 185. The processor 133 may optionally retrieve the output data 305 (e.g., according to conditions and / or criteria programmed in the instructions).

[0196] In some embodiments, the trigger parameters are configured in the data storage device 101. The data storage device provides a response to a request from the processor 133 to write an input data set into the neuron input namespace 185 when the output parameters in the external output 317 meet the requirements specified by the trigger parameters.

[0197] Fig.18 Communicating with a data storage device to perform neural network calculations according to one embodiment is shown. For example, Fig.18 The communication can be combined Fig.17 Communication implementation Figure 7 Or the data storage device 101 described in 10.

[0198] exist Fig.18 In the example, the model namespace 183 stores the model 313 of the entire ANN 125. In response to receiving an external input set 315 for a time instance from the input stream 303 in the buffer 152, the data storage device 101 can write the external input 315 to the input namespace 185 in parallel with retrieving the neuron model 312 containing the portion of the ANN model 313 corresponding to the parameters of the input neuron and / or the identity of the neuron connected to the input neuron. The buffer 152 allows the neural network accelerator 159 to combine the neuron model 312 and the external input 325 to produce the output 327 of the input neuron.

[0199] In general, neuron output 327 may include a portion of internal output 316 that is for further propagation within ANN 125 and / or a portion of external output 317 that is for processor 133 .

[0200] The internal output 316 is stored in the buffer 152 as the internal input 316 for further propagation in the ANN 125 in a similar manner to the generation of the neuron output 327 from the external input 315. For example, a portion of the internal input 316 may cause the controller 151 and / or the neural network accelerator 159 to retrieve the corresponding neuron model 312 associated with the internal input, so that the internal input is applied to the corresponding neuron model 312 in the neural network accelerator 159 to generate its neuron output 327.

[0201] When a complete set of external outputs 317 is available in buffer 152 , external outputs 317 may be stored in output namespace 187 .

[0202] Optionally, the storage device 101 does not store every external output set 317 corresponding to the stored external input set 315 sampled at a time. For example, the storage device 101 may be configured to store one external output set 317 for every predetermined number of external input sets (e.g., 315). Alternatively or in combination, the processor 133 may determine whether to store the external output 317. For example, the storage device 101 may be configured to store the external output 317 in response to the processor 133 retrieving the external output 317 for further processing. For example, the storage device 101 may be configured to store the external output 317 in response to a write command from the processor 133 after the external output 317 is processed in the processor 133.

[0203] Fig.19 A method of communicating with a data storage device to perform neural network calculations according to one embodiment is shown. For example, Fig.19 The method can be used Figure 7 or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.19 The method can be used with Figure 6 , 9 , 14, 15 and / or 16 are used in combination.

[0204] At box 341 , the one or more processors 133 of the vehicle 111 stores the model data (eg, 313 ) of the artificial neural network (eg, 125 ) in the data storage device 101 .

[0205] At box 343 , one or more processors 133 of the vehicle 111 receive a sensor data set from at least one sensor 103 configured on the vehicle 111 .

[0206] At box 345 , one or more processors 133 of the vehicle 111 generate a set of inputs to an artificial neural network (eg, 125 ) based on the sensor data.

[0207] At box 347, the one or more processors 133 of the vehicle 111 provide the input set to the data storage device 101. In response to the input set, the data storage device 101 is configured to generate an output set using the model data 313 of the artificial neural network (eg, 125).

[0208] At box 349 , the one or more processors 133 of the vehicle 111 retrieve the output set from the data storage device 101 .

[0209] For example, data storage device 101 generates an output set using at least a portion of model data 183 stored in the data storage device without transmitting the portion of model data 183 to one or more processors 133 between receiving the input set and completing calculation of the output set.

[0210] For example, portion of model data 183 may include static properties of neurons in an artificial neural network (eg, 125) and / or neuron connectivity data of the artificial neural network (eg, 125).

[0211] For example, to provide the input set to the data storage device 101, the one or more processors 133 of the vehicle 111 may transmit one or more write commands to the data storage device 101. The one or more write commands are configured to instruct the data storage device 101 to store the input set in the data storage device 101. After completing the calculation of the output set in the data storage device 101, the controller 151 of the data storage device 101 may transmit a response to the one or more write commands to the one or more processors 133. The response may include an indication that the output set is available for retrieval by the one or more processors 133.

[0212] In response to the indication, one or more processors 133 may optionally retrieve the output set from data storage device 101 by transmitting a read command to data storage device 101 to obtain the output set (eg, after determining to retrieve the output set from data storage device 101 for processing).

[0213] Alternatively or in combination, the one or more processors 133 of the vehicle 111 may determine whether to store the output set in the non-volatile memory of the data storage device. In response to determining to store the output set in the non-volatile memory of the data storage device 101, the one or more processors 133 of the vehicle 111 may transmit a write command to the data storage device 101.

[0214] Because output sets are initially generated in the data storage device 101 and then buffered in a buffer 152 (e.g., volatile memory), the data storage device 101 can execute a write command to store the output sets in the output name space 187 without transmitting the output sets to one or more processors 133 and / or receiving the output sets from one or more processors 133 in response to the write command.

[0215] For example, after receiving another set of sensor data 121 from at least one sensor 103 configured on the vehicle 111 , the one or more processors 133 of the vehicle 111 generate another set of inputs to the artificial neural network 125 based on the another set of sensor data.

[0216] The one or more processors 133 transmit another command to write another input set into the data storage device 101; and the data storage device 101 generates another output set using the model data 183 of the artificial neural network 125 and the another input set. After receiving a response to the another command to write another input set, the one or more processors 133 may determine to skip processing of another output set and transmit a continuous write command to store another output set to the data storage device 101. In response, the data storage device 101 may write another output set buffered in the data storage device 101 into the output namespace 187 without transmitting another output set from the one or more processors 133 of the vehicle 111 to the data storage device 101 and / or without transmitting another output set from the data storage device 101 to the one or more processors 133 of the vehicle 111.

[0217] The neural network techniques discussed above may be used to automate the control of audio content delivery in a vehicle.

[0218] For example, one or more microphones may be configured to monitor the audio content being played in the vehicle under various operating conditions of the vehicle. The characteristics of the audio content may be analyzed to train an artificial neural network (e.g., through reinforcement learning and / or supervised learning) to recognize patterns of audio activity under various conditions. The pattern may be identified in combination with audio characterization parameters, such as the type of audio content, the source of the audio content, the style and genre of music, etc. Subsequently, the artificial neural network may predict the audio content presentation mode for the current operating conditions of the vehicle and select the audio source and content that best matches the predicted audio content presentation mode. The prediction may reduce or eliminate the need for the occupants of the vehicle (e.g., the driver and / or the passenger to manually operate the control element to select the audio source and / or adjust the audio playback settings. The audio source and / or content selected according to the prediction made by the artificial neural network may be presented to the occupant for approval or confirmation. With or without confirmation from the occupant (e.g., via voice command), changing the audio source and / or content presented in the vehicle based on the prediction may greatly simplify the user interface for controlling the audio presentation in the vehicle.

[0219] Fig. 20 A system for selectively presenting audio content in a vehicle is shown according to one embodiment. For example, Fig. 20 The system can be Figure 7 The data storage device 101 of 10 is implemented in Figure 1 Or the vehicle 111 described in 2.

[0220] exist Fig. 20 In the embodiment of the present invention, an artificial neural network (ANN) 125 is configured to receive input related to the presentation of audio content in the vehicle 111 .

[0221] Inputs to the ANN 125 may include signals for one or more microphones 351 disposed in a cabin in which one or more occupants (e.g., a driver and / or passengers) of the vehicle 111 are located and / or seated. The microphones 351 monitor audio content delivered to the cabin of the vehicle 111 (e.g., via the infotainment system 149) and / or voice conversations or inputs generated in the cabin of the vehicle 111.

[0222] The infotainment system 149 can be connected to multiple audio sources, such as a radio with access to multiple stations, a compact disc (CD) player, a media player that can generate audio presentations from digital data stored in a computer-readable medium (e.g., a flash drive), an audio source from a mobile device 353 (e.g., connected to the infotainment system 149 via a Bluetooth connection), an audio source on the Internet (e.g., connected to the infotainment system 149 via a wireless connection and / or a telecommunications network connection), and / or an input from an auxiliary audio port / jack.

[0223] Optionally, the infotainment system 149 may provide information to the ANN 125 regarding audio content currently being presented in the cabin of the vehicle 111. For example, the information regarding the audio content may include identification of the source of the audio content, the type of audio content, the style and genre of music, and the like.

[0224] The inputs sent to the ANN 125 may include vehicle operating parameters 357, such as the driving speed of the vehicle 111, the ambient temperature outside the vehicle 111, the cabin temperature inside the vehicle 111, the orientation of the vehicle 111, whether the engine of the vehicle 111 is started or stopped, and / or whether the parking brake is pulled. In addition, the vehicle operating parameters 357 may optionally include the position / state of the folding convertible top of the vehicle 111, the position / setting of the sunroof / moonlight of the vehicle 111, and / or the opening state and / or position of the windows of the vehicle 111.

[0225] The ANN 125 is configured to recognize audio patterns in the signal from the microphone 351 that are associated with information received from the infotainment system 149. For example, the audio patterns may include volume levels, audio balance levels, acoustic characteristics or profiles, and the like.

[0226] During the training period, when the user does not adjust controls for the infotainment system 149, the ANN 125 is trained to predict the audio pattern 359 of the vehicle operating parameter 357. For example, the training may be performed by supervised machine learning techniques (e.g., 171) or unsupervised machine learning techniques (e.g., 175).

[0227] When the user adjusts the settings of the infotainment 149 using the control elements 132 of the user interface, the ANN 125 is further trained to predict the corresponding audio pattern 359 over the current vehicle operating parameters 357. The corresponding audio pattern 359 may correspond to the pattern 359 of the signal in the microphone 351 and / or associated characterizing parameters from the infotainment system 149, such as the type of audio content captured by the microphone 351, the musical style and genre of the audio content captured by the microphone 351, and / or the identification of the source of the audio content captured by the microphone 351.

[0228] Subsequently, when the predicted audio pattern 359 deviates from the audio pattern monitored by the microphone 351 and / or the infotainment system 149 , the computer system 131 is configured to select the source having the best match between the audio pattern of the selected source and the predicted audio pattern 359 .

[0229] For example, the computer system 131 may preview available content sources to determine audio patterns in the content that the computer system 131 is previewing. For example, the computer system 131 and / or the infotainment system 149 may virtually play the audio of the selected source to produce a signal that corresponds to the signal that would be detected by the microphone 351 when the audio of the selected source is presented in the cabin of the vehicle 111. The audio pattern in the signal and / or the corresponding information of the selected source may be compared to the predicted audio pattern 359 for a match determination. For example, a match score may be calculated to rank the available sources to select the best match.

[0230] In some embodiments, the predicted audio pattern 359 includes characterizing parameters of the preferred audio content, such as audio type, and / or music style and genre. The computer system 131 and / or the infotainment system 149 can select audio content based on the characterizing parameters identified in the predicted audio pattern 359.

[0231] Optionally, the computer system 131 is configured to effectuate a change in the settings of the infotainment system 149 to cause the infotainment system 149 to present audio content selected according to the predicted audio pattern 359 .

[0232] Optionally, the computer system 131 is configured to prompt the user to confirm the change in the settings of the infotainment system 149. For example, the computer system 131 may communicate the proposed change and / or provide a brief audio presentation of the content. The occupants in the vehicle 111 may be provided with an opportunity to accept or reject the proposed change in the settings of the infotainment system 149.

[0233] For example, the computer system 131 may announce the proposed changes via synthesized speech and detect user confirmation via voice within a predetermined time period for the user to respond to the announcement. The user may confirm or reject the proposed changes via voice commands, or reject the proposed changes by not responding within a predetermined time period.

[0234] Optionally, the ANN 125 is trained to not only predict the audio pattern 359 but also predict the likelihood that the current users of the vehicle 111 (eg, the driver and / or passengers) will accept a change in the predicted pattern 359 .

[0235] For example, computer system 131 may determine that the user has accepted a change to a setting of infotainment system 149 after automatically implementing the change without further manual adjustment of control elements 132 by the user.

[0236] For example, computer system 131 may determine that a user has rejected an automatically implemented change after the user manually modifies the change within a predetermined period of time.

[0237] When the computer system 131 is configured to prompt the user to confirm the changes, the user response may be used to determine whether the changes implemented according to the predicted audio pattern 359 are accepted or rejected.

[0238] When the change in audio content is rejected, the ANN 125 is further trained to decrease the likelihood of the predicted audio pattern 359 (e.g., decrease it by a predetermined increment). When the predicted adjustment is accepted, the ANN 125 is further trained to increase the likelihood of the predicted audio pattern 359 (e.g., increase it by a predetermined increment).

[0239] The computer system 131 is then configured to implement the predicted audio mode 359 or prompt the user to confirm a change to implement the predicted audio mode 359 when the predicted likelihood of acceptance is above a threshold for the predicted audio mode 359 .

[0240] Fig.21 A method for controlling the presentation of audio content in a vehicle according to one embodiment is shown. For example, Fig.21 The method can be used Figure 7 or the data storage device 101 of 10 and / or Figure 8 The storage media component 160 is implemented in Figure 1 or 2 in vehicle 111. For example, Fig.21 The method can be used with Figure 6 , 9 , 14, 15, 16 and / or 19 are used in combination.

[0241] At block 361, at least one microphone 351 disposed in the vehicle 111 generates a signal representative of audio content presented in the cabin of the vehicle 111. The infotainment system 149 disposed in the vehicle 111 may access a plurality of audio content sources.

[0242] Examples of audio content sources include radio stations on the infotainment system's radio, mobile devices connected to the infotainment system (e.g., via a wired or wireless connection), and files on computer storage media (e.g., compact discs (CDs), USB drives, flash drives, flash memory cards, etc.) installed in the infotainment system.

[0243] At box 363 , the artificial neural network 125 receives input parameters related to audio control in the vehicle 111 .

[0244] For example, the input parameters may include data representing a signal from at least one microphone 351 , data from the infotainment system 149 , and / or at least one operating parameter 357 of the vehicle 111 .

[0245] For example, at least one operating parameter 357 of vehicle 111 may include the speed of vehicle 111 traveling on the roadway, the position of vehicle 111, the temperature, the window opening setting, the setting of a top window or sunroof of vehicle 111, or the setting of a foldable convertible top of vehicle 111, or any combination thereof.

[0246] For example, data from infotainment system 149 may include the type of audio content currently being delivered by infotainment system 149 to the cabin of vehicle 111 , the style of music, the genre of music, or the source of audio content currently being delivered by infotainment system 149 to the cabin of vehicle 111 , or any combination thereof.

[0247] At box 365, the artificial neural network 125 generates a prediction of the audio pattern based on the input parameters that vary over time.

[0248] For example, the prediction of the audio pattern may include a predicted audio content type, a predicted music style, a predicted music genre, or a predicted source, or any combination thereof; and the vehicle 111 may be further configured to select a source of audio content based on the predicted audio content type, the predicted music style, the predicted music genre, or the predicted source, or any combination thereof.

[0249] At box 367, the vehicle 111 adjusts the settings of the infotainment system 149 based at least in part on the prediction.

[0250] For example, the setting may be a volume control for the infotainment system 149 , an audio equalizer control for the infotainment system 149 , and / or a source selection for audio content presented in the cabin of the vehicle.

[0251] Optionally, the vehicle may be configured to implement a first adjustment to the settings of the infotainment system 149 based at least in part on a prediction generated by the artificial neural network 125 without requesting confirmation from an occupant in the vehicle 111, and to request confirmation from the occupant before implementing a second adjustment to the settings of the infotainment system based at least in part on a prediction generated by the artificial neural network 125.

[0252] For example, the artificial neural network may be configured to target not only the audio mode 359 , but also the likelihood that one or more occupants in the vehicle 111 will accept adjusting the settings of the infotainment system based on the prediction.

[0253] For example, the ANN 125 may predict multiple audio patterns 359 and different likelihoods of being accepted by occupants of the vehicle 111. The computer system 131 is configured to select an adjustment to implement for the audio pattern based on the predicted likelihoods and the availability of matching sources for the audio pattern.

[0254] For example, computer system 131 of vehicle 111 may implement the adjustment in response to determining that the likelihood is above a threshold, or request approval from one or more occupants of the vehicle to implement the adjustment in response to determining that the likelihood is above a threshold.

[0255] The artificial neural network 125 may include a spiking neural network. In the absence of adjustments to the settings of the infotainment system 149 initiated by one or more occupants of the vehicle, the artificial neural network 125 may be trained to predict an audio pattern corresponding to a signal captured by the microphone 351, the signal representing audio content presented in the cabin of the vehicle. In response to adjustments to the settings of the infotainment system 149 initiated by one or more occupants of the vehicle using a control element 132 configured in the vehicle 111, the artificial neural network 125 may be further trained to predict an audio pattern corresponding to a signal captured by the microphone 351, the signal representing audio content presented in the cabin of the vehicle 111 due to the adjustments.

[0256] The data storage device 101 may be configured to store the model data 313 of the ANN 125 and calculate the predicted audio pattern 359 in response to receiving the input parameters. The predicted audio pattern 359 may be calculated in the data storage device 101 using the neural network accelerator 159 and the model data 313 stored in the data storage device 101. Optionally, the data storage device 101 may be configured as part of the infotainment system 149.

[0257] Server 119 , computer system 131 , and / or data storage device 101 may each be implemented as one or more data processing systems.

[0258] The present disclosure includes methods and apparatus for performing the methods described above, including data processing systems for performing these methods, and computer-readable media containing data that, when executed on a data processing system, causes the system to perform these methods.

[0259] A typical data processing system may include an interconnect (eg, a bus and system core logic) that interconnects a microprocessor and memory. A microprocessor is typically coupled to a cache memory.

[0260] The interconnect interconnects the microprocessor and memory together and also interconnects them to input / output (I / O) devices via I / O controllers. The I / O devices may include a display device and / or peripheral devices such as a mouse, keyboard, modem, network interface, printer, scanner, camera, and other devices known in the art. In one embodiment, when the data processing system is a server system, some of the I / O devices (such as a printer, scanner, mouse, and / or keyboard) are optional.

[0261] The interconnect may include one or more buses connected to each other through various bridges, controllers, and / or adapters. In one embodiment, the I / O controller includes a USB adapter for controlling a Universal Serial Bus (USB) peripheral device, and / or an IEEE-1394 bus adapter for controlling an IEEE-1394 peripheral device.

[0262] The memory may include one or more of: read-only memory (ROM), volatile random access memory (RAM), and non-volatile memory such as a hard drive, flash memory, and the like.

[0263] Volatile RAM is typically implemented as dynamic RAM (DRAM), which requires a constant supply of power to refresh or maintain the data in the memory. Nonvolatile memory is typically a magnetic hard drive, a magnetic optical drive, an optical drive (e.g., DVD RAM), or other types of memory systems that maintain data even after power is removed from the system. Nonvolatile memory can also be random access memory.

[0264] The non-volatile memory may be a local device directly coupled to the rest of the components in the data processing system. Non-volatile memory that is remote from the system may also be used, such as a network storage device coupled to the data processing system through a network interface such as a modem or Ethernet interface.

[0265] In the present disclosure, some functions and operations are described as being performed by software code or caused by software code to simplify the description. However, this expression is also used to specify that the functions are generated by a processor such as a microprocessor executing code / instructions.

[0266] Alternatively or in combination, the functions and operations as described herein may be implemented using dedicated circuitry with or without software instructions, such as using an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). Embodiments may be implemented using hard-wired circuitry without software instructions or in combination with software instructions. Thus, the techniques are not limited to any specific combination of hardware circuitry and software, nor to any particular source for the instructions executed by the data processing system.

[0267] While one embodiment may be implemented on a fully functional computer and computer system, various embodiments are capable of being distributed as a computing product in many forms and are applicable regardless of the particular type of machine or computer readable media actually used to achieve the distribution.

[0268] At least some aspects disclosed may be embodied at least in part in software. That is, these techniques may be implemented in a computer system or other data processing system in response to its processor, such as a microprocessor, executing sequences of instructions contained in a memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0269] The routines executed to implement the embodiments may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions referred to as a “computer program.” A computer program typically includes one or more sets of instructions at various times in various memories and storage devices in a computer, and when the instruction sets are read and executed by one or more processors in a computer, causes the computer to perform the necessary operations to perform elements related to various aspects.

[0270] Machine-readable media can be used to store software and data that, when executed by a data processing system, causes the system to perform various methods. Executable software and data can be stored in various locations including, for example, ROM, volatile RAM, nonvolatile memory, and / or cache memory. Portions of this software and / or data can be stored in any of these storage devices. In addition, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of data and instructions can be obtained from different centralized servers and / or peer-to-peer networks at different times and in different communication sessions or the same communication session. All data and instructions can be obtained before executing the application. Alternatively, portions of data and instructions can be obtained dynamically and in a timely manner when execution is required. Therefore, it is not required that data and instructions are all on machine-readable media at a specific time.

[0271] Examples of computer-readable media include, but are not limited to, non-transitory recordable and non-recordable types of media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, floppy and other removable disks, magnetic disk storage media, optical storage media (e.g., compact disk read-only memory (CDROM), digital versatile disk (DVD), etc.), etc. Computer-readable media can store instructions.

[0272] The instructions may also be embodied in digital and analog communication links for electrical, optical, acoustical or other forms of propagated signals such as carrier waves, infrared signals, digital signals, etc. However, propagated signals such as carrier waves, infrared signals, digital signals, etc. are not tangible machine-readable media and cannot be configured to store instructions.

[0273] Generally speaking, a machine-readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form accessible by a machine (e.g., a computer, a network device, a personal digital assistant, a manufacturing tool, any device with one or more processors, etc.).

[0274] In various embodiments, hard-wired circuits may be used in combination with software instructions to implement the technology.Thus, the technology is neither limited to any specific combination of hardware circuitry and software nor to any specific source for the instructions executed by the data processing system.

[0275] The above description and drawings are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details are not described to avoid obscuring the description. References to one or an embodiment in the present disclosure do not necessarily refer to the same embodiment; and such references mean at least one.

[0276] In the foregoing description, the present disclosure has been described with reference to specific exemplary embodiments of the present disclosure. It will be apparent that various modifications may be made without departing from the broader spirit and scope as set forth in the appended claims. Therefore, the description and drawings should be viewed in an illustrative rather than a restrictive sense.

Claims

1. A vehicle comprising: at least one microphone configured to generate a signal representative of audio content presented in a cabin of the vehicle; infotainment system, which accesses multiple sources of audio content; and an artificial neural network configured to receive input parameters related to audio controls in the vehicle and generate a prediction of an audio mode for adjusting settings of the infotainment system and a prediction of the likelihood that the predicted audio mode will be accepted by one or more occupants of the vehicle based on the input parameters as they vary over time, wherein the input parameters include data representative of the signal from the at least one microphone, and at least one operating parameter of the vehicle; Wherein the vehicle is configured to adjust a setting of the infotainment system based at least in part on the prediction generated by the artificial neural network in response to a determination that the likelihood is above a threshold. 2 . The vehicle of claim 1 , wherein the input parameters additionally include data from the infotainment system.

3. The vehicle of claim 2, wherein the at least one operating parameter of the vehicle comprises a speed of the vehicle traveling on a roadway, or a position of the vehicle, or any combination thereof.

4. The vehicle of claim 3, wherein the settings include selection of a source of audio content presented in the cabin of the vehicle. 5 . The vehicle of claim 4 , wherein the source is a station in a radio of the infotainment system, a mobile device connected to the infotainment system, or a file in a computer storage medium installed in the infotainment system.

6. The vehicle of claim 3, wherein the at least one operating parameter of the vehicle additionally comprises temperature, a window opening setting, a setting of a sunroof or skylight of the vehicle, or a setting of a convertible top of the vehicle, or any combination thereof.

7. The vehicle of claim 2, wherein the data from the infotainment system identifies the type, style of music, genre of music, or source of audio content currently being delivered by the infotainment system to the cabin of the vehicle, or any combination thereof.

8. A vehicle according to claim 7, wherein the prediction of the audio pattern includes a predicted audio content type, a predicted music style, a predicted music genre, or a predicted source, or any combination thereof; and the vehicle is further configured to select a source of audio content based on the predicted audio content type, the predicted music style, the predicted music genre, or the predicted source, or any combination thereof.

9. The vehicle of claim 8, wherein the vehicle is configured to implement the first adjustment to the setting of the infotainment system based at least in part on the prediction generated by the artificial neural network without requesting confirmation from an occupant in the vehicle.

10. The vehicle of claim 9, wherein the vehicle is configured to request confirmation from the occupant before implementing a second adjustment to the setting of the infotainment system based at least in part on the prediction generated by the artificial neural network. 11 . The vehicle of claim 1 , wherein the vehicle is configured to request approval from the one or more occupants of the vehicle to implement the adjustment in response to determining that the likelihood is above a threshold.

12. A method comprising: generating, by at least one microphone disposed in the vehicle, a signal representing audio content presented in a cabin of the vehicle, wherein the infotainment system accesses a plurality of audio content sources; In an artificial neural network, receiving input parameters related to audio control in the vehicle; generating, by the artificial neural network, a prediction of an audio mode for adjusting a setting of the infotainment system and a probability that the predicted audio mode is approved by at least one occupant of the vehicle based on the input parameters varying over time, wherein the input parameters include data representing the signal from the at least one microphone, data received from the infotainment system, and at least one operating parameter of the vehicle; and By the vehicle, a setting of the infotainment system is adjusted based at least in part on the prediction generated by the artificial neural network in response to the probability being greater than a threshold.

13. The method of claim 12, wherein the artificial neural network comprises a spiking neural network; and the method further comprises: In the vehicle, the artificial neural network is trained to predict an audio pattern corresponding to the signal representing audio content presented in the cabin of the vehicle in the absence of adjustments to settings of the infotainment system initiated by one or more occupants of the vehicle.

14. The method of claim 12, wherein the artificial neural network comprises a spiking neural network; and the method further comprises: In response to adjustments to settings of the infotainment system initiated by one or more occupants in the vehicle, the artificial neural network is trained in the vehicle to predict an audio pattern corresponding to the signal representing audio content presented in the cabin of the vehicle as a result of the adjustments.

15. An infotainment system comprising: at least one microphone configured to generate a signal representative of audio content presented in a cabin of a vehicle having a plurality of audio content sources; and an artificial neural network configured to receive input parameters related to audio controls in the vehicle and generate a prediction of an audio mode for adjusting settings of the infotainment system and a prediction of the likelihood that the predicted audio mode will be accepted by one or more occupants of the vehicle based on the input parameters as they vary over time, wherein the input parameters include data representative of the signal from the at least one microphone, and at least one operating parameter of the vehicle; Wherein the infotainment system is configured to adjust settings to change audio content presented in the cabin of the vehicle based at least in part on the prediction generated by the artificial neural network in response to a determination that the likelihood is above a threshold.

16. The infotainment system of claim 15, further comprising: A data storage device is configured to store model data of the artificial neural network and to calculate a prediction of the audio pattern in response to receiving the input parameters.

17. The infotainment system of claim 15, further configured to: providing a voice prompt to an occupant of the vehicle to adjust the setting based at least in part on the prediction generated by the artificial neural network; detecting a response made by the occupant in response to the voice prompt; calculating an adjusted probability that the occupant will accept an adjustment to the setting predicted by the artificial neural network; and The artificial neural network is trained to predict the adjusted probability based on the input parameters that vary over time.

Images