A neural network model fault monitoring method and device in an autonomous driving system

Abstract

A neural network model fault monitoring method and device in an automatic driving system, wherein the method comprises: obtaining a target output data set of a to-be-monitored neural network model in the automatic driving system (301), the target output data set comprising an output data set corresponding to each neural network layer in m neural network layers, wherein the to-be-monitored neural network model comprises M neural network layers, M is an integer greater than 1, and m is an integer greater than 1 and not greater than M; in the target output data set, a feature value set corresponding to each neural network layer is extracted (302); a relative entropy value between the feature value set and a first element set conforming to a preset probability distribution is calculated to obtain a relative entropy value set corresponding to the m neural network layers (303); and whether the to-be-monitored neural network model has a running fault is judged according to the relative entropy value set (304). Through the method, the real-time performance and accuracy of the fault monitoring of the to-be-monitored neural network model are improved, and the safety of the automatic driving vehicle is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a neural network model fault monitoring method and device in an automatic driving system. BACKGROUND

[0002] Due to the high complexity of computing platforms, artificial intelligence (AI) accelerators and other devices in an automatic driving system, the neural network model deployed on these devices is more susceptible to the influence of hardware failure and other factors when performing inference operations. Therefore, timely and accurate monitoring of whether the neural network model has a running fault is of great significance to ensuring the safety of an automatic driving vehicle. SUMMARY

[0003] In view of this, a neural network model fault monitoring method and device in an automatic driving system, a storage medium and a computer program product are proposed.

[0004] In a first aspect, an embodiment of the present application provides a neural network model fault monitoring method in an automatic driving system, which comprises: obtaining a target output data set of a to-be-monitored neural network model in an automatic driving system, the target output data set comprising a set of output data corresponding to each neural network layer in m neural network layers, wherein the to-be-monitored neural network model comprises M neural network layers, M is an integer greater than 1, and m is an integer greater than 1 and not greater than M; in the target output data set, a set of characteristic values corresponding to each neural network layer is extracted; a relative entropy value between the set of characteristic values and a first element set conforming to a preset probability distribution is calculated to obtain a set of relative entropy values corresponding to the m neural network layers; and whether the to-be-monitored neural network model has a running fault is determined according to the set of relative entropy values.

[0005] Based on the above technical scheme, based on the idea of the Monte Carlo method, the output data of each neural network layer is selectively sampled, part of the output data in the output data set is extracted as a characteristic value, and the distribution of the output data of each neural network layer is reflected by as few characteristic values as possible, thereby simplifying the calculation, saving the operation cost, and improving the operation efficiency; at the same time, by calculating the relative entropy value between the characteristic value set corresponding to each neural network layer and the first element set conforming to the preset probability distribution, a relative entropy value set is obtained, the data dimensionality is reduced, and the operation efficiency is further improved; thereby improving the real-time performance of fault monitoring, and realizing real-time monitoring of the running fault of the neural network model in the automatic driving system. At the same time, the relative entropy value is used to describe the distribution difference characteristics of the normal output data and the abnormal output data of each neural network layer, and the normal output data and the abnormal output data of each neural network layer are distinguished, so that whether the to-be-monitored neural network model has a running fault is more accurately judged through the relative entropy value set corresponding to the m neural network layers, and the accuracy of fault monitoring is improved; in addition, various running faults of the neural network model or running faults of various neural network models can be effectively monitored, and the application scope is wide.

[0006] According to the first aspect, in a first possible implementation manner of the first aspect, the extracting the characteristic value set corresponding to each neural network layer in the target output data set comprises: determining a first output data set with the smallest number of output data in the target output data set; and extracting the characteristic value set corresponding to each neural network layer in the output data set corresponding to each neural network layer according to the number of output data in the first output data set; wherein the number of characteristic values in the extracted characteristic value set corresponding to each neural network layer is less than or equal to the number of output data in the first output data set.

[0007] Based on the above technical scheme, considering that the neural network model in the automatic driving system is usually complex, the number of output data in the target output data set is large, and the characteristic value set corresponding to each neural network layer is adaptively extracted in the output data set corresponding to each neural network layer, the number of extracted characteristic values in each neural network layer is not greater than the number of output data in any neural network layer in the m neural network layers, thereby simplifying the operation cost, improving the subsequent processing efficiency, and meeting the real-time requirement of fault monitoring.

[0008] According to the first aspect, in a second possible implementation manner of the first aspect, the extracting the characteristic value set corresponding to each neural network layer in the target output data set comprises: extracting the characteristic value set corresponding to each neural network layer by taking the number of output data in the output data set corresponding to each neural network layer as a weight.

[0009] Based on the above technical solution, considering that the number of output data in different neural network layers may be different, the working state of the neural network model is also different; the weight of the number of output data corresponding to each neural network layer is used to allocate how much output data is extracted from each neural network layer, thereby realizing adaptive extraction of the feature value set corresponding to each neural network, and the extracted feature value set can more accurately reflect the distribution of the output data of each neural network layer. At the same time, through feature value extraction, the computational overhead is simplified, the subsequent processing efficiency is improved, and the real-time requirement of fault monitoring is met.

[0010] According to the first aspect or various possible implementation manners of the first aspect, in a third possible implementation manner of the first aspect, the determining whether the to-be-monitored neural network model has a running fault according to the relative entropy value set comprises: inputting the relative entropy value set into a preset classification model to determine whether the to-be-monitored neural network model has a running fault.

[0011] In some examples, the relative entropy value set is input into a preset classification model, the preset classification model classifies the relative entropy value set based on the relative entropy values between the feature value set extracted from the known normal output data and the element set conforming to the preset probability distribution, and the relative entropy values between the feature value set extracted from the abnormal output data and the element set conforming to the preset probability distribution, thereby accurately determining whether the to-be-monitored neural network model has a running fault.

[0012] According to the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, the preset classification model comprises a first classifier based on machine learning; the inputting the relative entropy value set into the preset classification model to determine whether the to-be-monitored neural network model has a running fault comprises: inputting the relative entropy value set into the first classifier to calculate the distance between the relative entropy value set and a plurality of relative entropy value sample sets; wherein the plurality of relative entropy value sample sets comprise a relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model has a fault and a relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model works normally; and determining whether the to-be-monitored neural network model has a running fault according to the distance between the relative entropy value set and the plurality of relative entropy value sample sets.

[0013] Based on the above technical solution, the first classifier based on machine learning is used, without pre-training, and the relative entropy value set can be automatically classified more conveniently and quickly according to the distance between the relative entropy value set and the plurality of relative entropy value sample sets, thereby determining in real time whether the to-be-monitored neural network model has a running fault.

[0014] In a sixth possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model is faulty includes relative entropy values between a first feature value sample set corresponding to each neural network layer in the m neural network layers and a second element set conforming to the preset probability distribution, the first feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is faulty; and the relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model is normal includes relative entropy values between a second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution, the second feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is normal.

[0015] According to the above technical solution, by using the second classifier based on deep learning, the accuracy of the relative entropy value set classification is effectively improved while the category to which the relative entropy value set belongs is determined in real time, so that whether the to-be-monitored neural network model is faulty or not is more accurately determined.

[0016] According to the fourth or fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model is faulty includes relative entropy values between a first feature value sample set corresponding to each neural network layer in the m neural network layers and a second element set conforming to the preset probability distribution, the first feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is faulty; and the relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model is normal includes relative entropy values between a second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution, the second feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is normal.

[0017] In a sixth possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model is faulty includes relative entropy values between a first feature value sample set corresponding to each neural network layer in the m neural network layers and a second element set conforming to the preset probability distribution, the first feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is faulty; and the relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model is normal includes relative entropy values between a second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution, the second feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is normal.

[0018] Based on the technical scheme, the output data of each neural network layer is selectively sampled based on the idea of the Monte Carlo method, part of the output data in the output data set is extracted as a characteristic value, and the distribution of the output data of each neural network layer is reflected by as few characteristic values as possible, thereby simplifying the calculation, saving the operation cost, and improving the operation efficiency. Meanwhile, by calculating the relative entropy value between the characteristic value set corresponding to each neural network layer and the first element set conforming to the preset probability distribution, a relative entropy value set is obtained, data dimensionality reduction is realized, and the operation efficiency is further improved. Therefore, the real-time performance of fault monitoring is improved, and real-time monitoring of the neural network model fault in the autonomous driving system is realized. Meanwhile, the relative entropy value is used to describe the distribution difference characteristics of the normal output data and the abnormal output data of each neural network layer, and the normal output data and the abnormal output data of each neural network layer are distinguished. Therefore, whether the to-be-monitored neural network model has a running fault is more accurately judged by the relative entropy value set corresponding to the m neural network layers, and the accuracy of fault monitoring is improved. In addition, various running faults of the neural network model or running faults of various neural network models can be effectively monitored, and the application scope is wide.

[0019] According to a first possible implementation manner of the second aspect, the processing module is further configured to: determine a first output data set with the smallest number of output data in the target output data set; and extract the characteristic value set corresponding to each neural network layer from the output data set corresponding to each neural network layer according to the number of output data in the first output data set.

[0020] Based on the technical scheme, considering that the neural network model in the autonomous driving system is usually complex and the number of output data in the target output data set is large, the characteristic value set corresponding to each neural network layer is adaptively extracted from the output data set corresponding to each neural network layer. The number of extracted characteristic values in each neural network layer is not greater than the number of output data in any neural network layer in the m neural network layers, thereby simplifying the operation cost, improving the subsequent processing efficiency, and meeting the real-time requirement of fault monitoring.

[0021] According to a second possible implementation manner of the second aspect, the processing module is further configured to: extract the characteristic value set corresponding to each neural network layer by taking the number of output data in the output data set corresponding to each neural network layer as a weight.

[0022] Based on the above technical solution, considering that the number of output data in different neural network layers is also different, the working state of the neural network model is also different; according to the weight of the number of output data corresponding to each neural network layer, how much output data each neural network layer extracts is allocated, thereby realizing adaptive extraction of the feature value set corresponding to each neural network, and the extracted feature value set can more accurately reflect the distribution of the output data of each neural network layer. At the same time, through feature value extraction, the computational overhead is simplified, the subsequent processing efficiency is improved, and the real-time requirement of fault monitoring is met.

[0023] According to the second aspect or various possible implementation manners of the second aspect, in a third possible implementation manner of the second aspect, the processing module is further configured to input the relative entropy value set into a preset classification model to determine whether the to-be-monitored neural network model has a running fault.

[0024] In some examples, the relative entropy value set is input into a preset classification model, the preset classification model classifies the relative entropy value set based on the relative entropy values between the feature value set extracted from the known normal output data and the element set conforming to the preset probability distribution, and the relative entropy values between the feature value set extracted from the abnormal output data and the element set conforming to the preset probability distribution, thereby accurately determining whether the to-be-monitored neural network model has a running fault.

[0025] According to the third possible implementation manner of the second aspect, in a fourth possible implementation manner of the second aspect, the preset classification model includes a first classifier based on machine learning; and the processing module is further configured to input the relative entropy value set into the first classifier to calculate the distance between the relative entropy value set and a plurality of relative entropy value sample sets; wherein the plurality of relative entropy value sample sets include a relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model has a fault and a relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model works normally; and according to the distance between the relative entropy value set and the plurality of relative entropy value sample sets, it is determined whether the to-be-monitored neural network model has a running fault.

[0026] Based on the above technical solution, by using the first classifier based on machine learning, without pre-training, the relative entropy value set can be automatically classified more conveniently and quickly according to the distance between the relative entropy value set and the plurality of relative entropy value sample sets, thereby determining in real time whether the to-be-monitored neural network model has a running fault.

[0027] In a sixth possible implementation manner of the second aspect, according to the fourth or fifth possible implementation manner of the second aspect, when the to-be-monitored neural network model is faulty, the relative entropy value sample set corresponding to the m neural network layers includes relative entropy values between a first feature value sample set corresponding to each neural network layer in the m neural network layers and a second element set conforming to the preset probability distribution, the first feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is faulty; and when the to-be-monitored neural network model is normal, the relative entropy value sample set corresponding to the m neural network layers includes relative entropy values between a second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution, the second feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is normal.

[0028] According to the above technical solution, by using the second classifier based on deep learning, the accuracy of the relative entropy value set classification is effectively improved while the category to which the relative entropy value set belongs is determined in real time, so that whether the to-be-monitored neural network model is faulty can be more accurately determined.

[0029] According to the fourth or fifth possible implementation manner of the second aspect, in a sixth possible implementation manner of the second aspect, when the to-be-monitored neural network model is faulty, the relative entropy value sample set corresponding to the m neural network layers includes relative entropy values between a first feature value sample set corresponding to each neural network layer in the m neural network layers and a second element set conforming to the preset probability distribution, the first feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is faulty; and when the to-be-monitored neural network model is normal, the relative entropy value sample set corresponding to the m neural network layers includes relative entropy values between a second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution, the second feature value sample set being extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is normal.

[0030] In a third aspect, embodiments of the present application provide a neural network model fault monitoring apparatus in an automatic driving system, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-mentioned first aspect or one or more of the first aspect when executing the instructions.

[0031] In a fourth aspect, embodiments of the present application provide a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions are executed by a processor to implement the first aspect or one or more of the first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, causes the computer to perform the neural network fault monitoring method in the autonomous driving system of the first aspect or one or more of the first aspect.

[0033] The technical effects of the third aspect to the fifth aspect are described above in the first aspect or the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 FIG. 1 shows an architecture schematic diagram of an autonomous driving system according to an embodiment of the present application;

[0035] Figure 2 FIG. 2 shows a schematic diagram of monitoring a neural network model according to an embodiment of the present application;

[0036] Figure 3 FIG. 3 shows a flowchart of a neural network model fault monitoring method in an autonomous driving system according to an embodiment of the present application;

[0037] Figure 4 FIG. 4 shows a flowchart of a method of obtaining a relative entropy value sample set according to an embodiment of the present application;

[0038] Figure 5 FIG. 5 shows a schematic diagram of a neural network model fault monitoring method in an autonomous driving system according to an embodiment of the present application;

[0039] Figure 6 FIG. 6 shows a structure schematic diagram of a neural network model fault monitoring device in an autonomous driving system according to an embodiment of the present application;

[0040] Figure 7 FIG. 6 shows a structure schematic diagram of a neural network model fault monitoring device in an autonomous driving system according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings represent elements or components serving the same function or having the same properties. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0042] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0043] In order to better understand the solutions of the embodiments of the present application, the related terms and concepts that the embodiments of the present application can involve are introduced as follows.

[0044] 1、Probability distribution

[0045] Probability distribution refers to the probability law of the value of a random variable. If the result of an experiment is represented by the value of a random variable, the probability distribution of the random experiment is the probability distribution of the random variable, i.e. the possible values of the random variable and the probability of obtaining the corresponding values. According to the different types of random variables, probability distribution can be divided into different forms, such as Gaussian distribution (also known as normal distribution), binomial distribution, Poisson distribution, uniform distribution, Bernoulli distribution, Laplace distribution, exponential distribution, gamma distribution, beta distribution, and polynomial distribution.

[0046] 2、Relative entropy

[0047] Relative entropy, also known as Kullback-Leibler divergence (KLD), is a measure of the asymmetry of the difference between two probability distributions P and Q. Relative entropy can measure the distance between two probability distributions. When the two probability distributions are the same, their relative entropy is zero, and when the difference between the two probability distributions increases, their relative entropy also increases accordingly.

[0048] Typically, P represents the true distribution of data, and Q represents the theoretical distribution of data, the estimated model distribution, or the approximate distribution of P. The relative entropy of P and Q is shown in the following formula (1),

[0049]

[0050] where P(i) represents the i-th element in P, Q(i) represents the i-th element in Q, and ln(·) represents the calculation of natural logarithm.

[0051] 3、Monte Carlo method

[0052] The Monte Carlo method, also known as statistical simulation method or statistical test method, is a numerical simulation method that takes probability phenomena as the research object. In statistical simulation, a probability model similar to the system performance is constructed, and random tests are conducted to simulate the random characteristics of the system.

[0053] 4、Classifier

[0054] Many neural network models have a classifier at the end, which is used to classify the input data. The classifier is generally composed of a fully connected layer and a softmax function (which can be called a normalized exponential function), and can output different categories or probabilities of different categories according to the input data.

[0055] 5. Multi-layer perceptron (MLP)

[0056] MLP is a kind of artificial neural network with forward structure, which maps a set of input vectors to a set of output vectors. MLP can be regarded as a directed graph, and the basic structure of multi-layer perceptron is composed of multiple node layers: input layer, intermediate hidden layer and output layer, and each node layer is fully connected to the next node layer. Except for input nodes, each node is a neuron with a nonlinear activation function; MLP follows the principle of human nervous system, learns and makes data prediction, and its main advantage lies in the ability to quickly solve complex problems.

[0057] 6. k-nearest neighbor (KNN)

[0058] The basic logic of KNN algorithm is as follows: classification is performed by measuring the distance between different characteristic values, and the algorithm only depends on the class of the nearest sample or several samples to determine the class to which the sample to be classified belongs. The basic idea is: if the majority of the k most similar (i.e. the nearest in the feature space) samples of a sample in the feature space belong to a certain class, then the sample also belongs to this class, where K is usually an integer not greater than 20. In KNN algorithm, the selected neighbors are all correctly classified samples.

[0059] 7. Neural network model

[0060] A neural network model is an operational model composed of a large number of nodes (or neurons) connected to each other. Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight, which is equivalent to the memory of the artificial neural network. The output of the neural network model is different depending on the connection mode of the neural network model, the weight value and the activation function. The neural network model itself is usually an approximation of an algorithm or function in nature, or an expression of a logic strategy. The neural network model usually includes multiple neural network layers, wherein each neural network layer can include one or more nodes. The neural network model can be divided into a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), etc. Among them, the deep neural network, also known as a multi-layer neural network, can be understood as a neural network model with many hidden layers, and the internal neural network layers can be divided into three categories: input layer, hidden layer and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the number of layers in between is the hidden layer; the layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. The convolutional neural network is a neural network model with a convolutional structure; the convolutional neural network includes a feature extractor composed of a convolutional layer and a subsampling layer, which can be regarded as a filter; the convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input data; in the convolutional layer, a neuron can be connected only to part of the adjacent layer neurons; a convolutional layer usually includes several feature planes, each of which can be composed of some rectangularly arranged neurons; the neurons in the same feature plane share weights, i.e., share convolution kernels.

[0061] 8. Neural network model failure monitoring

[0062] The neural network model fault monitoring refers to monitoring possible running faults of the neural network model in a reasoning operation process of the neural network model. The running faults can include a fault caused by hardware failure in a device deploying the neural network model, or an error reasoning result of the neural network model caused by an abnormal input, and the like. The fault caused by the hardware failure is generally collectively referred to as a soft error. Common soft errors can be divided into transient errors and permanent errors. The transient error is related to hardware failure caused by sudden changes in external environment such as radiation and temperature, and mutual interference of the hardware itself. The transient error is characterized in that the error will disappear after appearing at a certain time period. A common transient error is bit flip. Common permanent errors are stuck-at-0 and stuck-at-one, which are related to hardware failure caused by open circuit and short circuit of the hardware respectively. The permanent error is characterized in that the error will be retained at the occurrence position for a long time.

[0063] In the related art, a neural network model fault monitoring is performed by using a redundancy design, such as a triple modular redundancy (TMR) design. In the TMR design, multiple modules of the same structure are connected in parallel to perform the same function. A pre-prepared query table is used to collect as many neuron weight values as possible in an error-free state of the neural network model. In a reasoning operation process of the neural network model, if a weight value is not in the query table, the weight value is considered to be abnormal, that is, the neural network model has a running fault, and a weight value switching state is started. The weight value in the error neuron is distributed to other neurons, which plays a role of replacing the error neuron with other neurons. Alternatively, a symptom-based error detector (SED) is used to perform the neural network model fault monitoring. The SED determines whether the neural network model has a running fault according to the size of an output value of each neural network layer in the neural network model. A plurality of output values of each neural network layer are collected in an error-free state of the neural network model. A reasonable value range of the output value corresponding to each neural network layer is determined according to the collected plurality of output values. In the reasoning operation process of the neural network model, if the output value of each neural network layer exceeds 1.1 times of the reasonable value range, the output value is considered to be incorrect, and it is determined that the neural network model has a running fault.

[0064] Both of the above two ways of monitoring the faults of the neural network model have their own limitations. Among them, the way of adopting redundancy design is only applicable to multi-layer perception network, i.e. the input data is not dimensionally reduced by convolution, pooling and other means; when monitoring the possible faults of the neural network model with convolution layer and pooling layer, this way cannot effectively monitor the convolution layer or the pooling layer due to the design of the weight value lookup table; and this way can only monitor part of the stuck-at-one and bitflip errors; in addition, for a more complex neural network model, the operation cost of this way is large, and it cannot guarantee the real-time performance of fault monitoring. For example, in the input layer of the neural network model Alexnet, there are more than 3000 weights, and it is difficult to collect and query the weight values of all neurons in Alexnet, and the operation cost is large. When using the lookup table to monitor the faults of Alexnet, due to the large number of weights and the large number of weight values in the lookup table, the query rate is slow, and it cannot be applied to scenarios such as automatic driving systems that require high real-time performance of fault monitoring. The way of adopting SED extracts the maximum output value and the minimum output value of each neural network layer by using a simple enumeration algorithm to obtain the reasonable value range of the output value corresponding to each neural network layer. For a more complex neural network model, the operation cost of this way is very large. For example, in Alexnet, considering the output values of each neuron, pooling layer and fully connected layer, there will be more than 100,000 output values of single neurons in total. Taking the convolution layer as an example, there are more than 15000 output values. Therefore, collecting the output values of each neural network layer will bring a huge operation cost. When monitoring the faults of Alexnet, due to the large number of output values, the delay of fault monitoring is caused, and it cannot be applied to scenarios such as automatic driving systems that require high real-time performance of fault monitoring. In addition, this way can only monitor the occurrence of transient errors, and for zero and one, since the maximum output value of the hidden layer does not change significantly under these two conditions, permanent errors cannot be monitored.

[0065] Due to the limitations of the above two ways of monitoring the failure of the neural network model, the embodiment of the present application provides a neural network model failure monitoring method (see the detailed description below), which can be applied to a scenario configured with a neural network model, for example, an autonomous vehicle, a vehicle-mounted device or a vehicle-mounted system (such as an autonomous driving system (ADS) or an advanced driver assistant system (ADAS) deployed with a neural network model, a large-scale deep learning training server, an Internet of Things (IoT) device using a neural network model for object recognition, semantic recognition, etc., a security device using a neural network model for vehicle detection, object detection, etc. The neural network model failure monitoring method provided by the embodiment of the present application can accurately monitor various running failures of various neural network models configured in the above scenarios; especially for scenarios such as autonomous driving systems that require high real-time fault monitoring, real-time fault monitoring can be achieved to meet the real-time requirements of scenarios such as autonomous driving.

[0066] For ease of description, the neural network model failure monitoring method provided by the embodiment of the present application is exemplarily described by taking the fault monitoring of the neural network model in the autonomous driving system as an example.

[0067] Figure 1 An architecture schematic diagram of an autonomous driving system according to an embodiment of the present application is shown; as shown in Figure 1 The autonomous driving system can include a perception module (perception layer), a planning and decision module (planning&decision), and a transmission control module (motion controller).

[0068] The perception module is configured to perceive the environment around the vehicle or in the vehicle cabin, and can integrate data collected by vehicle-mounted sensors such as cameras, lidar, millimeter wave radar, ultrasonic radar, light sensors, etc. The perception module can perceive the environment around the vehicle or in the vehicle cabin, and can transmit the perception result to the planning and decision module. For example, the data collected by the vehicle-mounted sensors can include video streams, point cloud data of radar, or analyzed structured information of people, vehicles, and objects, such as position, speed, steering angle, size, etc. The perception module can process the data collected by the vehicle-mounted sensors through a neural network model to realize environment perception. For example, the neural network model can be deployed in a vehicle-mounted computing platform or an AI accelerator or other processing device. As an example, the perception module can obtain images of the environment around the vehicle collected by a vehicle-mounted camera, process the images using a deep neural network model for image recognition, and thus identify objects such as pedestrians, lane lines, vehicles, obstacles, traffic lights, etc.

[0069] The planning and decision module is configured to analyze and make decisions based on the perception result generated by the perception module, plan a control set that satisfies certain constraints (e.g., vehicle dynamics constraints, collision avoidance, passenger comfort, etc.), and transmit the control set to the transmission control module. As an example, the planning and decision module can use a neural network model for trajectory generation to process the perception result and the constraints, and generate a control set. The neural network model can be deployed in a vehicle-mounted computing platform or an AI accelerator or other processing device.

[0070] The transmission control module is configured to control the vehicle to travel according to the control set generated by the planning and decision module. For example, based on the control set, the transmission control module can generate control signals such as steering wheel angle, speed, acceleration, etc. in combination with vehicle dynamics information, and control the vehicle-mounted steering system or engine to execute the control signals, thereby controlling the vehicle to travel.

[0071] Exemplarily, the automatic driving system can further include other functional modules; for example, a positioning module, an interaction module, a communication module, etc. (not shown in the figure), without limitation. Among them, the positioning module can be used to provide the position information of the vehicle, and can also provide the attitude information of the vehicle. Exemplarily, the positioning module can include a satellite navigation system (Global Navigation Satellite System, GNSS), an inertial navigation system (Inertial Navigation System, INS), etc., and can be used to determine the position information of the vehicle. The interaction module can be used to send information to the driver and receive the instructions of the driver. The communication module can be used for communication between the vehicle and other devices, wherein the other devices can include a mobile terminal, a cloud device, other vehicles, a roadside device, etc., and can be connected through 2G / 3G / 4G / 5G, Bluetooth, frequency modulation (frequency modulation, FM), wireless local area networks (wireless local area networks, WLAN), long time evolution (long time evolution, LTE), vehicle to anything communication (vehicle to everything, V2X), vehicle to vehicle communication (Vehicle to Vehicle, V2V), long time evolution vehicle (long time evolution vehicle, LTE-V), etc. Wireless communication connection.

[0072] The neural network model fault monitoring method provided by the embodiment of the application can be executed by the neural network model fault monitoring device. As an example, the neural network model fault monitoring method is executed by the neural network model fault monitoring device to monitor the deep neural network model for image recognition in the perception module of the automatic driving system. Figure 1 The neural network model fault monitoring method provided by the embodiment of the application can be executed by the neural network model fault monitoring device. As an example, the neural network model fault monitoring method is executed by the neural network model fault monitoring device to monitor the deep neural network model for image recognition in the perception module of the automatic driving system. Figure 2 An example of monitoring the deep neural network model for image recognition in the perception module of the automatic driving system is shown in the schematic diagram of monitoring the neural network model according to an embodiment of the application; as shown in the figure, Figure 2 As shown, the neural network model fault monitoring device can obtain the intermediate data generated in the process of identifying a frame of image by the deep neural network model for image recognition in the perception module of the automatic driving system, execute the neural network model fault monitoring method in the embodiment of the application (for details, see the following text), and monitor the deep neural network model in real time and accurately, and feed back the fault monitoring result to the perception module in real time, so that the perception module can judge whether to pass the current recognition result to the planning and decision module. For example, the perception module can be fed back that the neural network model is working normally, so that the perception module can pass the recognition result of the frame of image to the planning and decision module; or the perception module can be fed back that the neural network is faulty, so that the perception module discards the recognition result of the frame of image.

[0073] The embodiments of the present application do not limit the type of the neural network model fault monitoring device.

[0074] Exemplarily, the neural network model fault monitoring device can be independently arranged, or integrated in other devices, or realized by software or a combination of software and hardware.

[0075] Exemplarily, the neural network model fault monitoring device can be an autonomous vehicle, or other components in the autonomous vehicle. The neural network model fault monitoring device includes but is not limited to a vehicle terminal, a vehicle controller, a vehicle module, a vehicle module, a vehicle component, a vehicle chip, a vehicle unit, a vehicle radar, a vehicle camera, and the like. As an example, the neural network model fault monitoring device can be integrated in a vehicle computing platform or an AI accelerator of the autonomous vehicle.

[0076] Exemplarily, the neural network model fault monitoring device can also be an intelligent terminal with data processing capability other than the autonomous vehicle, or a component or chip arranged in the intelligent terminal.

[0077] Exemplarily, the neural network model fault monitoring device can be a general-purpose device or a special-purpose device. For example, the device can also be a desktop computer, a laptop computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, an embedded device, or other devices with data processing functions, or a component or chip in these devices.

[0078] Exemplarily, the neural network model fault monitoring device can also be a chip or processor with processing function. The fault monitoring device can include multiple processors. The processor can be a single-CPU processor or a multi-CPU processor.

[0079] It should be noted that the above application scenarios described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems in other similar or new scenarios.

[0080] The neural network model fault monitoring method in the autonomous driving system provided by the embodiments of the present application will be described in detail below.

[0081] Figure 3 A flowchart of a neural network model fault monitoring method in an autonomous driving system according to an embodiment of the present application is shown. The method can be performed by the neural network model fault monitoring device described above.Figure 2 The neural network model fault monitoring apparatus executes, as shown in Figure 3 may include the following steps:

[0082] Step 301, obtaining a target output data set of a neural network model to be monitored in an automatic driving system.

[0083] The neural network model to be monitored can be any neural network model in the automatic driving system, for example, can be a deep neural network model configured in a perception module for image recognition or a neural network model for speech recognition, etc., can also be a neural network model configured in a planning and decision module for generating a control set, etc.

[0084] It should be noted that the type of neural network model is not limited in the embodiments of the present application, for example, can be a deep neural network, a convolutional neural network, a recurrent neural network, etc.

[0085] The target output data set can include a set of output data corresponding to each neural network layer in m neural network layers, the neural network model to be monitored includes M neural network layers, M is an integer greater than 1, and m is an integer greater than 1 and not greater than M. For example, for any neural network layer, the set of output data corresponding to the neural network layer includes the data output by all nodes in the neural network layer during the inference process of the neural network model to be monitored. The specific value of m can be pre-set according to the size of the neural network model to be monitored and / or the amount of actual operation resources, etc.; for example, the value of m can be set to be close to M, that is, as many sets of output data corresponding to neural network layers as possible are obtained, so as to improve the monitoring accuracy, for example, when m and M take the same value, it means that the neural network model fault monitoring apparatus obtains all sets of output data corresponding to neural network layers in the neural network model to be monitored; m can also be set to a smaller value, that is, a small amount of sets of output data corresponding to neural network layers are obtained, so as to save operation resources, improve processing efficiency, and better meet the real-time requirement.

[0086] As an example, the neural network model to be monitored can be a convolutional neural network for image recognition in the perception module of the automatic driving system, which can include a plurality of convolutional layers, pooling layers, fully connected layers, etc. The image collected by the perception module is input into the convolutional neural network, and after being processed by the convolutional layer, the pooling layer, and the fully connected layer, the image recognition result is output. Each convolutional layer can include one or more convolutional kernels, each convolutional kernel can extract a corresponding feature map, and the target output data set of the convolutional neural network can include the feature maps extracted by all convolutional kernels in each convolutional layer.

[0087] Step 302, extracting the feature value set corresponding to each neural network layer in the target output data set.

[0088] In the feature value set corresponding to any neural network layer, one or more feature values corresponding to the neural network layer can be included. For example, for any neural network layer in the m neural network layers, the output data in the output data set corresponding to the neural network layer can be extracted as the feature value, thereby obtaining the feature value set corresponding to the neural network layer. The number of extracted output data can be pre-set according to requirements. For example, the number of output data extracted by different neural network layers can be the same or different, which is not limited. This step can be understood as the extraction of feature engineering, and the output data as the feature value is extracted as little as possible to reflect the distribution of the output data of each neural network layer as comprehensively as possible.

[0089] For example, for any neural network layer in the m neural network layers, the output data in the output data set corresponding to the neural network layer can be extracted as the feature value in a pre-set probability distribution manner, thereby obtaining the feature value set corresponding to the neural network layer; for example, part of the output data in the output data set corresponding to the neural network layer can be extracted as the feature value in a Gaussian distribution manner, thereby obtaining the feature value set corresponding to the neural network layer.

[0090] The possible implementation ways of extracting the feature value set corresponding to each neural network layer are illustrated below.

[0091] Method one, determining a first output data set with the smallest number of output data in the target output data set; according to the number of output data in the first output data set, extracting the feature value set corresponding to each neural network layer in the output data set corresponding to each neural network layer; wherein the number of feature values in the extracted feature value set corresponding to each neural network layer is less than or equal to the number of output data in the first output data set.

[0092] For example, the number of output data to be extracted in each neural network layer can be determined according to the number of output data in the first output data set, and then the number of output data in each neural network layer can be extracted as the feature value, thereby obtaining the feature value set corresponding to each neural network layer.

[0093] Considering that the neural network model in the automatic driving system is usually complex and the number of output data in the target output data set is large, in the manner, the feature value set corresponding to each neural network layer is adaptively extracted in the output data set corresponding to each neural network layer, and the number of data of the feature values extracted in each neural network layer is not greater than the number of output data in any neural network layer of the m neural network layers, so as to simplify the operation cost, improve the subsequent processing efficiency, and meet the real-time requirement of fault monitoring.

[0094] As an example, a sampling coefficient can be preset, and the number of output data to be extracted in each neural network layer is determined according to the sampling coefficient and the number of output data in the first output data set; for example, the number of output data to be extracted in each neural network layer n can be determined by the following formula (2):

[0095] n = a * n tmp ..................................(2)

[0096] In formula (2), n tmp represents the number of output data in the first output data set, and a represents the sampling coefficient, and the value range of a is [0, 1].

[0097] The sampling coefficient a is used to balance the complexity and accuracy of fault monitoring of the neural network model to be monitored, and the specific value of the sampling coefficient can be set according to actual requirements; for example, in the case of high monitoring accuracy requirement, a is set to a high value, that is, for each neural network layer, a large number of output data in the corresponding output data set are extracted as the feature values corresponding to the neural network layer; in the case of not too high monitoring accuracy requirement, a is set to a small value, that is, for each neural network layer, a small number of output data in the corresponding output data set are extracted as the feature values corresponding to the neural network layer, so as to save operation resources, improve processing efficiency, and better meet the real-time requirement.

[0098] For example, a can be 10%. For example, when the value of a*n tmp is not an integer, a*n tmp is rounded down, so as to obtain n.

[0099] In formula (3), n tmp can be determined by the following formula (3):

[0100] n tmp = min i∈m φ(i)...................(3)

[0101] In formula (3), φ(i) represents the number of output data in the output data set corresponding to the i-th neural network layer in the m neural network layers.

[0102] In this way, the number of output data to be extracted by each neural network layer, i.e., the number of feature values in the feature value set, can be determined according to the above formula (2) and formula (3). As an example, 10% of the total number of output data contained in the first output data set with the smallest number of output data can be taken as the number of output data to be extracted by each neural network layer, thereby simplifying the computational overhead and improving the efficiency of subsequent processing.

[0103] Method two: taking the number of output data in the output data set corresponding to each neural network layer as a weight, extracting the feature value set corresponding to each neural network layer.

[0104] It is considered that the number of output data in different neural network layers is also different, and the influence on the working state of the neural network model is also different; therefore, the number of feature values extracted by each neural network layer can be appropriately changed. In this method, the number of output data extracted by each neural network layer can be allocated according to the weight of the number of output data corresponding to each neural network layer, i.e., the more the number of output data corresponding to the neural network layer, the more the number of output data extracted as feature values; correspondingly, the less the number of output data corresponding to the neural network layer, the less the number of output data extracted as feature values; thereby realizing adaptive extraction of the feature value set corresponding to each neural network, and the extracted feature value set can more accurately reflect the distribution of the output data of each neural network layer. At the same time, through feature value extraction, the computational overhead is simplified, the efficiency of subsequent processing is improved, and the real-time requirement of fault monitoring is met.

[0105] Step 303: calculating the relative entropy value between the feature value set corresponding to each neural network layer and the first element set conforming to the preset probability distribution, to obtain a set of relative entropy values corresponding to the m neural network layers.

[0106] The first element set can include a plurality of elements conforming to the preset probability distribution, and the first element set can be real-time generated or pre-stored; as an example, a preset number of random numbers conforming to the preset probability distribution can be generated in real time, and the preset number of random numbers form the first element set; as an example, the preset probability distribution can be a Gaussian distribution.

[0107] Exemplarily, for any one of the m neural network layers, a relative entropy value between the feature value set corresponding to the neural network layer and the first element set conforming to the preset probability distribution can be obtained, the relative entropy value being a real number, and a value of the real number representing a difference between a distribution composed of the feature values in the feature value set corresponding to the neural network layer and the preset probability distribution. In this way, by traversing all the neural network layers in the m neural network layers, the relative entropy values between the neural network layers and the first element set are calculated, that is, a plurality of real numbers are obtained, so as to obtain a relative entropy value set; wherein each relative entropy value in the relative entropy value set can represent a difference between a distribution composed of the feature values in the feature value set corresponding to each of the m neural network layers and the preset probability distribution. At the same time, by using the feature value set corresponding to each of the neural network layers, a relative entropy value set is obtained, data dimensionality reduction is achieved, and the operation efficiency is further improved.

[0108] In step 304, whether the to-be-monitored neural network model has a running fault is judged according to the relative entropy value set corresponding to the m neural network layers.

[0109] The relative entropy value between the feature value set extracted from the normal output data of each neural network layer in the inference process of the normal neural network model and the first element set conforming to the preset probability distribution can represent the difference between the normal output data of each neural network layer and the first element set. The relative entropy value between the feature value set extracted from the abnormal output data of each neural network layer in the inference process of the failed neural network model and the first element set conforming to the preset probability distribution can represent the difference between the abnormal output data of each neural network layer and the first element set. Since the normal output data of each neural network layer is different from the abnormal output data of each neural network layer, the relative entropy value between the normal output data of each neural network layer and the first element set is different from the relative entropy value between the abnormal output data of each neural network layer and the first element set. Therefore, the relative entropy value can be used to distinguish the normal output data of each neural network layer in the inference process of the normal neural network model from the abnormal output data of each neural network layer in the inference process of the failed neural network model. In addition, the data amount in the output data set (for example, normal output data or abnormal output data) of each neural network layer is usually large, that is, the output data set is widely distributed in the data space. Different relative entropy values are used to distinguish different output data sets, that is, the relative entropy value has a corresponding relationship with the output data set widely distributed in the data space, so as to pull apart the differences of different output data sets in the data space through different relative entropy values and reduce the coupling degree of different output data sets. In this step, instead of directly judging whether the to-be-monitored neural network model has a running failure according to the output data of each neural network layer in the inference process of the neural network model, the relative entropy value set corresponding to the m neural network layers is used to distinguish the normal output data and the abnormal output data of each neural network layer, so as to more accurately judge whether the to-be-monitored neural network model has a running failure. For example, if the difference between the normal output data of each neural network layer and the abnormal output data of each neural network layer is small, the difference between the two is not easy to distinguish directly. However, the relative entropy value between the feature value set extracted from the normal output data and the first element set conforming to the preset probability distribution is different from the relative entropy value between the feature value set extracted from the abnormal output data and the first element set conforming to the preset probability distribution. The normal output data and the abnormal output data are distinguished through the relative entropy value, so as to accurately judge whether the to-be-monitored neural network model has a running failure.

[0110] In a possible implementation, the step can include: inputting the relative entropy value set corresponding to the m neural network layers into a preset classification model to judge whether the to-be-monitored neural network model has a running failure.

[0111] Exemplarily, the preset classification model can automatically classify the set of relative entropy values according to the size of each relative entropy value in the set of relative entropy values, and accurately determine the category to which the set of relative entropy values belongs; wherein the category to which the set of relative entropy values belongs can include normal working of the neural network model to be monitored and failure of the neural network model to be monitored; exemplarily, the set of relative entropy values is input into the preset classification model, the preset classification model classifies the set of relative entropy values based on the relative entropy values between the set of feature values extracted based on the known normal output data and the set of elements conforming to the preset probability distribution, and the relative entropy values between the set of feature values extracted from the abnormal output data and the set of elements conforming to the preset probability distribution, so as to accurately determine whether the neural network model to be monitored has a running failure.

[0112] Exemplarily, the preset classification model can include a first classifier based on machine learning or a second classifier based on deep learning, etc.; for example, the first classifier can be KNN, and the second classifier can be MLP, etc.

[0113] The neural network model failure monitoring method in the automatic driving system provided by the embodiments has the characteristics of small operation overhead, high real-time performance, high accuracy, wide application range, etc.

[0114] In the embodiments, considering the complexity of the neural network model in the automatic driving system, the number of neural network layers contained is usually large, and the number of corresponding output data is large, therefore, based on the idea of the Monte Carlo method, the output data of each neural network layer is selectively sampled, and part of the output data in the output data set is extracted as a feature value, the distribution of the extracted feature value can be used as an estimate of the distribution of the output data of each neural network layer in the target output data set, that is, the distribution of the output data of each neural network layer is reflected by as few feature values as possible, thereby simplifying the calculation, saving the operation overhead, and improving the operation efficiency; at the same time, by calculating the relative entropy values between the set of feature values corresponding to each neural network layer and the first set of elements conforming to the preset probability distribution, a set of relative entropy values is obtained, data dimensionality reduction is realized, and the operation efficiency is further improved; thereby the real-time performance of the failure monitoring is improved, and the real-time monitoring of the neural network model failure in the automatic driving system is realized.

[0115] In the embodiments of the present application, the relative entropy values are used to describe the distribution difference characteristics of the normal output data and the abnormal output data of each neural network layer, and the normal output data and the abnormal output data of each neural network layer are distinguished through the relative entropy value set corresponding to the m neural network layers, so that whether the to-be-monitored neural network model has a running fault can be more accurately judged according to the relative entropy value set, and the accuracy of fault monitoring is improved. For example, compared with the fault monitoring method of SED, the accuracy of the fault monitoring of Alexnet is greatly improved when 500 errors occur in Alexnet.

[0116] In the embodiments of the present application, various running faults of neural network models or various running faults of neural network models can be effectively monitored, and the application range is wide. For example, the running faults of various neural network models such as deep neural network models and convolutional neural network models can be monitored. For another example, the running faults of the to-be-monitored neural network model caused by hardware failure in the equipment such as the vehicle-mounted computing platform or the AI accelerator deploying the to-be-monitored neural network model in the automatic driving system can be monitored in real time, including transient faults and permanent faults. The running faults of the to-be-monitored neural network model caused by abnormal input in the automatic driving system can also be monitored in real time, so as to improve the safety of the vehicle-mounted computing platform or the AI accelerator. In addition, the range of the neural network layer that may have a fault can be determined, that is, one or more neural network layers in the m neural network layers that cause the to-be-monitored neural network model to have a running fault can be determined.

[0117] The possible implementation manner of judging whether the to-be-monitored neural network model has a running fault according to the relative entropy value set in the above step 304 will be described below.

[0118] In the first mode, taking the preset classification model as the first classifier based on machine learning as an example, the relative entropy value set can be input into the first classifier, and the distance between the relative entropy value set and a plurality of relative entropy value sample sets can be calculated. Whether the to-be-monitored neural network model has a running fault can be judged according to the distance between the relative entropy value set and the plurality of relative entropy value sample sets.

[0119] The plurality of relative entropy value sample sets can include a relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model has a fault and a relative entropy value sample set corresponding to the m neural network layers when the to-be-monitored neural network model works normally.

[0120] Exemplarily, the plurality of relative entropy value sample sets can be pre-sampled, i.e., the category to which each relative entropy value sample set belongs is known, where the category can be divided into a normal working state of the neural network model to be monitored and a fault state of the neural network model to be monitored. The distance between the relative entropy value set and the plurality of relative entropy value sample sets can represent the difference between the relative entropy value set and each relative entropy value sample set in the plurality of relative entropy value sample sets; for example, if the distance between the relative entropy value set and a certain relative entropy value sample set is larger, it means that the difference between the relative entropy value set and the relative entropy value sample set is larger, and accordingly, the possibility that the relative entropy value set and the relative entropy value sample set belong to the same category is lower. If the distance between the relative entropy value set and a certain relative entropy value sample set is smaller, it means that the difference between the relative entropy value set and the relative entropy value sample set is smaller, and accordingly, the relative entropy value set and the relative entropy value sample set are more likely to belong to the same category.

[0121] Exemplarily, the relative entropy value set can be input into the first classifier, and the first classifier calculates the distance between the relative entropy value set and the plurality of relative entropy value sample sets, so that the relative entropy value sample sets of different categories can be divided in the feature space. It can be considered that the relative entropy value set and one or more relative entropy value sample sets closest to the relative entropy value set are more likely to belong to the same category, and then the category to which the majority of the one or more relative entropy value sample sets belong is determined to determine whether the neural network model to be monitored has a running fault.

[0122] As an example, taking the first classifier as a KNN classifier, the relative entropy value set is input into the KNN classifier, the KNN classifier can automatically calculate the distance between the relative entropy value set and each relative threshold value sample set in the plurality of relative entropy value sample sets, and select the K relative entropy value sample sets closest to the relative entropy value set. The category to which the majority of the K relative entropy value sample sets belong is taken as the category of the relative entropy value set in a majority voting manner; if the category of the relative entropy value set is a fault state of the neural network model to be monitored, it can be determined that the neural network model to be monitored has a running fault; if the category of the relative entropy value set is a normal working state of the neural network model to be monitored, it can be determined that the neural network model to be monitored has no running fault. In this way, by using the first classifier based on machine learning, without pre-training, the relative entropy value set can be automatically classified more conveniently and quickly according to the distance between the relative entropy value set and the plurality of relative entropy value sample sets, so as to determine in real time whether the neural network model to be monitored has a running fault.

[0123] In the second mode, the relative entropy value set is input into the second classifier based on deep learning, which is a preset classification model, to determine whether the to-be-monitored neural network model has a running fault. The second classifier is trained by multiple relative entropy value sample sets.

[0124] Exemplarily, the second classifier can be trained in advance according to the multiple relative entropy value sample sets and the known categories to which the relative entropy value sample sets belong. After training, the second classifier can accurately distinguish the relative entropy value sets of different categories. Then, when performing fault monitoring, the relative entropy value set can be input into the trained second classifier, and the second classifier can automatically determine the category to which the relative entropy value set belongs, thereby accurately determining whether the to-be-monitored neural network model has a running fault. In this way, by using the second classifier based on deep learning, the accuracy of the classification of the relative entropy value set is effectively improved while the category to which the relative entropy value set belongs is determined in real time, thereby more accurately determining whether the to-be-monitored neural network model has a running fault.

[0125] As an example, taking the second classifier as an example, the MLP, the topology structure of the MLP can be set according to the number of relative entropy values in the set of relative entropy values and the number of classification categories; for example, the topology structure of the MLP can be (n-20-2), where n represents the number of relative entropy values in the set of relative entropy values input to the input layer of the MLP; 20 represents the number of hidden layers of the MLP, and 2 represents two categories output by the output layer of the MLP, i.e., that the to-be-monitored neural network model has a fault and that the to-be-monitored neural network model is working normally. In the training stage, a plurality of sets of relative entropy value samples are used as training samples to train the MLP, wherein the set of relative entropy values corresponding to the m neural network layers when the to-be-monitored neural network model has a fault can be used as negative samples, and the set of relative entropy values corresponding to the m neural network layers when the to-be-monitored neural network model is working normally can be used as positive samples; the training samples and the corresponding category labels are input into the MLP, the weight parameters in the MLP are trained, for example, a training sample can be input into the MLP, the MLP outputs the category of the training sample, according to the category and the category label of the training sample, a loss function value is determined, back propagation is performed according to the loss function value, and the weight parameters in the MLP are adjusted; a plurality of training samples are used to repeat the above training process until convergence is reached, the weight parameters in the MLP at the time of convergence are fixed, and a trained MLP is obtained. In the fault monitoring stage, the set of relative entropy values is input into the trained MLP, and the MLP can automatically output the category to which the set of relative entropy values belongs, so as to accurately determine whether the current to-be-monitored neural network model has a running fault in real time. As an example, for Alexnet used for image recognition, when the trained MLP is used to determine whether Alexnet has a running fault, the accuracy is increased by about 15% compared with the SED method.

[0126] It should be noted that the KNN and the MLP described above are only examples, and other classifiers can be used as classification models as needed, and the present application is not limited in this regard.

[0127] For example, when the to-be-monitored neural network model has a fault, the set of relative entropy values corresponding to the m neural network layers can include the relative entropy values between the first feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution; wherein the first feature value sample set is extracted from the output data sample set corresponding to each neural network layer when the to-be-monitored neural network model has a fault; and when the to-be-monitored neural network model is working normally, the set of relative entropy values corresponding to the m neural network layers can include the relative entropy values between the second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution; wherein the second feature value sample set is extracted from the output data sample set corresponding to each neural network layer when the to-be-monitored neural network model is working normally.

[0128] Exemplarily, the second element set can be the same as the first element set described above; it can be understood that an element set conforming to a preset probability distribution, i.e., a second element set, can be determined in advance, and the element set conforming to the preset probability distribution is used as the first element set in the fault monitoring phase.

[0129] It can be understood that according to different scenarios, the corresponding relative entropy value sample set can be generated in advance for different neural network models to be monitored.

[0130] Figure 4 A method flowchart for obtaining a relative entropy value sample set according to an embodiment of the present application is shown, as shown in Figure 4 may include the following steps:

[0131] Step 401, respectively obtaining an output data sample set corresponding to at least one neural network layer in the neural network model to be monitored when the neural network model to be monitored fails and when the neural network model to be monitored works normally.

[0132] As an example, the output data sample set corresponding to each neural network layer in the m neural network layers in the neural network model to be monitored when the neural network model to be monitored works normally can be obtained.

[0133] Taking a deep neural network model for image recognition in a perception module as an example, for an original image collected by a vehicle-mounted camera, the object in the original image is pre-labeled as a pedestrian, the original image is input into the neural network model to be monitored, and the neural network model to be monitored discriminates that the object contained in the original image is a pedestrian through inference, then the output data of each neural network layer in the inference process is collected as an output data sample set corresponding to each neural network layer in the m neural network layers when the neural network model to be monitored works normally. Similarly, different original images can be used in turn, and the output data of each neural network layer in each inference process is collected accordingly, so as to obtain multiple output data sample sets corresponding to each neural network layer in the m neural network layers when the neural network model to be monitored works normally.

[0134] As another example, the failure of the inference process of the neural network model to be monitored can be simulated by means of fault injection, so as to obtain the output data sample set corresponding to each neural network layer in the m neural network layers in the neural network model to be monitored when the neural network model to be monitored fails.

[0135] Taking a deep neural network model used for image recognition in the perception module as an example, in an original image captured by a vehicle-mounted camera, the object in the original image is pre-labeled as a pedestrian. This original image is input into the neural network model to be monitored. A fault can be injected. The neural network model, through reasoning, determines that the object in the original image is not a pedestrian. The output data of each neural network layer during this reasoning process is collected, thus serving as a set of output data samples corresponding to each of the m neural network layers when the neural network model to be monitored malfunctions. Similarly, different faults or different original images can be injected sequentially, and the neural network model to be monitored can perform multiple reasoning calculations, collecting the output data of each neural network layer during each reasoning process, thereby obtaining multiple sets of output data samples corresponding to each of the m neural network layers when the neural network model to be monitored malfunctions.

[0136] As another example, adversarial examples can be generated to obtain the set of output data samples corresponding to each of the m neural network layers in the neural network model under monitoring when the model malfunctions. Here, adversarial examples represent the input data that the neural network model under monitoring cannot reason about normally.

[0137] Taking a deep neural network model used for image recognition in the perception module as an example, in a frame of original image captured by an in-vehicle camera, the objects in the original image are pre-labeled as pedestrians. By adding a very small amount of carefully constructed noise to the original image, an adversarial image is obtained. The human eye usually cannot distinguish this adversarial image from the original image. The neural network model under test may misclassify the objects in the adversarial image; for example, it may determine that the objects in the adversarial image are not pedestrians, thus causing an error. The output data of each neural network layer during this inference process is collected as a set of output data samples corresponding to each of the m neural network layers when the neural network model under test malfunctions. Similarly, different adversarial images can be generated, and the output data of each neural network layer during each inference process can be collected accordingly, thus obtaining multiple sets of output data samples corresponding to each of the m neural network layers when the neural network model under test malfunctions.

[0138] Step 402: Extract the feature value sample set corresponding to at least one neural network layer from the output data sample set corresponding to at least one neural network layer.

[0139] In this step, the manner of extracting the feature value sample set can refer to the related description in step 303, which will not be repeated here. For example, the number of feature value samples in the feature value sample set can be determined by formulas (1) and (2). For example, the value of the sampling coefficient can be set according to the requirements. For example, a small sampling coefficient can be set to reduce the number of feature value samples in the feature value sample set, thereby effectively improving the training efficiency of the second classifier, training the second classifier under a small amount of data, and effectively saving computing resources; or the efficiency of automatically classifying the relative entropy value set by the first classifier can be effectively improved, and the real-time requirement of fault monitoring can be better met.

[0140] For example, when the obtained neural network model to be monitored fails, the first feature value sample set corresponding to each neural network layer in the m neural network layers can be extracted from the output data sample set corresponding to each neural network layer; and when the obtained neural network model to be monitored works normally, the second feature value sample set corresponding to each neural network layer in the m neural network layers can be extracted from the output data sample set corresponding to each neural network layer.

[0141] For example, for any neural network layer in the m neural network layers, the output data sample in the output data sample set corresponding to the neural network layer can be extracted as a feature value sample in a manner of a preset probability distribution, thereby obtaining the feature value sample set corresponding to the neural network layer, and improving the robustness of the classification model.

[0142] In step 403, the relative entropy value between the feature value sample set corresponding to at least one neural network layer and the second element set conforming to the preset probability distribution is calculated to obtain a relative entropy value sample set.

[0143] For example, the relative entropy value between the first feature value sample set corresponding to each neural network layer and the second element set can be calculated to obtain the relative entropy value sample set corresponding to the m neural network layers when the neural network model to be monitored fails; and the relative entropy value between the second feature value sample set corresponding to each neural network layer and the second element set can be calculated to obtain the relative entropy value sample set corresponding to the m neural network layers when the neural network model to be monitored works normally.

[0144] For example, the relative entropy value sample set can also be labeled with a category, wherein the relative entropy value sample set corresponding to the m neural network layers when the neural network model to be monitored fails can be labeled as the neural network model to be monitored failing, and the relative entropy value sample set corresponding to the m neural network layers when the neural network model to be monitored works normally can be labeled as the neural network model to be monitored working normally.

[0145] As an example, the obtained relative entropy value sample set can be used to sample a first classifier based on machine learning to determine whether the to-be-monitored neural network model has a running fault; as another example, the obtained relative entropy value sample set can be used to train a second classifier based on deep learning, so that the second classifier is trained by using a small amount of relative entropy value samples, thereby effectively saving computing resources.

[0146] In addition, the method provided by the embodiments of the present application has strong expandability, and can further analyze the internal results of the neural network model, perform model-agnostic analysis, etc. on the basis of the above embodiments in combination with existing technologies; or can implement more levels of classification of running faults with the support of more relative entropy value sample sets.

[0147] The following takes the to-be-monitored neural network model as a deep neural network model for image recognition in a perception module of an automatic driving system as an example to exemplarily describe the neural network model fault monitoring method shown in the above Figure 3

[0148] Figure 5 FIG. 1 shows a schematic diagram of a neural network model fault monitoring method in an automatic driving system according to an embodiment of the present application. As shown in FIG. 1, a deep neural network model for image recognition in a perception module can be deployed in a vehicle-mounted computing platform or an AI accelerator. During the working process of the automatic driving system, after the perception module obtains each frame of image collected by a vehicle-mounted camera, the deep neural network model for image recognition is used for inference to output a recognition result. For any frame of image, the neural network model fault monitoring device can execute the above step 301 to obtain the output data set corresponding to each neural network layer in the m neural network layers in the deep neural network model for image recognition during the processing of the frame of image by the deep neural network model. Figure 5

[0149] Further, the neural network model fault monitoring device can execute the above step 302 to extract the feature value set corresponding to each neural network layer from the output data set corresponding to each neural network layer in the m neural network layers.

[0150] Exemplarily, for any neural network layer, the feature value set corresponding to the neural network layer can be represented in the form of a feature value vector. As an example, after extracting n feature values from the mth neural network layer, a feature value vector Am can be obtained:

[0151]

[0152] In formula (4), a and b respectively represent the extracted feature values, n represents the number of feature values, and m represents the number of neural network layers. In formula (4), a and b respectively represent the extracted feature values, n represents the number of feature values, and m represents the number of neural network layers.​​

[0153] Exemplarily, the same number of output data can be extracted as the feature value set corresponding to each neural network layer; and the feature value set corresponding to each neural network layer obtained is shown in the following formula (5):

[0154]

[0155] In formula (5), A1, A2…Am represent the feature value set corresponding to the m neural network layers. A is a feature value matrix of m rows and n columns, which includes the feature value set corresponding to each neural network layer.

[0156] The feature value matrix is constructed based on the Monte Carlo idea to reflect the running state of the deep neural network model for image recognition. During the inference operation of the deep neural network model for image recognition, a large amount of intermediate calculation data, i.e. output data corresponding to each neural network layer, is generated. The output data is used to generate the feature value matrix through experiments, thereby establishing an estimate of the output data corresponding to each neural network layer.

[0157] Further, the neural network model fault monitoring device can perform the above step 303 to calculate the relative entropy value between the feature value set corresponding to each neural network layer and the first element set conforming to the Gaussian distribution, to obtain a relative entropy value set corresponding to the m neural network layers.

[0158] Exemplarily, the first element set can be represented in the form of a reference matrix; and the relative entropy value set can be represented in the form of a relative entropy value matrix.

[0159] As an example, the reference matrix G can be shown in the following formula (6):

[0160] G = [g1 g2 g3 … gn] (6) n ]................(6)

[0161] In the above formula (6), g1, g2, …, gn represent a random number subject to a standard normal distribution (N~(0, 1)) respectively; that is, the reference matrix G includes a first element set conforming to the Gaussian distribution.

[0162] As an example, the relative entropy matrix can be determined according to the feature value matrix and the reference matrix; exemplarily, the relative entropy value KLm of the feature value vector Am and the reference matrix G can be obtained in combination with formula (4) and formula (6), as shown in the following formula (7):

[0163]

[0164] In formula (7), Am (i) represents the i-th element of the feature vector Am, G(i) represents the i-th element of the reference matrix; In(·) represents the calculation of the natural logarithm; ∑ n (·) represents the sum of n data.

[0165] Referring to formula (7), for any feature vector in formula (5), the relative entropy value of the reference matrix described in formula (6) is calculated, and a relative entropy value matrix KL is obtained:

[0166] KL = [KL1 KL2 … KL m ] T ...............(8)

[0167] Each element in the relative entropy value matrix KL represents a relative entropy value. That is, the relative entropy value matrix KL includes the relative entropy values of the feature value set corresponding to each neural network layer and the first element set.

[0168] The relative entropy value matrix KL shown in formula (8) is a 1xm matrix, thereby reducing the dimension of the mxn feature value matrix A shown in formula (5) to a 1xm matrix, realizing data dimension reduction, and further improving the operation efficiency.

[0169] In addition, the relative entropy value matrix KL describes the distribution difference between the m layers of extracted feature values in the neural network and the reference matrix G. In the embodiments of the present application, the inference data inside the neural network is not directly classified, but the feature value matrix A and the reference matrix G of the Gaussian distribution are projected, and each feature vector in formula (8) represents a feature point in the projection space. The categories corresponding to these feature points are two categories of the failure of the neural network model to be monitored or the normal working of the neural network model to be monitored, so that the difference between the normal output data of each neural network layer in the inference process when the neural network model is working normally and the abnormal output data of each neural network layer in the inference process when the neural network model fails can be pulled apart, and the coupling degree of the normal output data and the abnormal output data is reduced.

[0170] Furthermore, the neural network model fault monitoring device can execute step 304 above, using a classification model to quickly classify the relative entropy matrix KL, thereby determining in real time and accurately whether the monitored neural network model has an operational fault. The neural network model fault monitoring device can also feed back the monitoring results to the sensing module, the sensing fusion module, or the system health management module for early warning reporting; for example, when the classification model determines that the category corresponding to the relative entropy matrix KL is that the monitored neural network is working normally, it can feed back the result to the sensing module. After receiving the feedback, the sensing module transmits the current sensing result to the planning and decision-making module; when the classification model determines that the category corresponding to the relative entropy matrix KL is that the monitored neural network has a fault, it can feed back the result to the sensing module. After receiving the feedback, the sensing module discards the current sensing result.

[0171] Based on the same inventive concept as the above method embodiments, embodiments of this application also provide a neural network model fault monitoring device for an autonomous driving system. This neural network model fault monitoring device for an autonomous driving system can be used to execute the technical solutions described in the above method embodiments. For example, it can execute the above... Figure 3 , Figure 4 or Figure 5 The steps of the neural network model fault monitoring method in the autonomous driving system are shown below.

[0172] Figure 6 This diagram illustrates the structure of a neural network model fault monitoring device in an autonomous driving system according to an embodiment of this application. Figure 6 As shown, the device may include: a transmission module 601, used to acquire a target output data set of a neural network model to be monitored in an autonomous driving system, the target output data set including the output data set corresponding to each of m neural network layers, wherein the neural network model to be monitored includes M neural network layers, M is an integer greater than 1, and m is an integer greater than 1 and not greater than M; a processing module 602, used to extract the feature value set corresponding to each neural network layer from the target output data set; calculate the relative entropy value between the feature value set and a first element set conforming to a preset probability distribution to obtain the relative entropy value set corresponding to the m neural network layers; and determine whether the neural network model to be monitored has an operational fault based on the relative entropy value set.

[0173] In the embodiments of the present application, based on the idea of the Monte Carlo method, the output data of each neural network layer is selectively sampled, part of the output data in the output data set is extracted as a characteristic value, and the distribution of the output data of each neural network layer is reflected by as few characteristic values as possible, thereby simplifying the calculation, saving the operation cost, and improving the operation efficiency; at the same time, by calculating the relative entropy value between the characteristic value set corresponding to each neural network layer and the first element set conforming to the preset probability distribution, a relative entropy value set is obtained, the data dimensionality is reduced, and the operation efficiency is further improved; thereby the real-time performance of fault monitoring is improved, and the real-time monitoring of the neural network model fault in the automatic driving system is realized. At the same time, the relative entropy value is used to describe the distribution difference characteristics of the normal output data and the abnormal output data of each neural network layer, and the normal output data and the abnormal output data of each neural network layer are distinguished, so that whether the to-be-monitored neural network model has a running fault is more accurately judged through the relative entropy value set corresponding to the m neural network layers, and the accuracy of fault monitoring is improved. In addition, various running faults of the neural network model or running faults of various neural network models can be effectively monitored, and the application scope is wide.

[0174] In a possible implementation, the processing module 602 is further configured to: determine a first output data set with the smallest number of output data in the target output data set; and extract the characteristic value set corresponding to each neural network layer from the output data set corresponding to each neural network layer according to the number of output data in the first output data set; wherein the number of characteristic values in each characteristic value set corresponding to each neural network layer is less than or equal to the number of output data in the first output data set.

[0175] In a possible implementation, the processing module 602 is further configured to: extract the characteristic value set corresponding to each neural network layer by taking the number of output data in the output data set corresponding to each neural network layer as a weight.

[0176] In a possible implementation, the processing module 602 is further configured to: input the relative entropy value set into a preset classification model to determine whether the to-be-monitored neural network model has a running fault.

[0177] In a possible implementation, the preset classification model includes a first classifier based on machine learning; the processing module 602 is further configured to: input the set of relative entropy values into the first classifier, and calculate distances between the set of relative entropy values and a plurality of sets of relative entropy value samples; the plurality of sets of relative entropy value samples include a set of relative entropy value samples corresponding to the m neural network layers when the to-be-monitored neural network model is faulty and a set of relative entropy value samples corresponding to the m neural network layers when the to-be-monitored neural network model is normal; and determine whether the to-be-monitored neural network model is faulty according to the distances between the set of relative entropy values and the plurality of sets of relative entropy value samples.

[0178] In a possible implementation, the classification model includes a second classifier based on deep learning; the processing module 602 is further configured to: input the set of relative entropy values into the second classifier, and determine whether the to-be-monitored neural network model is faulty; the second classifier is trained by a plurality of sets of relative entropy value samples.

[0179] In a possible implementation, the set of relative entropy value samples corresponding to the m neural network layers when the to-be-monitored neural network model is faulty includes relative entropy values between a first feature value sample set corresponding to each neural network layer in the m neural network layers and a second element set conforming to the preset probability distribution; the first feature value sample set is extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is faulty; and the set of relative entropy value samples corresponding to the m neural network layers when the to-be-monitored neural network model is normal includes relative entropy values between a second feature value sample set corresponding to each neural network layer in the m neural network layers and the second element set conforming to the preset probability distribution; the second feature value sample set is extracted from an output data sample set corresponding to the each neural network layer when the to-be-monitored neural network model is normal.

[0180] The above Figure 6 The technical effects and specific descriptions of the neural network model fault monitoring apparatus in the automatic driving system and various possible implementations thereof can be referred to the neural network model fault monitoring method in the automatic driving system, which will not be described herein again.

[0181] It should be understood that the division of each module in the above apparatus is only a logical functional division, and all or part of the modules can be integrated into one physical entity or physically separated when actually implemented. In addition, the modules in the apparatus can be implemented in the form of processor calling software; for example, the apparatus includes a processor, the processor is connected with a memory, the memory stores instructions, and the processor calls the instructions stored in the memory to implement any one of the above methods or to realize the functions of the modules of the apparatus, wherein the processor is, for example, a general processor such as a central processing unit (CPU) or a microprocessor, and the memory is an internal memory of the apparatus or an external memory of the apparatus. Alternatively, the modules in the apparatus can be implemented in the form of hardware circuit, and the functions of part or all of the modules can be realized by the design of the hardware circuit, which can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of part or all of the modules are realized by the design of the logical relationship of elements in the circuit; for example, in another implementation, the hardware circuit is realized by a programmable logic device (PLD), and a field programmable gate array (FPGA) is taken as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of part or all of the modules. All the modules of the above apparatus can be realized by processor calling software, or all the modules can be realized by hardware circuit, or part of the modules can be realized by processor calling software and the remaining part can be realized by hardware circuit.

[0182] In the embodiments of the present application, the processor is a circuit with signal processing capability, which can be a circuit with instruction reading and running capability in one implementation, such as CPU, microprocessor, graphics processing unit (GPU) (which can be understood as a microprocessor), digital signal processor (DSP) and the like; in another implementation, the processor can realize certain functions through the logical relationship of hardware circuit, and the logical relationship of the hardware circuit is fixed or can be reconfigured, such as ASIC or PLD implemented hardware circuit, such as FPGA. In the reconfigurable hardware circuit, the processor loads the configuration document to realize the hardware circuit configuration process, which can be understood as the process of the processor loading instructions to realize the functions of part or all of the modules.

[0183] It can be seen that each module in the above apparatus can be one or more processors (or processing circuits) configured to implement the above embodiment methods, for example: CPU, GPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0184] In addition, each module in the above apparatus can be integrated together or can be independently implemented. In one implementation, these modules are integrated together to be implemented in the form of SOC. The SOC can include at least one processor for implementing any of the above methods or the functions of the modules of the apparatus, and the at least one processor can be different, for example, including CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0185] Embodiments of the present application also provide an apparatus for monitoring faults of a neural network model in an autonomous driving system, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-mentioned method when executing the instructions. Illustratively, each step of the above-mentioned method for monitoring faults of a neural network model in an autonomous driving system can be performed. Figure 3 、 Figure 4 or Figure 5

[0186] Figure 7 Fig. 1 shows a structural schematic diagram of an apparatus for monitoring faults of a neural network model in an autonomous driving system according to an embodiment of the present application. As shown in Fig. 1, the apparatus for monitoring faults of a neural network model in an autonomous driving system can comprise: at least one processor 701, a communication line 702, a memory 703, and at least one communication interface 704. Figure 7

[0187] The processor 701 can be a general central processing unit, a microprocessor, an application specific integrated circuit, or one or more integrated circuits for controlling the execution of programs of the present application; the processor 701 can also include a heterogeneous computing architecture of multiple general-purpose processors, for example, it can be a combination of at least two of CPU, GPU, microprocessor, DSP, ASIC, FPGA; as an example, the processor 701 can be CPU+GPU or CPU+ASIC or CPU+FPGA.

[0188] The communication line 702 can include a channel for transmitting information between the above components.

[0189] The communication interface 704 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, RAN, wireless local area networks (WLAN), etc. ​​

[0190] The memory 703 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through the communication line 702. The memory can also be integrated with the processor. The memory provided in the embodiments of the present application can generally be non-volatile. The memory 703 is used to store computer execution instructions for executing the solutions of the present application, and the processor 701 controls the execution. The processor 701 is used to execute the computer execution instructions stored in the memory 703, so as to realize the method provided in the above embodiments of the present application; for example, the steps of the neural network model fault monitoring method in the automatic driving system shown in the above Figure 3 、 Figure 4 or Figure 5 may be realized.

[0191] Optionally, the computer execution instructions in the embodiments of the present application can also be referred to as application program codes, which are not specifically limited in the embodiments of the present application.

[0192] For example, the processor 701 can include one or more CPUs, for example, the CPU0 in Figure 7 . The processor 701 can also include one CPU and any one of GPU, ASIC, FPGA, for example, the CPU0+GPU0 or CPU0+ASIC0 or CPU0+FPGA0 in Figure 7 .

[0193] For example, the neural network model fault monitoring apparatus in the automatic driving system can include multiple processors, for example Figure 7Processors 701 and 707 are mentioned. Each of these processors can be a single-core processor, a multi-core processor, or a heterogeneous computing architecture that includes multiple general-purpose processors. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0194] In a specific implementation, as one embodiment, the neural network model fault monitoring device in the autonomous driving system may further include an output device 705 and an input device 706. The output device 705 communicates with the processor 701 and can display information in various ways. For example, the output device 705 can be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, such as an in-vehicle HUD, AR-HUD, or monitor. The input device 706 communicates with the processor 701 and can receive user input in various ways. For example, the input device 706 can be a mouse, keyboard, touchscreen device, or sensor device.

[0195] As an example, combined Figure 7 The above-mentioned neural network model fault monitoring device in the autonomous driving system is shown. Figure 6 The transmission module 601 in the middle can be made by Figure 7 This is implemented using the communication interface 704; the above Figure 6 The processing module 602 in the middle can be made by Figure 7 It is implemented using the 701 processor.

[0196] Embodiments of this application provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the methods described in the above embodiments. Exemplarily, the above can be implemented... Figure 3 , Figure 4 or Figure 5 The steps of the neural network model fault monitoring method in the autonomous driving system are shown below.

[0197] Embodiments of this application provide a computer program product, which may include, for example, computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code; when the computer program product is run on a computer, the computer performs the methods described in the above embodiments. Exemplarily, the above... Figure 3 , Figure 4 or Figure 5Steps of the neural network model fault monitoring method in the autonomous driving system shown in the middle.

[0198] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0199] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0200] Computer readable program instructions for carrying out operations of the present application can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0201] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0202] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the instructions which operate on the computer or other programmable data processing apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0203] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0204] The flow diagrams and the block diagrams in the drawings are presented to illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved.

[0205] It also is noted that each of the blocks of the flowchart, and / or combinations of blocks, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0206] Although the present application has been described in connection with various embodiments thereof, those skilled in the art will understand that many modifications can be made thereto without departing from the application as set forth in the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. The reference signs in the claims should not be construed as limiting the scope of the application.

[0207] The above description is only specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for fault monitoring of a neural network model in an autonomous driving system, characterized in that, The method includes: Obtain the target output data set of the neural network model to be monitored in the autonomous driving system. The target output data set includes the output data set corresponding to each neural network layer in m neural network layers. The neural network model to be monitored includes M neural network layers, where M is an integer greater than 1 and m is an integer greater than 1 and not greater than M. Extract the feature value set corresponding to each neural network layer from the target output data set; Calculate the relative entropy between the feature value set and the first element set that conforms to a preset probability distribution to obtain the relative entropy value set corresponding to the m neural network layers; Based on the set of relative entropy values, determine whether the neural network model to be monitored has any operational faults.

2. The method according to claim 1, characterized in that, The step of extracting the feature value set corresponding to each neural network layer from the target output data set includes: Determine the first output data set with the smallest number of output data in the target output data set; Based on the number of output data in the first output data set, a feature value set corresponding to each neural network layer is extracted from the output data set corresponding to each neural network layer; wherein the number of feature values ​​in the extracted feature value set corresponding to each neural network layer is less than or equal to the number of output data in the first output data set.

3. The method according to claim 1, characterized in that, The step of extracting the feature value set corresponding to each neural network layer from the target output data set includes: Using the number of output data in the output data set corresponding to each neural network layer as the weight, the feature value set corresponding to each neural network layer is extracted.

4. The method according to any one of claims 1-3, characterized in that, The step of determining whether the monitored neural network model has an operational fault based on the set of relative entropy values ​​includes: The set of relative entropy values ​​is input into a preset classification model to determine whether the neural network model to be monitored has any operational faults.

5. The method according to claim 4, characterized in that, The preset classification model includes a first classifier based on machine learning; The step of inputting the set of relative entropy values ​​into a preset classification model to determine whether the neural network model to be monitored has a malfunction includes: The relative entropy value set is input into the first classifier, and the distance between the relative entropy value set and multiple relative entropy value sample sets is calculated; wherein, the multiple relative entropy value sample sets include the relative entropy value sample sets corresponding to the m neural network layers when the neural network model under monitoring malfunctions and the relative entropy value sample sets corresponding to the m neural network layers when the neural network model under monitoring is working normally; Based on the distance between the set of relative entropy values ​​and multiple sets of relative entropy value samples, it is determined whether the neural network model to be monitored has an operational fault.

6. The method according to claim 4, characterized in that, The classification model includes a second classifier based on deep learning; The step of inputting the set of relative entropy values ​​into a preset classification model to determine whether the neural network model to be monitored has a malfunction includes: The set of relative entropy values ​​is input into the second classifier to determine whether the neural network model under monitoring has a malfunction; wherein, the second classifier is trained from multiple sets of relative entropy value samples.

7. The method according to claim 5, characterized in that, The relative entropy sample set corresponding to the m neural network layers when the monitored neural network model malfunctions includes: the relative entropy value between the first feature value sample set corresponding to each of the m neural network layers and the second element set conforming to the preset probability distribution; wherein, the first feature value sample set is extracted from the output data sample set corresponding to each neural network layer when the monitored neural network model malfunctions; the relative entropy sample set corresponding to the m neural network layers when the monitored neural network model is working normally includes: the relative entropy value between the second feature value sample set corresponding to each of the m neural network layers and the second element set conforming to the preset probability distribution; wherein, the second feature value sample set is extracted from the output data sample set corresponding to each neural network layer when the monitored neural network model is working normally.

8. A fault monitoring device for a neural network model in an autonomous driving system, characterized in that, The device includes: The transmission module is used to acquire the target output data set of the neural network model to be monitored in the autonomous driving system. The target output data set includes the output data set corresponding to each neural network layer in m neural network layers. The neural network model to be monitored includes M neural network layers, where M is an integer greater than 1 and m is an integer greater than 1 and not greater than M. The processing module is used to extract the feature value set corresponding to each neural network layer from the target output data set; calculate the relative entropy value between the feature value set and the first element set that conforms to a preset probability distribution to obtain the relative entropy value set corresponding to the m neural network layers; and determine whether the neural network model to be monitored has an operational fault based on the relative entropy value set.

9. The apparatus according to claim 8, characterized in that, The processing module is further configured to: determine a first output data set with the smallest number of output data in the target output data set; and extract a feature value set corresponding to each neural network layer from the output data set corresponding to each neural network layer based on the number of output data in the first output data set; wherein the number of feature values ​​in the extracted feature value set corresponding to each neural network layer is less than or equal to the number of output data in the first output data set.

10. The apparatus according to claim 8, characterized in that, The processing module is further configured to: extract the feature value set corresponding to each neural network layer, using the number of output data in the output data set corresponding to each neural network layer as the weight.

11. The apparatus according to any one of claims 8-10, characterized in that, The processing module is further configured to: input the set of relative entropy values ​​into a preset classification model to determine whether the neural network model to be monitored has a running fault.

12. The apparatus according to claim 11, characterized in that, The preset classification model includes a first classifier based on machine learning; The processing module is further configured to: input the relative entropy value set into the first classifier, and calculate the distance between the relative entropy value set and multiple relative entropy value sample sets; wherein, the multiple relative entropy value sample sets include the relative entropy value sample sets corresponding to the m neural network layers when the neural network model under monitoring malfunctions and the relative entropy value sample sets corresponding to the m neural network layers when the neural network model under monitoring is working normally; and determine whether the neural network model under monitoring has an operational malfunction based on the distance between the relative entropy value set and the multiple relative entropy value sample sets.

13. The apparatus according to claim 11, characterized in that, The classification model includes a second classifier based on deep learning; The processing module is further configured to: input the set of relative entropy values ​​into the second classifier to determine whether the neural network model to be monitored has a running fault; wherein the second classifier is trained from multiple sets of relative entropy value samples.

14. The apparatus according to claim 12, characterized in that, The relative entropy sample set corresponding to the m neural network layers when the monitored neural network model malfunctions includes: the relative entropy value between the first feature value sample set corresponding to each of the m neural network layers and the second element set conforming to the preset probability distribution; wherein, the first feature value sample set is extracted from the output data sample set corresponding to each neural network layer when the monitored neural network model malfunctions; the relative entropy sample set corresponding to the m neural network layers when the monitored neural network model is working normally includes: the relative entropy value between the second feature value sample set corresponding to each of the m neural network layers and the second element set conforming to the preset probability distribution; wherein, the second feature value sample set is extracted from the output data sample set corresponding to each neural network layer when the monitored neural network model is working normally.

15. A fault monitoring device for a neural network model in an autonomous driving system, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-7 when executing the instructions.

16. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

17. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method described in any one of claims 1-7.

Images