Method, device and computer equipment for monitoring health status of vehicle sensors

Abstract

The present application relates to a method, device and computer equipment for monitoring the health status of vehicle sensors. The method includes: obtaining operation data of at least one sensor of a vehicle within a preset time period; obtaining the health status information of each sensor according to each of the operation data and a preset pulse residual network model, wherein the pulse residual network model includes a pulse feature encoder network for extracting first pulse features of each of the operation data, and the first pulse features are fault feature information corresponding to each of the operation data and related to time. By using this method, the accuracy of monitoring the health status of vehicle sensors can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving, and particularly to a method, device, and computer device for monitoring the health status of vehicle sensors. Background Art

[0002] Generally, the decision-making ability of autonomous vehicles depends on the environmental perception data collected by the sensors on the autonomous vehicles. However, since autonomous vehicles operate in a complex and open environment for a long time, their sensors will inevitably be contaminated, interfered with, or their performance will degrade, thus causing sensor anomalies. Therefore, in order to ensure the safe driving of autonomous vehicles, it is necessary to diagnose and monitor the health of the sensors.

[0003] In traditional technologies, neural network models are used to diagnose the faults of the sensors of autonomous vehicles. However, traditional fault diagnosis methods have the problem of low accuracy. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method, device, and computer device for monitoring the health status of vehicle sensors that can improve the accuracy of monitoring the health status of vehicle sensors.

[0005] In a first aspect, the present application provides a method for monitoring the health status of vehicle sensors. The method includes:

[0006] Obtain the operation data of at least one sensor of a vehicle within a preset time period;

[0007] According to each of the operation data and a preset pulse residual network model, obtain the health status information of each of the sensors; wherein, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is used to extract the first pulse features of each of the operation data; the first pulse features are fault feature information corresponding to each of the operation data and related to time.

[0008] In one embodiment, the pulse residual network model further includes a pulse classifier network; the step of obtaining the health status information of each of the sensors according to each of the operation data and the preset pulse residual network model includes:

[0009] According to each of the operation data and the pulse feature encoder network, obtain the first pulse features corresponding to each of the operation data;

[0010] Input the first pulse features corresponding to each of the operation data into the pulse classifier network to obtain the health status information of each of the sensors.

[0011] In one embodiment, the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module; obtaining the first pulse features corresponding to the respective operation data according to the respective operation data and the pulse feature encoder network includes:

[0012] Inputting the respective operation data into the static convolution module for pulse encoding to obtain pulse sequences corresponding to the respective operation data; the pulse sequences are sequences related to time;

[0013] Inputting the respective pulse sequences into the pulse convolution module to obtain second pulse features corresponding to the respective operation data;

[0014] Inputting the second pulse features corresponding to the respective operation data into the pulse residual module to obtain the respective first pulse features; wherein, the richness of the information included in the respective first pulse features is greater than the richness of the information included in the respective second pulse features.

[0015] In one embodiment, the pulse classifier network includes a pulse fully connected layer; inputting the first pulse features corresponding to the respective operation data into the pulse classifier network to obtain the health state information of the respective sensors includes:

[0016] Mapping the first pulse features corresponding to the respective operation data to the fault category space through the pulse fully connected layer to obtain the health state information of the respective sensors.

[0017] In one embodiment, the health state information includes fault information and health index information; mapping the first pulse features corresponding to the respective operation data to the fault category space through the pulse fully connected layer to obtain the health state information of the respective sensors includes:

[0018] Mapping the first pulse features corresponding to the respective operation data to the fault category space through the pulse fully connected layer to obtain the fault information of the respective sensors;

[0019] Obtaining the health index information corresponding to the respective sensors according to the fault information of the respective sensors.

[0020] In one embodiment, the method further includes:

[0021] Obtaining the sample operation data of at least one test sensor of the vehicle within a preset time period and the golden standard health state information corresponding to the respective sample operation data;

[0022] Inputting the respective sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain first sample pulse features corresponding to the respective sample data;

[0023] Input each of the first sample pulse features into the initial pulse classifier network of the initial pulse residual network model to obtain the sample health status information of each test sensor;

[0024] Train the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain the pulse residual network model.

[0025] In one embodiment, the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module, and an initial pulse residual module. The step of inputting each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse feature corresponding to each sample data includes:

[0026] Input each sample operation data into the initial static convolution module for pulse encoding to obtain a sample pulse sequence corresponding to each sample operation data; the sample pulse sequence is a time-related sequence;

[0027] Input each sample pulse sequence into the initial pulse convolution module to obtain a second sample pulse feature corresponding to each sample operation data;

[0028] Input each second sample pulse feature into the initial pulse residual module to obtain each first sample pulse feature; wherein, the richness of the information included in each first sample pulse feature is greater than the richness of the information included in each second sample pulse feature.

[0029] In one embodiment, the step of training the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain the pulse residual network model includes:

[0030] Obtain the value of the loss function of the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor;

[0031] Adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient substitution algorithm to obtain the pulse residual network model.

[0032] In one embodiment, the pulse feature encoder network is a membrane-learnable leaky integrate-and-fire model.

[0033] In a second aspect, the present application also provides a vehicle sensor health status monitoring device. The device includes:

[0034] A first acquisition module, configured to acquire operation data of at least one sensor of a vehicle within a preset time period;

[0035] A second acquisition module, configured to obtain health status information of each of the sensors according to the respective operation data and a preset pulse residual network model; wherein, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is configured to extract first pulse features of the respective operation data; the first pulse features are fault feature information corresponding to the respective operation data and related to time.

[0036] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect above is implemented.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0038] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0039] For the above vehicle sensor health status monitoring method, device and computer device, by acquiring operation data of at least one sensor of a vehicle within a preset time period, the health status information of each sensor can be obtained according to the respective operation data and a preset pulse residual network model including a pulse feature encoder network. Since the pulse feature encoder network can extract first pulse features including fault feature information related to time, the processing accuracy of the pulse residual network model for the operation data of each sensor is improved, and the accuracy of the pulse residual network model for processing sensor operation data is improved; in addition, since the pulse residual network model is adopted, the problems of gradient disappearance and gradient explosion existing in the backpropagation process of the model can be overcome, and the original continuous signal form of operation data can be converted into a pulse signal through the pulse feature encoder, so that the pulse signal can be processed for fault feature extraction. Compared with the traditional technology, the computing requirement of the model can be reduced, and the computing efficiency of the model can be further improved. Description of the Drawings

[0040] Figure 1 It is an application environment diagram of the vehicle sensor health status monitoring method in an embodiment;

[0041] Figure 2Schematic flowchart of a method for monitoring the health status of vehicle sensors in one embodiment;

[0042] Figure 3 Schematic flowchart of a method for monitoring the health status of vehicle sensors in another embodiment;

[0043] Figure 4 Schematic flowchart of a method for monitoring the health status of vehicle sensors in another embodiment;

[0044] Figure 5 Schematic diagram of the structure of a pulse residual network model in one embodiment;

[0045] Figure 6 Schematic flowchart of a method for monitoring the health status of vehicle sensors in another embodiment;

[0046] Figure 7 Schematic flowchart of a method for monitoring the health status of vehicle sensors in another embodiment;

[0047] Figure 8 Schematic flowchart of a method for monitoring the health status of vehicle sensors in another embodiment;

[0048] Figure 9 Schematic block diagram of the structure of a vehicle sensor health status monitoring device in one embodiment;

[0049] Figure 10 Schematic block diagram of the structure of a vehicle sensor health status monitoring device in another embodiment. Detailed implementation manners

[0050] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] Autonomous vehicles, with their high level of intelligence, have been successfully applied in daily life and promoted the development of the automotive industry. The autonomous decision-making ability of autonomous vehicles mainly depends on a large number of sensors, and environmental perception and driving decisions are realized through the data collected by the sensors. Therefore, the accuracy of the information obtained by the sensors determines the safety and reliability of autonomous vehicles. However, when autonomous vehicles operate in a complex and open environment for a long time, their sensors will inevitably be contaminated, interfered with, or experience performance degradation, resulting in sensor anomalies. Once incorrect information is obtained, it will cause the decision-making model to make incorrect decisions, thus seriously endangering the driving safety of autonomous vehicles. Therefore, in order to ensure the safe driving of autonomous vehicles, it is essential to accurately and effectively monitor the health status of sensors and monitor the health indicators of sensors.

[0052] In recent years, with the breakthroughs in artificial intelligence and big data technologies, data-driven intelligent fault diagnosis algorithms have also made breakthrough progress. Traditional data-driven diagnosis algorithms rely on experts' domain knowledge to extract features and then construct machine learning models for fault identification. However, this method is difficult to handle the massive data scale and thus cannot fully represent the complexity and dynamics of the data. For the fault diagnosis of sensors in autonomous vehicles, deep learning technologies such as artificial neural networks (ANNs) can be combined for sensor fault diagnosis. ANNs use high-precision floating-point numbers as information carriers and update weights through backpropagation algorithms to aggregate valuable information flows for intelligent decision-making. Convolutional neural networks (CNNs), based on convolutional topologies, optimize the aggregation method of information flows, achieve weight sharing, and bring about a breakthrough in the performance of ANN-based models. However, to achieve efficient fault diagnosis of sensors in autonomous vehicles, ANN-based fault diagnosis algorithms have problems such as poor generalization ability, strong data dependence, and high computational resource requirements and energy consumption because they use high-precision floating-point numbers as information carriers, resulting in low diagnostic efficiency in sensor fault diagnosis applications. In addition, autonomous vehicles have numerous sensors, which requires diagnostic models to have excellent spatio-temporal feature representation capabilities to accurately identify fault signals from a large amount of multi-source sensor data.

[0053] Therefore, the spiking neural network (SNN) model, known as the third-generation neural network model, has been proposed. The SNN model fully simulates the information transmission mechanism and dynamic characteristics between biological neurons. It uses spike signals as information carriers and aggregates valuable information flows through the dynamic mechanism between neuron synapses to represent the features of input data. With the development of SNN models and brain-inspired computing technologies, this new computing paradigm has made remarkable breakthroughs. The SNN model with excellent spatio-temporal information representation potential is expected to provide a new solution for the state monitoring of sensors in autonomous vehicles. In particular, its excellent low-power consumption characteristics are also very suitable for autonomous vehicles, drones, or other devices sensitive to energy consumption.

[0054] Based on the above motivations, this application proposes a method of applying the SNN model to monitor the status of autonomous vehicle sensors. This method is based on a convolutional topology architecture and fully draws on the design idea of the ANN-based residual network to implement a deep residual convolutional SNN architecture, constructing a deep brain-inspired spiking residual network (RSNN) model to achieve efficient fault diagnosis and health index prediction of autonomous vehicle sensors. This model can improve the training efficiency of the SNN model while expanding the encoding ability of the deep features of the SNN. Secondly, by introducing a learnable membrane time constant, a more realistic neuron interaction mechanism is achieved, improving the feature representation ability of the SNN. Moreover, this model optimizes the parameters during the training process of the RSNN model by introducing a gradient substitution algorithm, and can accurately determine the health status of autonomous vehicle sensors.

[0055] The sensor data processing method provided by the embodiments of this application can be applied to, for example, Figure 1 the application environment shown. Figure 1 A computer device is provided. This computer device can be a server, and its internal structure diagram can be as Figure 1 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sensor data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a sensor data processing method.

[0056] In one embodiment, as Figure 2 shown, a sensor data processing method is provided. Taking the computer device in Figure 1 as an example, it includes the following steps:

[0057] S201, Obtain the operation data of at least one sensor of the vehicle within a preset time period.

[0058] Among them, the preset time period is a certain time period when the sensors in the vehicle are in an operating state. Optionally, the preset time period can be 1 day or 3 days. Optionally, the sensors of the vehicle include, but are not limited to, radar sensors, cameras, acceleration sensors, angular velocity sensors, brake pressure sensors, throttle pressure sensors, steering angle sensors, etc. The operation data of the sensors are the data generated by each sensor during the operation of the vehicle. For example, the operation data of the acceleration sensor can be the acceleration data during the operation of the vehicle.

[0059] Optionally, the computer device can obtain the operation data within the preset time period from each sensor by sending a acquisition instruction to each sensor, or each sensor can periodically and autonomously send the operation data to the computer device.

[0060] Exemplarily, the acquired sensor operation data can be represented as H = {h 1 , h 2 , …, h i}, where h i represents the operation data of the i-th sensor. For example, h 1 represents the acquired operation data of the acceleration sensor, and h 2 represents the acquired operation data of the brake pressure sensor.

[0061] S202. Obtain the health state information of each sensor according to each operation data and a preset pulse residual network model; among them, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is used to extract the first pulse features of each operation data; the first pulse features are fault feature information corresponding to each operation data related to time.

[0062] Among them, the health status information of the sensor includes fault category information and the health index information of the sensor. Optionally, the fault category information may include offset faults, spatio-temporal faults, unstable faults, irregular faults, etc. The health index information is used to represent the performance degradation of the sensor, and the health index information can be represented in exponential form. The pulse feature encoder network refers to a network that converts the data input into the network into a pulse sequence. Optionally, in this embodiment, the pulse feature encoder network may include a convolutional encoder and a pulse encoder, which are used to extract the corresponding pulse sequence feature information from the input operation data to obtain the first pulse feature. The fault feature information in the first pulse feature includes the fault result of the sensor and the fault category information. Among them, a pulse refers to a brief fluctuating electrical shock like a pulse, which is a discrete signal, and the waveform of the pulse sequence is discontinuous on the time axis. That is to say, in this embodiment, each operation data can be input into a preset pulse residual network model, and the pulse feature encoder in the pulse residual network model converts each operation data into a time-related pulse sequence, and performs feature extraction on the converted pulse sequence to obtain the first pulse feature including the corresponding fault feature information of each operation data, so as to obtain the health status information of each sensor according to the first pulse feature.

[0063] Exemplarily, the operation data H = {h 1 , h 2} of the acceleration sensor and the brake pressure sensor are input into a preset pulse residual network model, and the pulse feature encoder extracts the feature information of the pulse sequence corresponding to each operation data, and respectively obtains the fault feature information of the pulse sequence corresponding to the operation data of the acceleration sensor and the fault feature information of the pulse sequence corresponding to the operation data of the brake pressure sensor, and determines the health status information of the acceleration sensor and the health status information of the brake pressure sensor respectively according to the obtained fault feature information of each sensor.

[0064] In the above vehicle sensor health status monitoring method, by obtaining the operation data of at least one sensor of the vehicle within a preset time period, the health status information of each sensor can be obtained according to each operation data and a preset pulse residual network model including a pulse feature encoder network. Since the pulse feature encoder network can extract a first pulse feature including fault feature information related to time, the processing accuracy of the pulse residual network model for the operation data of each sensor is improved, and the accuracy of the pulse residual network model for processing sensor operation data is improved. In addition, since the pulse residual network model is adopted, the problems of gradient disappearance and gradient explosion existing in the backpropagation process of the model can be overcome, and the operation data in the form of the original continuous signal can be converted into a pulse signal through the pulse feature encoder, so that the pulse signal can be processed for fault feature extraction. Compared with the traditional technology, the computing requirement of the model can be reduced, and the computing efficiency of the model can be further improved.

[0065] In the scenario of obtaining the health status information of each sensor according to each operation data and the preset pulse residual network model, the pulse residual network model includes a pulse classifier network in addition to the pulse feature encoder network. In one embodiment, as Figure 3 shown, the above S202 includes:

[0066] S301, according to each operation data and the pulse feature encoder network, obtain the first pulse feature corresponding to each operation data.

[0067] In this embodiment, the operation data of each sensor can be input into the pulse feature encoder network, and the pulse feature encoder network processes the operation data for pulse feature extraction to obtain the first pulse feature corresponding to each operation data. Exemplarily, the operation data of the acceleration sensor and the brake pressure sensor are input into the pulse feature encoder network, and the pulse feature encoder network can process the input operation data in the form of a continuous signal into a pulse sequence related to time, and extract the first pulse feature including the fault feature information of the acceleration sensor and the fault feature information of the brake pressure sensor from the pulse sequence.

[0068] S302, input the first pulse feature corresponding to each operation data into the pulse classifier network to obtain the health status information of each sensor.

[0069] Among them, the pulse classifier network refers to a network that classifies the data in the input network. In this embodiment, the first pulse features corresponding to each piece of operation data can be input into the pulse classification network. The pulse classification network can classify the first pulse features according to the feature information to obtain a classification result, and output the health status information of each sensor according to the classification result. Exemplarily, the first pulse feature including the fault feature information of the acceleration sensor is input into the pulse classification network. If the value of the offset fault in the fault feature information of the acceleration sensor in the first pulse feature is the largest, the result output by the pulse classification network is that the health status information corresponding to the acceleration sensor is in a fault state, and the fault category is an offset fault; or, the first pulse feature including the fault feature information of the brake pressure sensor is input into the pulse classification network. If the value of the instability fault in the fault feature information of the brake pressure sensor in the first pulse feature is the largest, the result output by the pulse classification network is that the health status information corresponding to the brake pressure sensor is in a fault state, and the fault category is an instability fault.

[0070] In this embodiment, according to each piece of operation data and the pulse feature encoder network, the first pulse features corresponding to each piece of operation data can be obtained. Thus, by inputting the first pulse features corresponding to each piece of operation data into the pulse classifier network, the health status information of each sensor can be obtained. Since the pulse classifier network can map the first pulse features containing fault feature information to the fault feature space, classify and identify the first pulse features, the accuracy of the pulse residual network model is improved, and further the accuracy of the pulse residual network model for processing each sensor's data is improved.

[0071] In the scenario of obtaining the first pulse features corresponding to each piece of the operation data according to each piece of the operation data and the pulse feature encoder network as described above, the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module. In one embodiment, as Figure 4 shown, the above S301 includes:

[0072] S401, input each piece of operation data into the static convolution module for pulse encoding to obtain a pulse sequence corresponding to each piece of operation data; the pulse sequence is a sequence related to time.

[0073] Among them, at least one static convolution encoder is included in the static convolution module, and the static convolution encoder is used to convert the input data into a pulse sequence. In this embodiment, the operation data of each sensor can be input into the static convolution module of the pulse feature encoder network for pulse encoding, so as to obtain a time-related pulse sequence corresponding to each piece of operation data. Exemplarily, by inputting the operation data of 3 sensors within 5 hours into the static convolution module, the pulse sequences corresponding to the operation data of the 3 sensors within 5 hours can be obtained.

[0074] S402. Input each pulse sequence into a pulse convolution module to obtain second pulse features corresponding to each piece of operation data.

[0075] Among them, the pulse convolution module is a module for performing feature extraction processing on the pulse sequence. Each pulse convolution module may include a pulse convolution layer, a pulse normalization layer, and a pulse pooling layer. In the embodiments of the present application, the pulse residual network model includes at least one pulse convolution module; the second pulse features output by the pulse convolution module refer to pulse sequences related to time corresponding to the fault feature information of each sensor.

[0076] In this embodiment, the pulse sequences corresponding to each sensor can be input into the pulse convolution module for feature extraction, so as to obtain each second pulse feature. For example, the pulse sequence corresponding to the acceleration sensor is input into the pulse convolution module. The pulse convolution module can extract feature information related to the fault information and output the extracted feature information as the second pulse feature of the operation data.

[0077] Optionally, the feature information related to the fault information may be the feature information of the distribution of the sensor data fluctuating up and down near the normal value, or the feature information of the distribution of the sensor data slowly deviating from the normal value, or the feature information of the distribution of the sensor data quickly exceeding the measurable range.

[0078] S403. Input the second pulse features corresponding to each piece of operation data into the pulse residual module to obtain each first pulse feature; among them, the richness of the information included in each first pulse feature is greater than the richness of the information included in each second pulse feature.

[0079] Among them, the pulse residual module refers to a module that uses a residual network to perform pulse feature addition processing or pulse feature splicing processing on the input data. In this embodiment, the second pulse features output by the pulse convolution module can be input into the pulse residual module to obtain each first pulse feature containing richer fault feature information.

[0080] Assume that the input of the l-th layer of the pulse residual module of the pulse residual network model is X l [t], and the output is S l [t]. The residual mapping of the pulse residual module is represented as Y l [t]=LIF(f(X l [t])), where f represents the non-linear mapping of the pulse convolution layer and the pulse normalization layer. Then the pulse residual module can be expressed as:

[0081] S 1 [t]=Φ(LIF(f(X l [t])), X l [t]=Φ(Yl [t], X l [t])

[0082] Wherein, Φ(·) represents the implementation manner of the residual module, and generally, it is implemented by adding corresponding pulse features or splicing pulse features.

[0083] It should be noted that in the process of backpropagation from the deep layer to the shallow layer in the pulse residual model, due to the relatively deep network structure, there will be problems of gradient disappearance and gradient explosion. Therefore, in the embodiments of the present application, an identity mapping is introduced on the basis of the pulse residual module. By setting Y l [t]=0, that is, setting f(X l [t]=0, at this time:

[0084] S l [t]=Φ(LIF(0)), X l [t]=Φ(0, X l [t])

[0085] Therefore, when the implementation manner of Φ(·) is addition, assuming that there are k pulse residual modules and the input is X l [t] and the output is S k [t], then the gradient of S k [t] with respect to X l [t] can be expressed as:

[0086]

[0087] According to the above formula, it can be obtained that when the identity mapping is introduced, the gradients of k pulse residual modules are all maintained at 1. Therefore, the pulse residual network model in the embodiments of the present application can overcome the problems of gradient disappearance and gradient explosion.

[0088] Optionally, the pulse feature encoder network can be a membrane-learnable leaky integrate-and-fire (LIF) network. It should be noted that the modeling of the information transmission between neurons and the biokinetics by the LIF model can be expressed as:

[0089]

[0090]

[0091] Wherein, V(t) represents the membrane potential, V rest represents the resting potential, T m represents the membrane time constant, I(t) represents the input current, r m represents the leakage resistance, represents the firing threshold, V rIndicates the reset potential.

[0092] Assume that the post - neuron receives spike information from K pre - neurons, and the learnable weights of each pre - neuron are W 1 , W 2 , …, W n , then the spiking residual network model can be expressed as:

[0093]

[0094]

[0095]

[0096] In the formula, S(t) represents the output of the post - neuron at time step S(t).

[0097] To more accurately describe the dynamic mechanism of the neurons inside the spiking residual network model, the following discrete time is given:

[0098]

[0099]

[0100] V[t] = H[t](1 - S[t])+V rest S[t]

[0101] In the formula, X[t] represents the input potential at time t, H[t] and V[t] both represent the membrane potential at time t. When the neuron fires, V[t] will become V rest , Θ(·) represents the Heaviside step function, and S[t] represents the output of the neuron at time t. Describes the dynamics of the neuron, then the LIF model can be expressed as:

[0102] It should be noted that in the above LIF model, T m As a hyper - parameter, it is usually set by experience. In the embodiments of the present application, in order to enable each neuron to have different forgetting gates and update gates and implement a dynamic information update mechanism, a learnable membrane time constant is introduced in each spiking convolutional layer, and T m Is set to a learnable function η(ω τ ), ω τ Represents the learnable parameter, then the membrane - learnable LIF model in the embodiments of the present application can be expressed as:

[0103]

[0104] When V restWhen set to 0, the above formula can be expressed as:

[0105]

[0106] In the formula, X[t] represents the current input information, and V[t - 1] represents the past information. represents that in the model, the forget gate forgets part of the past information. represents that the update gate retains part of the new information.

[0107] In this embodiment, by inputting each running data into the static convolution module for pulse coding, a time-related pulse sequence corresponding to each running data can be obtained. Then, by inputting each pulse sequence into the pulse convolution module, a second pulse feature corresponding to each running data can be obtained. Further, by inputting the second pulse feature corresponding to each running data into the pulse residual module, each first pulse feature whose information richness is greater than that of each second pulse feature can be obtained. Since the first pulse feature includes a higher richness of fault feature information, the accuracy of the pulse residual network model can be further improved, and thus a more accurate fault result of the sensor can be obtained. In addition, the static convolution coding module adopted in the embodiment of the present application has better performance and adaptability compared with the traditional method, thereby further improving the accuracy of the pulse residual network model.

[0108] In the scenario of inputting the first pulse feature corresponding to each running data into the pulse classifier network to obtain the health status information of each sensor, the pulse classifier network includes a pulse fully connected layer. In one embodiment, the above S302 includes: mapping the first pulse feature corresponding to each running data to the fault category space through the pulse fully connected layer to obtain the health status information of each sensor.

[0109] Among them, the pulse fully connected layer is used to integrate the local information with category discrimination in the pulse feature information output by the pulse feature encoder. Optionally, the pulse classifier network in the embodiment of the present application may include two layers of pulse fully connected layers.

[0110] Optionally, through the pulse fully connected layer, the first pulse feature corresponding to each running data can be mapped to the fault category space to obtain the fault information of each sensor, and then according to the fault information of each sensor, the health index information corresponding to each sensor can be obtained.

[0111] It should be noted that, in this embodiment, the first pulse features corresponding to each operation data can be input into the pulse fully connected layer. The pulse fully connected layer can map the first pulse features to the fault category space according to the fault feature information included in the input first pulse features, so as to obtain the status information corresponding to the first pulse features and the fault category information. According to the fault information of each sensor and combining with the signal whose amplitude grows exponentially, the health index information corresponding to each sensor can be obtained.

[0112] Exemplarily, the structure of the pulse residual network model of the present application can be as Figure 5 shown. As can be Figure 5 seen, the pulse residual network model includes at least one static convolution encoding module, at least one pulse convolution module, a plurality of pulse residual modules, and a pulse classifier network. Among them, the static convolution encoding module includes at least one static convolution encoder, each pulse convolution module includes a pulse convolution layer, a pulse standard layer, and a pulse pooling layer, each pulse residual module includes at least one pulse convolution layer, at least one pulse standard layer, and at least one pulse residual model, and the pulse classifier network includes two layers of pulse fully connected layers.

[0113] In this embodiment, by mapping the first pulse features corresponding to each operation data to the fault category space through the pulse fully connected layer, the health status information of each sensor can be obtained. Since the pulse fully connected layer can directly map the fault feature information of the sensor to the fault category space, the efficiency of the pulse residual network model in obtaining the sensor fault result is improved.

[0114] Before obtaining the health status information of each sensor according to each operation data and the preset pulse residual network model, it is necessary to train the pulse residual network model, and use the trained pulse residual network model as the preset pulse residual network model. In one embodiment, as Figure 6 shown, the above method further includes:

[0115] S501, obtaining the sample operation data of at least one test sensor of the vehicle within a preset time period and the golden standard health status information corresponding to each sample operation data.

[0116] Among them, the preset time period is the time period during which the sensors in the vehicle operate. Optionally, the preset time period can be 1 day or 3 days. The test sensors can include, but are not limited to, radar sensors, cameras, acceleration sensors, angular velocity sensors, brake pressure sensors, throttle pressure sensors, steering angle sensors, and other sensors. The sample operation data is the data generated by each test sensor during the operation of the vehicle. The golden standard health status information refers to the standard health status information corresponding to the sample operation data. Exemplarily, the obtained sample operation data of the brake pressure sensor is H = {h 1}, a sample data set can be established based on the collected operation data where N is the number of data set samples, and K i is the sensor fault category is the health index information of the sensor. For example, K i represents an unstable fault If the value of is 0.95, the gold standard health status information corresponding to the brake pressure sensor includes that the fault category of the brake pressure sensor is an unstable fault, and the health index information of the brake pressure sensor is 0.95.

[0117] In this embodiment, the computer device can obtain the sample operation data within a preset time period from the historical operation data of each sensor, and obtain the gold standard health status information of the sensor within the above preset time period from the corresponding maintenance information of each sensor.

[0118] S502: Input each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse features corresponding to each sample data.

[0119] Among them, the first sample pulse features include the fault category information corresponding to the sample operation data of each test sensor. In this embodiment, each sample operation data can be input into the initial pulse feature encoder network of the initial pulse residual network model, and the pulse feature encoder network can perform pulse feature extraction processing on the operation data to obtain the first sample pulse features corresponding to each sample operation data.

[0120] S503: Input each first sample pulse feature into the initial pulse classifier network of the initial pulse residual network model to obtain the sample health status information of each test sensor.

[0121] Among them, the initial pulse classifier network refers to a network that classifies the data input into the network. In this embodiment, the first sample pulse features corresponding to each sample operation data can be input into the initial pulse classification network, and the initial pulse classification network can classify the first pulse features according to the feature information to obtain a classification result, and output the sample health status information of each test sensor according to the classification result.

[0122] S504: Train the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain the pulse residual network model.

[0123] In this embodiment, the sample health state information of each test sensor output by the initial pulse residual network model can be compared with the gold standard health state information of each test sensor. According to the comparison result, the parameters of the initial pulse residual network model are adjusted and updated. Then, the sample health state information corresponding to the sample operation data is obtained according to the updated initial pulse residual network model until the value of the comparison result reaches stability. At this time, the initial pulse residual network model corresponding to the stable comparison result value can be determined as the pulse residual network model.

[0124] In this embodiment, by obtaining the sample operation data of at least one test sensor of the vehicle within a preset time period and the gold standard health state information corresponding to each sample operation data, and then inputting each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model, the first sample pulse features corresponding to each sample data can be obtained. Then, by inputting each first sample pulse feature into the initial pulse classifier network of the initial pulse residual network model, the sample health state information of each test sensor can be obtained. Thus, the initial pulse residual network model can be trained according to the sample health state information of each test sensor and the gold standard health state information of each test sensor, and then the pulse residual network model can be obtained. Since the gold standard health state information is obtained based on the sample operation data of each test sensor, the correctness and accuracy of the gold standard health state information are guaranteed. Further, by using the obtained gold standard health state information to train the initial pulse residual network model, the accuracy of the obtained pulse residual network model is improved, and thus the accuracy of the health state information of each sensor output is improved.

[0125] In the scenario of inputting each of the sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse features corresponding to each of the sample data, the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module, and an initial pulse residual module. In one embodiment, as Figure 7 shown, the above S502 includes:

[0126] S601, input each sample operation data into the initial static convolution module for pulse encoding to obtain the sample pulse sequence corresponding to each sample operation data; the sample pulse sequence is a time-related sequence.

[0127] Among them, the initial static convolution module is used to convert the input sample operation data into a pulse sequence. In this embodiment, the sample operation data of each test sensor can be input into the static convolution module of the initial pulse feature encoder network for pulse encoding, so as to obtain the time-related pulse sequence corresponding to each sample operation data.

[0128] S602. Input each sample pulse sequence into the initial pulse convolution module to obtain the second sample pulse features corresponding to the operation data of each sample.

[0129] Among them, the initial pulse convolution module is a module for extracting features from the pulse sequence. Each initial pulse convolution module includes an initial pulse convolution layer, an initial pulse normalization layer, and an initial pulse pooling layer. In the embodiments of the present application, the initial pulse residual network model may include multiple pulse convolution modules; the second pulse features output by the initial pulse convolution module refer to the pulse sequences related to time corresponding to the fault feature information of each test sensor. In this embodiment, the sample pulse sequences corresponding to each test sensor can be input into the initial pulse convolution module for feature extraction, so as to obtain the second sample pulse features of each.

[0130] S603. Input each second sample pulse feature into the initial pulse residual module to obtain the first sample pulse features of each; among them, the richness of the information included in each first sample pulse feature is greater than the richness of the information included in each second sample pulse feature.

[0131] Among them, the initial pulse residual module refers to a module that uses the residual network to perform pulse feature addition processing or pulse feature splicing processing on the input data. In this embodiment, the second sample pulse features output by the initial pulse convolution module can be input into the initial pulse residual module to obtain the first sample pulse features containing fault feature information with higher richness.

[0132] In this embodiment, by inputting the operation data of each sample into the initial static convolution module for pulse coding, the sample pulse sequences related to time corresponding to the operation data of each sample can be obtained. Then, by inputting each sample pulse sequence into the initial pulse convolution module, the second sample pulse features corresponding to the operation data of each sample can be obtained. Next, by inputting each second sample pulse feature into the initial pulse residual module, the first sample pulse features containing fault feature information with higher richness can be obtained, so as to improve the accuracy and precision of the fault feature information contained in the obtained first sample pulse features, and further improve the accuracy of the processing results of the operation data of each test sensor; in addition, since the operation data of each sample can be encoded into a pulse sequence, the resource requirements in the calculation process of the initial pulse residual network model can be reduced, thereby improving the calculation efficiency of the model.

[0133] In the scenario of training the initial pulse residual network model according to the sample health state information of each test sensor and the gold standard health state information of each test sensor to obtain the pulse residual network model, the parameters of the initial pulse residual network model can be adjusted by the value gradient substitution algorithm of the loss function. In one embodiment, such as Figure 8As shown above, S501 includes:

[0134] S701, obtaining the value of the loss function of the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor.

[0135] Among them, the loss function refers to the degree of deviation used to estimate the predicted value and the true value of the model. Optionally, the loss function can include the surplus loss function or the mean square error loss function. Optionally, the loss function of the fault category in the sample health status information of each test sensor and the fault category in the gold standard health status information of each test sensor can be calculated by the surplus loss function, and the loss function of the health index information in the sample health status information of each test sensor and the health index information in the gold standard health status information of each test sensor can be calculated by the mean square error loss function.

[0136] In this embodiment, the sample health status information of each test sensor and the gold standard health status information of each test sensor can be compared, and the value of the loss function of the initial pulse residual network model can be determined according to the comparison result. Optionally, it can be assumed that the time step of the initial pulse residual network model is T and the fault category is ε, then the number of output neurons is ε, and the initial pulse residual network model will run for T time steps, and the output of the model is Thus, the output pulses of each neuron are counted, and the pulse activation degree of each neuron is obtained through the following formula:

[0137]

[0138] In the formula, i is the fault category of the input sample operation data.

[0139] Optionally, the embodiment of the present application can use the surplus loss function to calculate the deviation degree between the fault category information in the sample health status information output by the initial pulse residual network model and the fault category information in the gold standard health status information. The surplus loss function can be expressed as:

[0140]

[0141] In the formula, κ represents the fault category, N represents the total number of sample operation data, margin and p are hyperparameters, κ p is the predicted label, indicating the index of the neuron with the maximum discharge rate:

[0142] Optionally, in the embodiments of the present application, the mean square loss function may be used to calculate the deviation degree between the health index information in the sample health status information output by the initial pulse residual network model and the health index information in the golden standard health status information. The mean square loss function may be expressed as:

[0143]

[0144] In the formula, represents the health index information in the sample health status information output by the initial pulse residual network model, represents the health index information in the golden standard health status information.

[0145] S702. Adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient replacement algorithm to obtain the pulse residual network model.

[0146] Among them, the gradient replacement algorithm refers to an algorithm that dynamically adjusts the learning rate of each parameter according to the value of the loss function. In this embodiment, the parameters of the initial pulse residual network model can be updated and optimized through an optimization algorithm based on gradient replacement according to the value of the loss function, so as to determine the initial pulse residual network model corresponding to the parameters that meet the preset threshold after updating as the pulse residual network model. Optionally, in the embodiments of the present application, the Adam optimization algorithm can be used to iteratively update the weights in the initial pulse residual network model based on the sample operation data, and the cosine annealing learning rate update mechanism is adopted. The Adam optimization algorithm can be expressed as:

[0147] η t =η min +0.5*(η max -η min )(1 + cos(πT vur / T max ))

[0148] In the formula, η t represents the current learning rate, η max represents the maximum learning rate, and the initial value can be set to 0.001 respectively. η min represents the minimum learning rate, and the initial value can be set to 0 respectively. The learning step of the learning rate can be set to 32.

[0149] In this embodiment, by using the sample health status information of each test sensor and the gold standard health status information of each test sensor, the value of the loss function of the initial pulse residual network model can be obtained. Further, according to the value of the loss function and the gradient replacement algorithm, the parameters of the initial pulse residual network model can be adjusted, so as to obtain a trained pulse residual network model. Since the gradient replacement algorithm is adopted, the learning step size of the iterative parameters of the initial pulse residual network model has a definite range during the training process, so that the values of the parameters in the model are more stable, and thus the stability and accuracy of the trained pulse residual network model are improved.

[0150] To verify the effectiveness of the proposed pulse residual network model, taking the health monitoring performance of the accelerator pedal sensor as an example, this application sets up a comparative experiment to illustrate the accuracy of the pulse residual network model of this application. In the comparative experiment, the evaluation results of the fault diagnosis accuracy and the health index prediction performance after processing the operation data of the sensor by the pulse residual network model of this application, the existing LIF-based pulse residual network model, and the existing membrane-learnable LIF-based pulse network model are described respectively. Among them, the LIF-based pulse residual network model does not have a learnable membrane time constant, and the membrane-learnable LIF-based pulse network model does not have a residual learning mechanism. As shown in Table 1, the experimental results of the pulse residual network model of this application, the existing LIF-based pulse residual network model, and the existing membrane-learnable LIF-based pulse network model are shown. For the accuracy of sensor fault diagnosis, the pulse residual network model of this application obtains a diagnosis accuracy of 96.93%, the diagnosis accuracy of the LIF-based pulse residual network model is 95.99%, and the accuracy of the membrane-learnable LIF-based pulse network model is 95.70%, indicating that the pulse characterization ability of the model can be improved by setting learnable membrane parameters to enhance the diagnosis performance; in addition, the pulse residual architecture can also better improve the gradient propagation of the model to better optimize the parameters. For the evaluation of the health index prediction performance, the mean square error value of the pulse residual network model is 0.0029, and its determination coefficient is 0.0395, which is better than the mean square error values of the LIF-based pulse residual network model and the membrane-learnable LIF-based pulse network model. For example, the evaluation coefficients of the health index prediction performance of the LIF-based pulse residual network model and the membrane-learnable LIF-based pulse network model are 0.9219 and 0.9223 respectively, which shows that the proposed method also has excellent performance in the health index prediction application based on the regression task. Moreover, the 96.93% diagnosis accuracy and the 0.0029 mean square error value of the pulse residual network model also demonstrate its excellent sensor health monitoring ability, which can accurately identify the abnormal state of the sensor, thus ensuring the safe driving of autonomous vehicles.

[0151] Table 1

[0152]

[0153] Taking the accelerator pedal pressure sensor, brake pressure sensor, and steering angle sensor as examples, this application evaluates the fault diagnosis results and health index prediction results of the pulse residual network model of this application and six other existing fault diagnosis models and prediction models for sensors. As shown in Table 2, the performance of the pulse residual network model, multi-pulse network model, convolutional pulse network model, multi-long short-term memory model, one-dimensional residual network model, first-layer wide convolutional deep neural network model, and two-dimensional convolutional neural network model for sensor fault diagnosis and health index prediction is presented. Among them, the multi-pulse network model is used for bearing fault diagnosis in the prior art; the convolutional pulse network model is an SNN model with a convolutional topology structure, which consists of multiple pulse convolutional layers, and this model is used for fault diagnosis of unmanned underwater vehicles in the prior art; the multi-long short-term memory model uses multiple recurrent neural units with the ability to learn sequence information to achieve feature learning, and 3 recurrent units are selected to process sensor data in the prior art; the one-dimensional residual network model is a one-dimensional implementation version of the residual network, which is used to process multi-sensor data and consists of multiple one-dimensional residual modules; the first-layer wide convolutional deep neural network model is a prediction model based on a wide convolutional kernel, which obtains a wide range of information perception through an ultra-large convolutional kernel; the two-dimensional convolutional neural network model is an architecture based on two-dimensional convolution, which uses two-dimensional convolution to learn the spatial feature representation of signals. For sensor fault diagnosis, the accuracy of the pulse residual network model has been improved by more than 1% compared with all the comparison methods. For example, the diagnostic accuracy of the pulse residual network model is 96.93%, while the diagnostic accuracies of the convolutional pulse network model, multi-long short-term memory model, one-dimensional residual network model, and first-layer wide convolutional deep neural network model are only 94.56%, 95.41%, 95.76%, and 94.14% respectively, which demonstrates that the pulse residual network model has considerable multi-temporal data processing capabilities. For sensor health index prediction, the pulse residual network model of this application has a considerable improvement compared with the comparison methods. For example, the determination coefficient of the pulse residual network model is 0.9540, while the determination coefficients of the convolutional pulse network model, multi-long short-term memory model, one-dimensional residual network model, and first-layer wide convolutional deep neural network model are 0.9432, 0.9494, 0.9434, and 0.8962 respectively. The above results also demonstrate the excellent performance of the pulse residual network model in health index prediction applications.

[0154] Table 2

[0155]

[0156]

[0157] For the convenience of understanding by those skilled in the art, the following provides a detailed introduction to the brightness adjustment method provided by this application. This method may include:

[0158] S1. Obtain the sample operation data of each sensor of the vehicle within a preset time period.

[0159] S2. Perform sliding segmentation on the obtained sample operation data to obtain a sample set for training the initial pulse residual network model, and perform [0, 1] normalization processing on the operation data in the sample set.

[0160] S3. Input the processed sample operation data into the initial pulse feature encoding network in the initial pulse residual network model.

[0161] S4. The static convolution module in the initial pulse feature encoding network performs pulse encoding processing on the input sample operation data to obtain a time-related pulse sequence corresponding to each sample operation data.

[0162] S5. The initial pulse convolution module in the initial pulse feature encoding network processes the time-related pulse sequence corresponding to each sample operation data to obtain a second sample pulse feature containing fault feature information.

[0163] S6. The initial pulse residual module in the initial pulse feature encoding network processes the second sample pulse feature to obtain a first sample pulse feature containing fault feature information. The richness of the fault feature information contained in the first sample pulse feature is greater than that of the fault feature information contained in the second sample pulse feature.

[0164] S7. According to the sample health status information of each test sensor and the gold standard health status information of each test sensor, obtain the value of the loss function of the initial pulse residual network model.

[0165] S8. According to the value of the loss function and the gradient replacement algorithm, adjust the parameters of the initial pulse residual network model to obtain a pulse residual network model.

[0166] S9. Obtain the operation data of at least one sensor of the vehicle within a preset time period.

[0167] S10. According to each operation data and the preset pulse residual network model, obtain the health status information of each sensor.

[0168] It should be understood that although the steps in the flowcharts involved in the above embodiments are sequentially shown according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0169] Based on the same inventive concept, an embodiment of the present application further provides a processing device for sensor data for implementing the above-mentioned processing method of sensor data. The implementation solution provided by this device to solve problems is similar to the implementation solution described in the above method. Therefore, the specific limitations in one or more embodiments of the following processing device for sensor data can refer to the limitations on the processing method of sensor data in the above text, and will not be repeated here.

[0170] In one embodiment, as Figure 9 shown, a processing device for sensor data is provided, including: a first acquisition module 10 and a second acquisition module 11, where:

[0171] The first acquisition module 10 is used to acquire the operation data of at least one sensor of the vehicle within a preset time period.

[0172] The second acquisition module 11 is used to obtain the health status information of each sensor according to the operation data and a preset pulse residual network model; wherein, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is used to extract the first pulse features of the operation data; the first pulse features are fault feature information corresponding to the operation data related to time.

[0173] The processing device for sensor data provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, and will not be repeated here.

[0174] In one of the embodiments, as Figure 10 shown, the pulse residual network model further includes a pulse classifier network. The above first acquisition module 10 includes: a first acquisition unit 101 and a second acquisition unit 102, where:

[0175] The first acquisition unit 101 is used to obtain the first pulse features corresponding to the operation data according to the operation data and the pulse feature encoder network;

[0176] A second acquisition unit 102 is configured to input the first pulse features corresponding to the respective operation data into a pulse classifier network to obtain the health status information of each sensor.

[0177] The processing device for sensor data provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0178] In one embodiment, the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module; please continue to refer to Figure 10 The above first acquisition unit 101 is specifically configured to:

[0179] Input the respective operation data into the static convolution module for pulse encoding to obtain a pulse sequence corresponding to each operation data; the pulse sequence is a sequence related to time; input the respective pulse sequences into the pulse convolution module to obtain a second pulse feature corresponding to each operation data; input the second pulse feature corresponding to each operation data into the pulse residual module to obtain each first pulse feature; wherein, the richness of the information included in each first pulse feature is greater than the richness of the information included in each second pulse feature.

[0180] The processing device for sensor data provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0181] In one embodiment, the pulse classifier network includes a pulse fully connected layer; please continue to refer to Figure 10 The above second acquisition unit 102 is specifically configured to: map the first pulse features corresponding to the respective operation data to the fault category space through the pulse fully connected layer to obtain the health status information of each sensor.

[0182] The processing device for sensor data provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0183] In one embodiment, the health status information includes fault information and health index information, and the health status information includes fault information and health index information; please continue to refer to Figure 10 The above second acquisition unit 102 is specifically configured to: map the first pulse features corresponding to the respective operation data to the fault category space through the pulse fully connected layer to obtain the fault information of each sensor; and obtain the health index information corresponding to each sensor according to the fault information of each sensor.

[0184] Optionally, the pulse feature encoder network is a membrane-learnable leaky integrate-and-fire model.

[0185] The processing device for sensor data provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0186] In one embodiment, please continue to refer to Figure 10 , the above method further includes: a third acquisition module 12, a fourth acquisition module 13, a fifth acquisition module 14, and a sixth acquisition module 15, where:

[0187] The third acquisition module 12 is configured to acquire sample operation data of at least one test sensor of the vehicle within a preset time period and the corresponding gold standard health status information of each sample operation data;

[0188] The fourth acquisition module 13 is configured to input each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse features corresponding to each sample data;

[0189] The fifth acquisition module 14 is configured to input each first sample pulse feature into the initial pulse classifier network of the initial pulse residual network model to obtain the sample health status information of each test sensor;

[0190] The sixth acquisition module 15 is configured to train the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain the pulse residual network model.

[0191] The processing device for sensor data provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effects are similar, which will not be elaborated here.

[0192] In one embodiment, please continue to refer to Figure 10 , the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module, and an initial pulse residual module. The above fourth acquisition module 13 includes: a third acquisition unit 131, a fourth acquisition unit 132, and a fifth acquisition unit 133, where:

[0193] The third acquisition unit 131 is configured to input each sample operation data into the initial static convolution module for pulse encoding to obtain a sample pulse sequence corresponding to each sample operation data; the sample pulse sequence is a sequence related to time;

[0194] The fourth acquisition unit 132 is configured to input each sample pulse sequence into the initial pulse convolution module to obtain the second sample pulse features corresponding to each sample operation data;

[0195] A fifth acquisition unit 133 is configured to input each second sample pulse feature into an initial pulse residual module to obtain each first sample pulse feature; wherein, the richness of the information included in each first sample pulse feature is greater than the richness of the information included in each second sample pulse feature.

[0196] The sensor data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0197] In one embodiment, please continue to refer to Figure 10 , the above sixth acquisition module 15 includes:

[0198] A sixth acquisition unit 151 is configured to obtain the value of the loss function of the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor;

[0199] A seventh acquisition unit 152 is configured to adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient replacement algorithm to obtain a pulse residual network model.

[0200] The sensor data processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be elaborated here.

[0201] Each module in the above sensor data processing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0202] In one embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0203] Obtain the operation data of at least one sensor of the vehicle within a preset time period;

[0204] According to each operation data and a preset pulse residual network model, obtain the health status information of each sensor; wherein, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is configured to extract the first pulse feature of each operation data; the first pulse feature is the corresponding fault feature information including each operation data related to time.

[0205] In one embodiment, the pulse residual network model further includes a pulse classifier network, and when the processor executes the computer program, the following steps are further implemented:

[0206] Based on each operation data and the pulse feature encoder network, obtain the first pulse feature corresponding to each operation data;

[0207] Input the first pulse feature corresponding to each operation data into the pulse classifier network to obtain the health status information of each sensor.

[0208] In one embodiment, the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module. When the computer program is executed by the processor, the following steps are further implemented: Input each operation data into the static convolution module for pulse encoding to obtain a pulse sequence corresponding to each operation data; the pulse sequence is a sequence related to time;

[0209] Input each pulse sequence into the pulse convolution module to obtain the second pulse feature corresponding to each operation data;

[0210] Input the second pulse feature corresponding to each operation data into the pulse residual module to obtain each first pulse feature; wherein, the richness of the information included in each first pulse feature is greater than the richness of the information included in each second pulse feature.

[0211] In one embodiment, the pulse classifier network includes a pulse fully connected layer. When the processor executes the computer program, the following steps are further implemented: Map the first pulse feature corresponding to each operation data to the fault category space through the pulse fully connected layer to obtain the health status information of each sensor.

[0212] In one embodiment, the health status information includes fault information and health index information. When the computer program is executed by the processor, the following steps are further implemented:

[0213] Map the first pulse feature corresponding to each operation data to the fault category space through the pulse fully connected layer to obtain the fault information of each sensor;

[0214] Obtain the health index information corresponding to each sensor according to the fault information of each sensor.

[0215] In one embodiment, when the processor executes the computer program, the following steps are further implemented: Obtain the sample operation data of at least one test sensor of the vehicle within a preset time period and the golden standard health status information corresponding to each sample operation data;

[0216] Input each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse feature corresponding to each sample data;

[0217] Input each first sample pulse feature into the initial pulse classifier network of the initial pulse residual network model to obtain the sample health status information of each test sensor;

[0218] Train an initial pulse residual network model based on the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain a pulse residual network model.

[0219] In one embodiment, the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module, and an initial pulse residual module. When the computer program is executed by a processor, the following steps are further implemented:

[0220] Input each sample operation data into the initial static convolution module for pulse encoding to obtain a sample pulse sequence corresponding to each sample operation data; the sample pulse sequence is a sequence related to time;

[0221] Input each sample pulse sequence into the initial pulse convolution module to obtain a second sample pulse feature corresponding to each sample operation data;

[0222] Input each second sample pulse feature into the initial pulse residual module to obtain each first sample pulse feature; wherein, the richness of the information included in each first sample pulse feature is greater than the richness of the information included in each second sample pulse feature.

[0223] In one embodiment, when the processor executes the computer program, the following steps are further implemented: Obtain the value of the loss function of the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor;

[0224] Adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient substitution algorithm to obtain a pulse residual network model.

[0225] Optionally, the pulse feature encoder network is a membrane-learnable leaky integrate-and-fire model.

[0226] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0227] Obtain the operation data of at least one sensor of the vehicle within a preset time period;

[0228] Obtain the health status information of each sensor according to each operation data and a preset pulse residual network model; wherein, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is used to extract the first pulse feature of each operation data; the first pulse feature is a fault feature information corresponding to each operation data related to time.

[0229] In one embodiment, the pulse residual network model further includes a pulse classifier network. When the computer program is executed by a processor, the following steps are further implemented:

[0230] Based on each operation data and the pulse feature encoder network, obtain the first pulse features corresponding to each operation data;

[0231] Input the first pulse features corresponding to each operation data into the pulse classifier network to obtain the health status information of each sensor.

[0232] In one embodiment, the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module. When the computer program is executed by the processor, the following steps are further implemented: Input each operation data into the static convolution module for pulse encoding to obtain a pulse sequence corresponding to each operation data; the pulse sequence is a sequence related to time;

[0233] Input each pulse sequence into the pulse convolution module to obtain the second pulse features corresponding to each operation data;

[0234] Input the second pulse features corresponding to each operation data into the pulse residual module to obtain each first pulse feature; wherein, the richness of the information included in each first pulse feature is greater than the richness of the information included in each second pulse feature.

[0235] In one embodiment, the pulse classifier network includes a pulse fully connected layer. When the processor executes the computer program, the following steps are further implemented: Map the first pulse features corresponding to each operation data to the fault category space through the pulse fully connected layer to obtain the health status information of each sensor.

[0236] In one embodiment, the health status information includes fault information and health index information. When the computer program is executed by the processor, the following steps are further implemented:

[0237] Map the first pulse features corresponding to each operation data to the fault category space through the pulse fully connected layer to obtain the fault information of each sensor;

[0238] According to the fault information of each sensor, obtain the health index information corresponding to each sensor.

[0239] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:

[0240] Obtain the sample operation data of at least one test sensor of the vehicle within a preset time period and the golden standard health status information corresponding to each sample operation data;

[0241] Input each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse features corresponding to each sample data;

[0242] Input the first sample pulse features of each into the initial pulse classifier network of the initial pulse residual network model to obtain the sample health status information of each test sensor;

[0243] Train the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain the pulse residual network model.

[0244] In one embodiment, the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module, and an initial pulse residual module. When the computer program is executed by a processor, the following steps are further implemented:

[0245] Input each sample operation data into the initial static convolution module for pulse encoding to obtain the sample pulse sequence corresponding to each sample operation data; the sample pulse sequence is a sequence related to time;

[0246] Input each sample pulse sequence into the initial pulse convolution module to obtain the second sample pulse feature corresponding to each sample operation data;

[0247] Input each second sample pulse feature into the initial pulse residual module to obtain the first sample pulse feature of each; wherein, the richness of the information included in each first sample pulse feature is greater than the richness of the information included in each second sample pulse feature.

[0248] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: Obtain the value of the loss function of the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor;

[0249] Adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient replacement algorithm to obtain the pulse residual network model.

[0250] Optionally, the pulse feature encoder network is a membrane-learnable leaky integrate-and-fire model.

[0251] In one embodiment, a computer program product is provided, including a computer program, which when executed by a processor implements the following steps:

[0252] Obtain the operation data of at least one sensor of the vehicle within a preset time period;

[0253] Obtain the health status information of each sensor according to each operation data and the preset pulse residual network model; wherein, the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is used to extract the first pulse feature of each operation data; the first pulse feature is a fault feature information corresponding to each operation data related to time.

[0254] In one embodiment, the pulse residual network model further includes a pulse classifier network, and when the computer program is executed by a processor, the following steps are further implemented:

[0255] According to each piece of operation data and the pulse feature encoder network, obtain the first pulse features corresponding to each piece of operation data;

[0256] Input the first pulse features corresponding to each piece of operation data into the pulse classifier network to obtain the health status information of each sensor.

[0257] In one embodiment, the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module. When the computer program is executed by a processor, the following steps are further implemented: Input each piece of operation data into the static convolution module for pulse encoding to obtain a pulse sequence corresponding to each piece of operation data; the pulse sequence is a sequence related to time;

[0258] Input each pulse sequence into the pulse convolution module to obtain the second pulse features corresponding to each piece of operation data;

[0259] Input the second pulse features corresponding to each piece of operation data into the pulse residual module to obtain each first pulse feature; wherein, the richness of the information included in each first pulse feature is greater than the richness of the information included in each second pulse feature.

[0260] In one embodiment, the pulse classifier network includes a pulse fully connected layer. When the processor executes the computer program, the following steps are further implemented: Map the first pulse features corresponding to each piece of operation data to the fault category space through the pulse fully connected layer to obtain the health status information of each sensor.

[0261] In one embodiment, the health status information includes fault information and health index information. When the computer program is executed by a processor, the following steps are further implemented:

[0262] Map the first pulse features corresponding to each piece of operation data to the fault category space through the pulse fully connected layer to obtain the fault information of each sensor;

[0263] According to the fault information of each sensor, obtain the health index information corresponding to each sensor.

[0264] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0265] Obtain the sample operation data of at least one test sensor of the vehicle within a preset time period and the golden standard health status information corresponding to each sample operation data;

[0266] Input each sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain the first sample pulse features corresponding to each sample data;

[0267] Input the first sample pulse features of each into the initial pulse classifier network of the initial pulse residual network model to obtain the sample health status information of each test sensor;

[0268] Train the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor to obtain the pulse residual network model.

[0269] In one embodiment, the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module, and an initial pulse residual module. When the computer program is executed by a processor, the following steps are further implemented:

[0270] Input each sample operation data into the initial static convolution module for pulse encoding to obtain the sample pulse sequence corresponding to each sample operation data; the sample pulse sequence is a sequence related to time;

[0271] Input each sample pulse sequence into the initial pulse convolution module to obtain the second sample pulse features corresponding to each sample operation data;

[0272] Input each second sample pulse feature into the initial pulse residual module to obtain the first sample pulse features; wherein, the richness of the information included in each first sample pulse feature is greater than the richness of the information included in each second sample pulse feature.

[0273] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: Obtain the value of the loss function of the initial pulse residual network model according to the sample health status information of each test sensor and the gold standard health status information of each test sensor;

[0274] Adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient replacement algorithm to obtain the pulse residual network model.

[0275] Optionally, the pulse feature encoder network is a membrane-learnable leaky integrate-and-fire model.

[0276] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0277] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0278] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0279] The above-described embodiments merely represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for monitoring the health status of vehicle sensors, characterized in that, the method includes: Obtaining the operation data of at least one sensor of the vehicle within a preset time period; According to each of the operation data and a preset pulse residual network model, obtaining the health status information of each of the sensors; wherein, the health status information includes fault information and health index information; the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is used to extract the first pulse features of each of the operation data; the first pulse features are fault feature information corresponding to each of the operation data and related to time; the pulse feature encoder network includes a static convolution module, a pulse convolution module and a pulse residual module; the obtaining process of the first pulse features corresponding to each of the operation data includes: Inputting each of the operation data into the static convolution module for pulse encoding to obtain a pulse sequence corresponding to each of the operation data; the pulse sequence is a sequence related to time; Inputting each of the pulse sequences into the pulse convolution module to obtain a second pulse feature corresponding to each of the operation data; Inputting the second pulse features corresponding to each of the operation data into the pulse residual module to obtain the first pulse features corresponding to each of the operation data; wherein, the richness of the information included in the first pulse features corresponding to each of the operation data is greater than the richness of the information included in the second pulse features corresponding to each of the operation data.

2. The method according to claim 1, characterized in that, the pulse residual network model further includes a pulse classifier network; the method further includes: Inputting the first pulse features corresponding to each of the operation data into the pulse classifier network to obtain the health status information of each of the sensors.

3. The method according to claim 2, characterized in that, the pulse classifier network includes a pulse fully-connected layer; the inputting the first pulse features corresponding to each of the operation data into the pulse classifier network to obtain the health status information of each of the sensors includes: Mapping the first pulse features corresponding to each of the operation data to the fault category space through the pulse fully-connected layer to obtain the health status information of each of the sensors.

4. The method according to claim 3, characterized in that, the mapping the first pulse features corresponding to each of the operation data to the fault category space through the pulse fully-connected layer to obtain the health status information of each of the sensors includes: Mapping the first pulse features corresponding to each of the operation data to the fault category space through the pulse fully-connected layer to obtain the fault information of each of the sensors; According to the fault information of each of the sensors, obtaining the health index information corresponding to each of the sensors.

5. The method according to any one of claims 1 to 4, characterized in that, the method further includes: Obtaining the sample operation data of at least one test sensor of the vehicle within a preset time period and the gold standard health status information corresponding to each of the sample operation data; Input each of the sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain first sample pulse features corresponding to each of the sample operation data; Input each of the first sample pulse features into the initial pulse classifier network of the initial pulse residual network model to obtain sample health state information of each of the test sensors; Train the initial pulse residual network model according to the sample health state information of each of the test sensors and the gold standard health state information of each of the test sensors to obtain the pulse residual network model.

6. The method according to claim 5, wherein, the initial pulse feature encoder network includes an initial static convolution module, an initial pulse convolution module and an initial pulse residual module, and the step of inputting each of the sample operation data into the initial pulse feature encoder network of the initial pulse residual network model to obtain first sample pulse features corresponding to each of the sample operation data includes: Input each of the sample operation data into the initial static convolution module for pulse encoding to obtain sample pulse sequences corresponding to each of the sample operation data; the sample pulse sequences are sequences related to time; Input each of the sample pulse sequences into the initial pulse convolution module to obtain second sample pulse features corresponding to each of the sample operation data; Input each of the second sample pulse features into the initial pulse residual module to obtain each of the first sample pulse features; wherein, the richness of the information included in each of the first sample pulse features is greater than the richness of the information included in each of the second sample pulse features.

7. The method according to claim 5, wherein, the step of training the initial pulse residual network model according to the sample health state information of each of the test sensors and the gold standard health state information of each of the test sensors to obtain the pulse residual network model includes: Obtain the value of the loss function of the initial pulse residual network model according to the sample health state information of each of the test sensors and the gold standard health state information of each of the test sensors; Adjust the parameters of the initial pulse residual network model according to the value of the loss function and the gradient replacement algorithm to obtain the pulse residual network model.

8. The method according to any one of claims 1 to 4, wherein, the pulse feature encoder network is a membrane-learnable leaky integrate-and-fire model.

9. A vehicle sensor health state monitoring device, wherein, the device includes: A first acquisition module, configured to acquire operation data of at least one sensor of a vehicle within a preset time period; A second acquisition module, configured to obtain the health status information of each of the sensors according to the respective operation data and a preset pulse residual network model; wherein, the health status information includes fault information and health index information; the pulse residual network model includes a pulse feature encoder network, and the pulse feature encoder network is configured to extract first pulse features of the respective operation data; the first pulse features are fault feature information corresponding to the respective operation data and related to time; the pulse feature encoder network includes a static convolution module, a pulse convolution module, and a pulse residual module; the process of obtaining the first pulse features corresponding to the respective operation data includes: Inputting the respective operation data into the static convolution module for pulse encoding to obtain pulse sequences corresponding to the respective operation data; the pulse sequences are sequences related to time; Inputting the respective pulse sequences into the pulse convolution module to obtain second pulse features corresponding to the respective operation data; Inputting the second pulse features corresponding to the respective operation data into the pulse residual module to obtain first pulse features corresponding to the respective operation data; wherein, the richness of the information included in the first pulse features corresponding to the respective operation data is greater than the richness of the information included in the second pulse features corresponding to the respective operation data.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, wherein, when the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium, having a computer program stored thereon, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, wherein, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images