Mechanical intelligent fault diagnosis method based on neuromorphic calculation and multi-modal data

Abstract

A mechanical intelligent fault diagnosis method based on neuromorphic calculation and multi-modal data comprises the steps that firstly, a vibration, current and torque multi-modal data set is preprocessed, and a multi-modal sample set is constructed; then establishing a multi-modal bionic pulse neural network model, realizing bionic coding of original multi-modal information, establishing a bionic fusion layer to realize bionic fusion of different sensor signal features, inputting a training set of a multi-modal sample set into the model for training, performing bionic extraction on personalized information in the multi-modal sample set, and fusing advanced representation; calculating classification loss, iteratively updating and optimizing model parameters until training is completed, and obtaining a final multi-modal bionic spiking neural network model; deploying the final multi-modal bionic spiking neural network model to a neuromorphic hardware system, inputting a test set in the multi-modal sample set into the neuromorphic hardware system, and obtaining a tag of a category corresponding to test set data; according to the invention, low-power-consumption and high-precision mechanical intelligent fault diagnosis based on multi-modal data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of mechanical equipment fault diagnosis, and specifically relates to a mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data. Background Art

[0002] The health status of mechanical equipment directly affects production efficiency and safety. Mechanical fault diagnosis plays a vital role in the maintenance and management of industrial equipment. Early detection and diagnosis of potential faults can improve productivity and reduce maintenance costs. Therefore, it is particularly important to develop efficient mechanical fault diagnosis methods.

[0003] Deep learning technology has demonstrated significant advantages in processing complex mechanical signals through its powerful feature extraction and pattern recognition capabilities, but it usually requires a large amount of labeled data for training, which is time-consuming and requires high computing resources. Traditional computing systems usually require high computing power and have difficulty in achieving long-term real-time monitoring at the edge. This fact has not received much attention in the public literature because people are usually more concerned about computing performance in different situations rather than energy efficiency. In addition, as industrial equipment becomes larger and more complex, the demand for richer and more comprehensive data is also increasing to analyze the operating status and health of the equipment. For complex machines, the data from a single sensor usually cannot fully reflect the health of the equipment, making it difficult to accurately diagnose faults.

[0004] Spiking neural network (SNN) is a computational model inspired by neurobiology that has been developed in recent years. It processes information by simulating the spiking activity of neurons and consumes energy only when neurons emit pulses. It has efficient spatiotemporal feature processing capabilities and low energy consumption characteristics, which makes the SNN model have great potential in realizing efficient and low-power intelligent diagnostic systems.

[0005] Although current research shows that SNN models have great potential in machine fault diagnosis ([1]Z.Xu,Y.Ma,Z.Pan,and X.Zheng,“Deep spiking residual shrinkage network for bearing fault diagnosis,”IEEE Transactions on Cybernetics,vol.54,no.3,pp.1608–1613,2024;[2]L.Xu and D.Ji,“Online fault diagnosis using bioinspired spike neuralnetwork,”IEEE Transactions on Industrial Informatics,pp.1–9,2024), their energy efficiency advantages in actual AI systems have not been fully utilized; in addition, existing SNN models still have limitations in processing complex multimodal data fusion. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data, which solves the shortcomings of traditional methods in comprehensively analyzing multimodal information and realizing long-term real-time online monitoring, and realizes low-power and high-precision mechanical intelligent fault diagnosis based on multimodal data.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data. First, the collected vibration, current and torque multimodal data sets are preprocessed. Secondly, a bionic pulse neural network model for multimodal data is established to convert the preprocessed data from different sensor sources into pulse sequences, bionically fusing the fault characteristics of different modes. Finally, it is deployed to a neuromorphic hardware system to verify the energy consumption advantage of neuromorphic computing.

[0009] A method for mechanical intelligent fault diagnosis based on neuromorphic computing and multimodal data, comprising the following steps:

[0010] Step 1: Normalize and preprocess the vibration, current, and torque multimodal data collected by the three sensors, limit the data range to the interval [0,1], build a multimodal sample set based on the normalized preprocessed data, and divide it into a training set and a test set;

[0011] Step 2: Design a bionic coding layer based on Integrate And Fire (IAF) spiking neurons and establish a multimodal bionic spiking neural network model; realize the bionic coding of the original multi-class information after normalization preprocessing through the bionic coding layer, complete the conversion of different types of data into pulse sequences, and obtain a multimodal pulse sequence; extract high-level representations from the multimodal pulse sequence through multiple parallel SNN modules, establish a bionic fusion layer based on IAF spiking neurons, realize the bionic fusion of different sensor signal features, and obtain deep pulse features; then the deep pulse features are flattened to one dimension through the flattening layer, and output through the linear pulse layer, and the neurons with the most pulses are selected to complete the accurate classification of the pulse pattern;

[0012] Step 3: Input the training set of the multimodal sample set obtained in step 1 into the multimodal bionic spiking neural network model constructed in step 2, bionic extract personalized information from the multimodal sample set and fuse high-level representations, that is, obtain bionic fusion high-level representations;

[0013] Step 4: Use the bionic fusion high-level representation obtained in step 3 to calculate the classification loss, optimize and update the parameters of the multimodal bionic spiking neural network model;

[0014] Step 5: Repeat steps 3 to 4, iterate and update the parameters of the optimized multimodal bionic spiking neural network model until the model training is completed to obtain the final multimodal bionic spiking neural network model;

[0015] Step 6: Deploy the final multimodal bionic pulse neural network model obtained in step 5 to the neuromorphic hardware system, and input the test set in the multimodal sample set obtained in step 1 into the neuromorphic hardware system to obtain the labels of the corresponding categories of the test set data to verify the energy consumption advantage of neuromorphic computing.

[0016] The step 1 is specifically as follows:

[0017] Normalize and preprocess the vibration, current, and torque multimodal data to make them in the range of [0, 1]:

[0018]

[0019] In the formula, x′ ij represents the i-th sample value of the j-th class of data in the multimodal data after normalization preprocessing, x ij is the i-th sample value of the j-th class data in the original multimodal data before normalization, min(x j ) and max(x j ) represent the minimum and maximum values ​​of the j-th category data respectively;

[0020] A multimodal sample set is constructed for the normalized preprocessed data, and the training set and test set are divided in a ratio of 8:2.

[0021] The step 2 is specifically as follows:

[0022] Based on the computational complexity of the Integrate And Fire (IAF) neuron, a variety of pulse representations are provided:

[0023]

[0024] Where t represents the time step, τ m is the time constant of the neuron, u(t) is the membrane potential of the neuron, and u rest is the reset potential, R is the equivalent resistance of the neuron ion channel, I(t) is the current of the synaptic input neuron, and w n is a single synaptic weight, represents the synaptic input pulse train, is the sth pulse of the nth synapse The time step of being excited; each neuron is synaptically connected to the input channel. If the sum of the weighted inputs exceeds the threshold membrane potential preset by the neuron, the neuron will generate a binary pulse output, and the membrane potential will return to the reset potential u rest , and enter the refractory period;

[0025] A parallel spike convolutional network architecture is constructed based on IAF neurons. Each SNN module consists of a spike convolution layer, an IAF spike neuron layer and a pooling layer. Different signals after normalization preprocessing are directly input into the multimodal bionic spike neural network model. After the bionic coding layer, the non-binary multimodal input is converted into a pulse pattern. Multiple parallel SNN modules receive input from the bionic coding layer, extract and fuse high-level representations of different sensor signals, and then use the bionic fusion layer to realize bionic fusion of multimodal signals. The output pulse of the bionic fusion layer is flattened to one dimension and output to the final IAF neuron layer through the linear pulse layer, that is, the neuron with the most pulses is selected, and the pulse pattern is classified by selectively discharging when a specific pulse pattern exists. Specifically, the pulses generated by the IAF neurons are counted and the most active category is selected.

[0026] The step 4 is specifically as follows:

[0027] All modules in the multimodal bionic spiking neural network model communicate through pulses. After calculating the number of pulses emitted by the output neuron in the entire time window, the cross entropy is used as the loss function, and the model is trained from scratch using error back propagation;

[0028]

[0029] In the formula, u l (t) is the vector of membrane potentials of C output neurons in the lth layer, C is the number of fault diagnosis categories, is the synaptic weight matrix of the cth class in the lth layer, is the output binary pulse of the l-1th layer, is the probability value predicted as class c, that is, the vector containing softmax, y c is the true label, is the loss function;

[0030] Synaptic weight matrix w l Update as follows,

[0031] w l =w l -θΔw l

[0032]

[0033] Where θ is the learning rate.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention proposes a mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data, constructs a multimodal bionic pulse neural network model, the bionic coding layer effectively encodes the information of signals from different sources into the pulse sequence, and the bionic fusion layer expands the modal representation method from different angles to achieve low-power and high-precision multimodal bionic signal fusion intelligent fault diagnosis; finally, the multimodal bionic pulse neural network model is deployed to the neuromorphic hardware system, which naturally reflects the unique advantages of neuromorphic computing in memory, energy consumption, latency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0037] Figure 2 This is a structural diagram of a multimodal bionic spiking neural network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention is described in detail below in conjunction with embodiments and drawings.

[0039] Reference Figure 1 , a mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data, comprising the following steps:

[0040] Step 1: According to the different data dimensions, the vibration, current, and torque multimodal data collected by the three sensors are normalized and preprocessed to limit the data range to the interval [0, 1] to facilitate subsequent conversion into binary pulse signals; a multimodal sample set is constructed based on the normalized data, and divided into a training set and a test set;

[0041] In view of the different numerical ranges and distributions of various types of data, this embodiment normalizes the vibration, current, and torque multi-modal data to be in the interval [0, 1]:

[0042]

[0043] In the formula, x′ ij represents the i-th sample value of the j-th class of data in the multimodal data after normalization preprocessing, x ij is the i-th sample value of the j-th class data in the original multimodal data before normalization, min(x j ) and max(x j ) represent the minimum and maximum values ​​of the j-th category data respectively;

[0044] A multimodal sample set is constructed for the normalized data, and the training set and test set are divided in a ratio of 8:2;

[0045] Step 2: Design a bionic coding layer based on Integrate And Fire (IAF) spiking neurons and establish a multimodal bionic spiking neural network model; realize the bionic coding of the original multi-class information after normalization preprocessing through the bionic coding layer, complete the effective conversion of different types of data into pulse sequences, and obtain a multimodal pulse sequence;

[0046] This embodiment is based on the Integrate And Fire (IAF) neuron and can provide a variety of pulse representations with relatively low computational complexity:

[0047]

[0048] Where t represents the time step, τ m is the time constant of the neuron, u(t) is the membrane potential of the neuron, and u rest is the reset potential, R is the equivalent resistance of the neuron ion channel, I(t) is the current of the synaptic input neuron, and w n is a single synaptic weight, represents the synaptic input pulse train, is the sth pulse of the nth synapse The time step of being excited; each neuron is synaptically connected to the input channel. If the sum of the weighted inputs exceeds the threshold membrane potential preset by the neuron, the neuron will generate a binary pulse output, and the membrane potential will return to the reset potential u rest , and enter the refractory period;

[0049] According to the difference information of multimodal data such as vibration, current, torque, etc., a parallel pulse convolution network architecture is constructed based on IAF neurons. Each SNN module consists of a pulse convolution layer, an IAF pulse neuron layer and a pooling layer. Different signals after normalization preprocessing are directly input into the multimodal bionic pulse neural network model. After the bionic encoding layer, the non-binary multimodal input is converted into a pulse pattern. The multimodal pulse sequence will be extracted through multiple parallel SNN modules respectively. A bionic fusion layer is established based on IAF pulse neurons to achieve bionic fusion of different sensor signal features. These deep pulse features are flattened to one dimension by the flattening layer and output through the linear pulse layer. The final bionic neuron completes the accurate classification of the pulse pattern by selectively discharging when a specific pulse pattern is detected.

[0050] Reference Figure 2 , this embodiment specifically includes the following steps: first, three parallel SNN modules receive input from the bionic encoding layer, extract and fuse high-level representations of different sensor signals, and enrich fault feature information; then, the bionic fusion layer is used to realize multimodal signal bionic fusion; finally, the output pulse of the bionic fusion layer is flattened to one dimension, and output through the linear pulse layer, and then sent to the final IAF neuron layer; these bionic neurons classify the pulse patterns by selectively discharging when there are specific pulse patterns, specifically, counting the pulses generated by the IAF neurons and selecting the most active category;

[0051] Step 3: Input the training set of the multimodal sample set obtained in step 1 into the multimodal bionic spiking neural network model constructed in step 2, bionic extract the rich personalized information in the multimodal sample set and fuse the high-level representation, that is, obtain the bionic fusion high-level representation;

[0052] Step 4: Use the bionic fusion high-level representation obtained in step 3 to calculate the classification loss, optimize and update the parameters of the model;

[0053] All modules in the multimodal bionic spiking neural network model of this embodiment communicate via pulses, and after calculating the number of pulses emitted by the output neuron in the entire time window, the cross entropy is used as the loss function, and the model is trained from scratch using error back propagation;

[0054]

[0055] In the formula, ul (t) is the vector of membrane potentials of C output neurons in the lth layer, C is the number of fault diagnosis categories, is the synaptic weight matrix of the cth class in the lth layer, is the output binary pulse of the l-1th layer, is the probability value predicted as class c, that is, the vector containing softmax, y c is the true label, is the loss function;

[0056] Synaptic weight matrix w l Update as follows,

[0057] w l =w l -θΔw l

[0058]

[0059] Where θ is the learning rate;

[0060] Step 5: Repeat steps 3 to 4, iterate and update the parameters of the optimized multimodal bionic spiking neural network model until the model training is completed to obtain the final multimodal bionic spiking neural network model;

[0061] Step 6: Deploy the final multimodal bionic pulse neural network model to the neuromorphic hardware system, and input the test set in the multimodal sample set obtained in step 1 into the neuromorphic hardware system to obtain the labels of the corresponding categories of the test set data to verify the energy consumption advantage of neuromorphic computing.

[0062] This embodiment is based on the multi-modal experimental data of the motor drive end bearing to verify the effectiveness of the method of the present invention:

[0063] The experimental bench used in this embodiment is driven by a motor and loaded by a magnetic eddy current brake; three types of sensors are placed, namely a KISTLER three-axis acceleration sensor, a Fluke i200s current clamp and a torque sensor; two three-axis acceleration sensors are arranged on both sides of the fan end and the drive end of the motor, respectively, and the three current clamps are connected to the U, V, and W phases of the motor respectively, and the torque sensor is located between the motor and the brake; the experiment collects three types of signals, vibration, current and torque, as well as monitoring data under eight different health states, including normal (NC), outer ring (OI), rolling element (BI), outer ring-inner ring composite (O-III), phase loss (PL), broken bar (RB), voltage imbalance (VIB), and short circuit (SC); the sampling frequency of vibration data is 25.6kHz, and that of current and torque is 12.8kHz.

[0064] In addition, the neuromorphic hardware system Speck is used to verify the proposed multimodal bionic spike neural network model. Speck can meet the hardware requirements of dynamic computing. Its processor only runs when receiving incoming events. Its hardware circuit design is used to implement asynchronous event-driven distributed convolution processing of spike sequences. At the same time, it provides a complete software tool chain, making the deployment of SNN algorithms more efficient, making it easy to achieve ultra-low power consumption that is always on ([3] Yao M, Richter O, Zhao G, et al. Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip. Nature Communications, 15(1): 4464, 2024).

[0065] In order to verify the effectiveness of the method of the present invention, the present invention is compared with different encoding methods and traditional artificial neural networks (ANN), including frequency encoding, delay encoding, incremental modulation encoding and the multimodal bionic pulse neural network model encoding proposed by the present invention; at the same time, the performance of single source data (i.e. current, torque and vibration) and multimodal data (i.e. data-level fusion and feature-level fusion) is compared. Each method takes the average value of five experiments as the result. As shown in Table 1, it can be clearly observed that the method of the present invention is very competitive compared with the traditional SNN method and ANN based on single source data. Table 2 shows the comparison of power consumption, energy and delay of the method of the present invention on three different types of hardware systems: NVIDIA GeForce GTX 3050GPU, Intel Core i5-12490FCPU and Speck2e Dev Kit neuromorphic chip. The power consumption of the neuromorphic hardware system is only 0.16% and 0.04% of the mainstream computing system (i.e. GPU and CPU), which proves that the neuromorphic computing combined with asynchronously driven neuromorphic hardware system has obvious advantages in power consumption and delay, and is suitable for always-online fault monitoring. Table 3 lists the specific power consumption of the three different power rails of Speck, specifically showing the power consumed by different channels of Speck. Table 4 shows the memory usage of different encoding methods and ANNs, showing that extremely low power consumption and efficient calculations can be achieved by using only binary (1 / 0) bit values.

[0066] Table 1 Comparison of experimental results of different encoding methods and ANN in single source data and multimodal data

[0067]

[0068] Table 2 Comparison of power, energy, and latency on different hardware systems

[0069]

[0070]

[0071] Table 3 Specific power of different channels on Speck

[0072]

[0073] Table 4 Different encoding methods and ANN memory usage (M)

[0074]

Claims

1. A mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data, characterized by: First, the collected multimodal data sets of vibration, current and torque are preprocessed. Second, a bionic pulse neural network model for multimodal data is established to convert the preprocessed data from different sensor sources into pulse sequences, bionically fusing the fault characteristics of different modes. Finally, it is deployed to a neuromorphic hardware system to verify the energy consumption advantage of neuromorphic computing.

2. A mechanical intelligent fault diagnosis method based on neuromorphic computing and multimodal data, characterized in that: The following steps are involved: Step 1: Normalize and preprocess the vibration, current, and torque multimodal data collected by the three sensors, limit the data range to the interval [0,1], build a multimodal sample set based on the normalized preprocessed data, and divide it into a training set and a test set; Step 2: Design a bionic coding layer based on the Integrate-And-Fire (IAF) spiking neuron and establish a multimodal bionic spiking neural network model; realize the bionic coding of the original multi-class information after normalization preprocessing through the bionic coding layer, complete the conversion of different types of data into pulse sequences, and obtain a multimodal pulse sequence; The multimodal pulse sequence is extracted through multiple parallel SNN modules to extract high-level representations, and a bionic fusion layer is established based on IAF pulse neurons to achieve bionic fusion of different sensor signal features and obtain deep pulse features. The deep spike features are then flattened to one dimension through the flattening layer and output through the linear spike layer. The neurons with the most spikes are selected to accurately classify the spike patterns. Step 3: Input the training set of the multimodal sample set obtained in step 1 into the multimodal bionic spiking neural network model constructed in step 2, bionic extract personalized information from the multimodal sample set and fuse high-level representations, that is, obtain bionic fusion high-level representations; Step 4: Use the bionic fusion high-level representation obtained in step 3 to calculate the classification loss, optimize and update the parameters of the multimodal bionic spiking neural network model; Step 5: Repeat steps 3 to 4, iterate and update the parameters of the optimized multimodal bionic spiking neural network model until the model training is completed to obtain the final multimodal bionic spiking neural network model; Step 6: Deploy the final multimodal bionic pulse neural network model obtained in step 5 to the neuromorphic hardware system, and input the test set in the multimodal sample set obtained in step 1 into the neuromorphic hardware system to obtain the labels of the corresponding categories of the test set data to verify the energy consumption advantage of neuromorphic computing.

3. The method according to claim 2, characterized in that The step 1 is specifically as follows: Normalize and preprocess the vibration, current, and torque multimodal data to make them in the [0,1] interval: In the formula, x' ij represents the i-th sample value of the j-th class of data in the multimodal data after normalization preprocessing, x ij is the i-th sample value of the j-th class data in the original multimodal data before normalization, min(x j ) and max(x j ) represent the minimum and maximum values ​​of the j-th category data respectively; A multimodal sample set is constructed for the normalized preprocessed data, and the training set and test set are divided in a ratio of 8:

2.

4. The method according to claim 2, characterized in that: The step 2 is specifically as follows: Based on the computational complexity of the Integrate And Fire (IAF) neuron, a variety of pulse representations are provided: Where t represents the time step, τ m is the time constant of the neuron, u(t) is the membrane potential of the neuron, and u rest is the reset potential, R is the equivalent resistance of the neuron ion channel, I(t) is the current of the synaptic input neuron, and w n is a single synaptic weight, represents the synaptic input pulse train, is the sth pulse of the nth synapse The time step of being excited; each neuron is synaptically connected to the input channel. If the sum of the weighted inputs exceeds the threshold membrane potential preset by the neuron, the neuron will generate a binary pulse output, and the membrane potential will return to the reset potential u rest , and enter the refractory period; A parallel spike convolutional network architecture is constructed based on IAF neurons. Each SNN module consists of a spike convolution layer, an IAF spike neuron layer and a pooling layer. Different signals after normalization preprocessing are directly input into the multimodal bionic spike neural network model. After the bionic coding layer, the non-binary multimodal input is converted into a pulse pattern. Multiple parallel SNN modules receive input from the bionic coding layer, extract and fuse high-level representations of different sensor signals, and then use the bionic fusion layer to realize bionic fusion of multimodal signals. The output pulse of the bionic fusion layer is flattened to one dimension and output to the final IAF neuron layer through the linear pulse layer, that is, the neuron with the most pulses is selected, and the pulse pattern is classified by selectively discharging when a specific pulse pattern exists. Specifically, the pulses generated by the IAF neurons are counted and the most active category is selected.

5. The method according to claim 2, characterized in that: The step 4 is specifically as follows: All modules in the multimodal bionic spiking neural network model communicate through pulses. After calculating the number of pulses emitted by the output neuron in the entire time window, the cross entropy is used as the loss function, and the model is trained from scratch using error back propagation; In the formula, u l (t) is the vector of membrane potentials of C output neurons in the lth layer, C is the number of fault diagnosis categories, is the synaptic weight matrix of the cth class in the lth layer, is the output binary pulse of the l-1th layer, is the probability value predicted as class c, that is, the vector containing softmax, y c is the true label, is the loss function; Synaptic weight matrix w l Update as follows, w l =w l -θΔw l Where θ is the learning rate.

Images