Event camera based cross-location dynamic visual mechanical intelligence fault diagnosis method

Abstract

A cross-position dynamic visual intelligent fault diagnosis method for machinery based on event cameras is proposed. First, dynamic visual data of mechanical equipment under different health states is collected using an event camera, and the event stream is converted into a two-channel positive / negative polarity event representation, which serves as the input to a deep neural network. Then, the rectangular region with the most events is adaptively extracted as the region of interest (ROI). Next, several ROI position variants are randomly generated in each training epoch, and a self-supervised event learning loss is constructed to ensure that the high-level feature vector representations of different position variants of the same sample tend to be consistent in the feature subspace. Simultaneously, using easily collected unlabeled parallel data, a cross-supervised learning loss based on cosine similarity contrastive learning is constructed to align the feature representations of the same health state between the source and target domains. Finally, the deep neural network is iteratively trained with the overall loss function as the optimization objective to achieve robust mechanical fault diagnosis under different camera positions.

Description

Technical Field

[0001] This invention belongs to the field of mechanical equipment health status monitoring and intelligent fault diagnosis technology, specifically relating to a cross-position dynamic vision intelligent fault diagnosis method for machinery based on an event camera. Background Technology

[0002] Health monitoring and fault diagnosis of mechanical equipment are of great significance in industrial sectors such as manufacturing, transportation, and aviation. Timely and accurate fault diagnosis can significantly improve the safety and reliability of equipment operation and greatly reduce maintenance costs; conversely, inadequate fault diagnosis may lead to damage to mechanical systems or even catastrophic failures, causing serious casualties and property losses.

[0003] Currently, the most common fault monitoring method in industrial settings is to use accelerometers to collect vibration signals and then combine this with deep learning methods to extract fault features, achieving high-precision fault identification. However, the installation of contact sensors alters the mechanical structure and is difficult to apply in conditions where installation space is limited or temperature is a significant factor. Non-contact solutions, such as eddy current sensors, are only suitable for metallic materials and have stringent installation clearance requirements, while laser vibration meters are expensive and have high requirements for the working environment, making them difficult to widely promote in industrial settings.

[0004] The event camera is a novel biomimetic vision sensor that asynchronously detects brightness changes at the pixel level. It only outputs an event when the pixel brightness change exceeds a preset threshold, thus having extremely high temporal resolution (on the order of microseconds) and extremely low latency. It can accurately capture the minute vibrations generated when mechanical parts are running at high speed, providing a new visual perception method for non-contact mechanical fault diagnosis.

[0005] Variations in camera mounting position are a key factor affecting the practicality of dynamic vision fault diagnosis methods. In real-world industrial scenarios, precisely fixing the relative position between the camera and mechanical equipment is extremely difficult, leading to frequent positional deviations. Changes in camera position cause significant alterations in the distribution of dynamic vision data (i.e., neighborhood shift), resulting in a sharp decline in the performance of diagnostic models trained at specific locations at other locations. Existing data-driven methods (Xiang Li, Shupeng Yu, YaguoLei, Naipeng Li, and Bin Yang. "Intelligent machinery fault diagnosis with event-based camera.") IEEE Transactions on Industrial Informatics (20, no. 1(2023): 380-389.) It is difficult to effectively address the above-mentioned cross-location fault diagnosis problem, and relevant research is urgently needed to improve the location robustness of dynamic visual fault diagnosis methods. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, the present invention aims to provide a cross-position dynamic visual mechanical intelligent fault diagnosis method based on event cameras. By using self-supervised event learning and cross-supervised learning based on unlabeled parallel data, the generalization ability and robustness of the deep learning fault diagnosis model under different camera positions are improved.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A cross-location dynamic vision-based intelligent fault diagnosis method for machinery based on event cameras includes the following steps: Step 1: Use an event camera to capture images of the vibration area of ​​the mechanical equipment to obtain dynamic visual data, i.e., an event stream; Step 2: Convert the event stream into a two-channel event representation and construct a sample dataset suitable for deep network processing, i.e., the source domain, as the input to the deep neural network; Step 3: Adaptively extract the region of interest from the event frames represented by the two-channel event representation, and select the rectangular region with the most events as the region of interest; Step 4: Randomly generate several location offset variants for each region of interest sample, and construct a self-supervised event learning loss function. This drives the variants at different locations of the same sample to tend to be consistent in the feature subspace; Step 5: Using unlabeled parallel data collected from different camera locations, i.e. the target domain, construct a cross-supervised learning loss function based on cosine similarity comparison learning. Align feature representations of the same health state in the source and target domains; Step 6: Using the overall loss function To optimize the objective, a deep neural network fault diagnosis model is iteratively trained, where, The cross-entropy classification loss is used for labeled training data. and These are the penalty weight coefficients for self-supervised loss and cross-supervised loss, respectively. Step 7: Input the test event data of the target domain into the trained network, output the mechanical health status prediction result, and complete the fault diagnosis.

[0008] Step 1 specifically involves the event camera asynchronously detecting brightness changes in each pixel, outputting an event only when the pixel brightness change exceeds a preset threshold, with each event represented as a four-dimensional vector. ,in and For pixel coordinates, For timestamps, The polarity of brightness change For positive polarity events, This is a negative polarity event; during the monitoring time All events collected internally constitute the event stream. , This represents the total number of events.

[0009] Step 2 specifically involves dividing the event stream into multiple samples, each sample containing a fixed number of... A series of events; the first The two-channel representation of each sample is as follows: ,in and Pixel-level cumulative count representations of positive and negative polarity events; each sample corresponds to a health status label. , constitute the training dataset The source domain is the target domain, and the test dataset is the target domain. From different camera positions.

[0010] Step 3 specifically involves: setting the width of the region of interest to be... Height is Pixel; the The region of interest for each sample is denoted as ,in and The coordinates of the bottom left pixel of the region of interest; by traversing all... The rectangular block selects the region of interest based on the number of events.

[0011] Step 4 specifically involves: assuming direction and The range of movement in each direction is respectively and Randomly generate for each original sample in each training round. There are several variants, among which , From the interval Randomly select an integer from within. From the interval Random integers are selected within the range of the original frame, and all variants are limited to the range of the original frame; the self-supervised loss function is defined as: This represents what is learned from the feature extraction module. High-level characteristics, Represents the center vector; The center vector is defined as: In the formula, This indicates the number of additional samples generated. For feature extractor, , .

[0012] Step 5 specifically involves: parallel datasets Includes unlabeled event data collected from different camera locations under the same unknown health state; parallel samples of the target domain. , For a given unlabeled health state, the contrastive loss is defined as: Cosine similarity is calculated as follows: In the formula, cos() represents the cosine similarity. and Representing the source and target domains respectively. High-level features of parallel samples within a category; The temperature scaling factor is used; the total loss of cross-supervised learning is defined as: .

[0013] Step 6 specifically involves defining the overall loss function as follows: Iteratively update network parameters using gradient descent: In the formula, For network parameters, The learning rate is used; iteration continues until the model converges or the maximum number of training epochs is reached.

[0014] The deep neural network comprises two modules: a feature extractor and a fault detector. The feature extractor consists of two convolutional layers followed by a max-pooling layer. The fault detector includes fully connected layers. The output layer of 1 neuron and the Softmax classification function This represents the number of mechanical health status categories.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention fully utilizes the high temporal resolution and low latency characteristics of event cameras to capture mechanical vibration information in a non-contact manner, overcoming the shortcomings of traditional contact sensors such as complex installation and impact on mechanical structures. The proposed self-supervised event learning method achieves domain generalization by generating position variants of the region of interest, without the need for additional labeled data. The proposed cross-supervised learning method makes full use of easily obtainable unlabeled parallel data and achieves cross-domain feature alignment through comparative learning, effectively solving the problem of cross-domain fault diagnosis caused by camera position changes. The diagnostic accuracy exceeds 97% in tasks with regular position changes and exceeds 91% in tasks with significant position changes, significantly outperforming existing methods. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram comparing data from an event camera and a traditional camera in an embodiment of the present invention.

[0018] Figure 3 This is a comparison diagram of the confusion matrix of the method and the comparison method of the present invention in the fault diagnosis task. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the embodiments and accompanying drawings.

[0020] Reference Figure 1 A cross-location dynamic vision-based intelligent fault diagnosis method for machinery based on event cameras includes the following steps: Step 1: Use an event camera to capture images of the vibration area of ​​the mechanical equipment to obtain dynamic visual data, i.e., an event stream; In this embodiment, the event camera detects the brightness change of each pixel asynchronously, and only outputs an event when the pixel brightness change exceeds a preset threshold. Each event is represented as a four-dimensional vector. In the formula, and These are the horizontal and vertical coordinates of the pixel that triggered the event. For event timestamps, The polarity of brightness change; when pixel brightness increases (Positive polarity event) When pixel brightness decreases (Negative polarity event); during the monitoring time Inside, the event camera can record a large number of events, forming an event stream. ,in For the first One event, The total number of events; this characteristic enables the event camera to capture high-frequency, low-amplitude vibrations of mechanical equipment with a time resolution on the order of microseconds; reference Figure 2 Compared to traditional frame cameras, event cameras only output the pixel polarity changes in the moving area, with low information redundancy, making them very suitable for capturing mechanical vibration features. Step 2: Convert the event stream into a two-channel event representation and construct a sample dataset suitable for deep network processing, i.e., the source domain, as the input to the deep neural network; Due to the asynchronous nature of event data, traditional deep learning methods struggle to directly process event streams. This embodiment divides the event stream into multiple samples, each containing a fixed number of... A series of events; the first The two-channel representation of a sample is as follows: In the formula, This represents the pixel-level cumulative count representation of positive polarity events. Pixel-level cumulative count representation of negative polarity events; each sample corresponds to a mechanical health status label. This constitutes a labeled training dataset. (Source domain, camera located at a fixed position), target domain, test dataset The training and test distributions differ due to the different camera positions (i.e., there is a neighborhood shift). Step 3: Adaptively extract the region of interest from the event frames represented by the two-channel event representation, and select the rectangular region with the most events as the region of interest; Event frames typically contain a large background area. Vibrations from the target mechanical component trigger the vast majority of events. In this embodiment, the Region of Interest (ROI) is defined as the rectangular region with the highest number of events. Let the width of the ROI be... Height is (pixel), the first The ROI of each sample is denoted as: In the formula, and These are the horizontal and vertical coordinates of the bottom left pixel of the ROI; by iterating through all... Rectangular blocks are selected, and the region with the most events is chosen as the ROI. The region is then adaptively located to the area of ​​most active mechanical vibration. Step 4: Randomly generate several location offset variants for each region of interest sample, and construct a self-supervised event learning loss function. This drives the variants at different locations of the same sample to tend to be consistent in the feature subspace; This embodiment simulates the impact of small-range changes in camera position on visual data, assuming... and They are respectively direction and The range of block movement in the direction is additionally generated for each ROI sample in each training epoch. Variations ,in: and Randomly select from the following integer ranges respectively: All variants are confined to the original frame and can be viewed as simulations of minor camera positional perturbations; intuitively, variants of the same sample at different positions are processed by the feature extractor. A similar high-level feature representation should then be obtained; let... For the first Feature vectors of each variant The feature vector of the original ROI, and the self-supervised loss function. Defined as the average L2 distance between each variant's eigenvector and its center vector: This represents what is learned from the feature extraction module. High-level characteristics, This represents the center vector.

[0021] center vector Defined as: In the formula, This indicates the number of additional samples generated. This refers to the feature extractor, i.e. , ;minimize This allows variants of the same sample at different locations to tend to cluster in the feature subspace, thereby learning position-invariant feature representations; Step 5: Using unlabeled parallel data collected from different camera locations, i.e. the target domain, construct a cross-supervised learning loss function based on cosine similarity comparison learning. Align feature representations of the same health state in the source and target domains; In real-world industrial scenarios, it is difficult to obtain labeled data from the same device at different camera positions. However, unlabeled parallel data (i.e., unlabeled data captured by the same operating device at different camera positions) is relatively easy to collect. Let the parallel dataset be... It includes the source domain and the target domain in the health state set. Parallel samples under various states, parallel data do not require detailed health status labeling; This embodiment proposes a cross-supervised learning method: it aims to concentrate parallel samples from the source and target domains with the same health state in the feature subspace, while separating samples with different health states; for parallel samples in the target domain... ( For a given unlabeled health state, the contrastive loss is defined as: In the formula, Represents cosine similarity. The cosine similarity is calculated as follows, using the temperature scaling factor: For the high-level feature vectors of the target domain samples, and These are the high-level feature vectors of the source domain under the same health state and under different health states, respectively. The numerator encourages features of samples from the same health state to be close, while the denominator promotes the separation of features of samples from different health states; the total loss of cross-supervised learning is the sum of the contrast losses of all parallel samples. minimize Simultaneously, it brings together the features of cross-domain samples with the same health status and separates the features of different health statuses to achieve unsupervised domain alignment; the self-supervised variant samples generated in step 4 can also be included in the comparative learning to further enhance the domain adaptation effect; Step 6: Using the overall loss function To optimize the objective, a deep neural network fault diagnosis model is iteratively trained; The overall optimization objective is defined as: In the formula, The cross-entropy classification loss is used for labeled training data. and These are the penalty weight coefficients for the self-supervised loss and the cross-supervised loss, respectively. In the mini-batch update of each training epoch, self-supervised variant samples are first generated according to step 4, and then the network parameters are updated according to the following gradient descent rule. : In the formula, The learning rate is used as the basis for the above iterations, which continue until the model converges or the maximum number of training epochs is reached. This embodiment's deep neural network architecture includes a feature extractor. The module consists of two parts: a feature extractor and a fault detector. The feature extractor comprises two convolutional layers (with kernel size 3, and 64 and 32 filters respectively) and a max-pooling layer. The fault detector includes a 128-neuron fully connected layer. The output layer of one neuron and the Softmax classification function The target number of health status categories; Step 7: Input the test event data of the target domain into the trained network, output the mechanical health status prediction result, and complete the fault diagnosis.

[0022] After training convergence in step 6, the unlabeled test data of the target domain (different camera positions) is processed according to steps 2 and 3 and then fed into the network. The Softmax classification layer outputs the probability of each health state, and the category with the highest probability is taken as the fault diagnosis result.

[0023] This embodiment verifies the effectiveness of the method of the present invention based on experimental data from a rotating machinery test bench. The test bench mainly consists of a motor, a rotating shaft, and rolling bearings (model: ER-8K). A total of 7 bearing health states are set, covering healthy state (H), inner ring failure (IR, mild / severe), rolling element failure (BF, mild / severe), and outer ring failure (OR, mild / severe), for a total of 6 failure types, as shown in Table 1.

[0024] Table 1 Health Status Information of Dynamic Vision Mechanical Fault Diagnosis Dataset Health Status Categories 1 2 3 4 5 6 7 Fault location N / A (Healthy) Inner ring (IR) Inner ring (IR) Rolling element (BF) Rolling element (BF) Outer ring (OR) Outer ring (OR) Severity of the fault N / A Mild Severe Mild Severe Mild Severe The event camera used in the experiment and its parameters are shown in Table 2. The event camera was positioned directly opposite the bearing body, and the internal parameters were calibrated using a checkerboard pattern.

[0025] Table 2 Parameters of the event camera used in the experiment model type Resolution (pixels) Delay (μs) Maximum throughput (Mevents / s) Gen 3.1 Prophesee Silicon retina (event camera) 640×480 200 50 The experiment used six camera positions (A, B, C, D, E, F): A, B, and C were fixed positions on the desktop (at the same height as the bearing); D, E, and F corresponded to the horizontal positions of A, B, and C, respectively, but the cameras were raised by approximately 5 cm and subjected to small, continuous positional perturbations by hand. Rotation speeds covered four conditions: 1000, 1500, 2000, and 2500 r / min. Each sample contained 20,000 events, and the default region of interest size was [size missing]. Pixels. All experiments were implemented on the PyTorch platform and an NVIDIA GeForce RTX 4090 GPU. The model parameters are shown in Table 3.

[0026] Table 3 Model parameters used in this embodiment parameter Value parameter Value Batch size 128 Additional sample number 10 Training rounds 500 x-direction range 30 Learning rate λ 1×10-4 y-direction range ry 30 Wide area of ​​interest 300 Self-supervised weights 1 High area of ​​interest 200 Cross-supervision weights 1 Temperature coefficient γ 2 — — To evaluate the effectiveness of the method of the present invention, three types of diagnostic tasks were set up: same-location tasks (training and testing from the same camera position), regular cross-location tasks (vertical position change, such as A→D), and significant cross-location tasks (horizontal position change, such as A→B). The comparison methods include: (1) DNN: basic deep neural network (using only (2) NoSelf: Ablation method to remove self-supervised modules; (3) NoCrossSup: Ablation method to remove cross-supervised modules; (4) DA: Domain adaptation method based on maximum mean difference (MMD).

[0027] Experimental results show that in the same-location task, all methods achieve a diagnostic accuracy close to 100%, validating the effectiveness of the constructed dataset. In the conventional cross-location task, the proposed method achieves a diagnostic accuracy exceeding 97%, significantly outperforming DNN (approximately 20%) and DA (approximately 93%). In the significantly cross-location task, the proposed method achieves an accuracy exceeding 91%, still outperforming the comparative methods. (See reference...) Figure 3 All seven health conditions can be accurately identified. Furthermore, with... Increase diagnostic performance As technology becomes increasingly saturated, dynamic visual non-contact mechanical fault diagnosis in industrial settings provides a robust and practical technical solution.

Claims

1. A cross-position dynamic vision-based intelligent fault diagnosis method for machinery based on event cameras, characterized in that: Event cameras are used to collect dynamic visual data caused by vibration of mechanical equipment under different health conditions, and the event stream is converted into a two-channel event representation. Adaptive extraction of regions of interest (ROIs), and construction of a self-supervised event learning loss function by generating ROI location variants. ; Constructing a cross-supervised learning loss function using unlabeled parallel data ; with the overall loss function Iterative training of a deep neural network enables robust mechanical fault diagnosis under different camera positions.

2. The method according to claim 1, characterized in that, Includes the following steps: Step 1: Use an event camera to capture images of the vibration area of ​​the mechanical equipment to obtain dynamic visual data, i.e., an event stream; Step 2: Convert the event stream into a two-channel event representation and construct a sample dataset suitable for deep network processing, i.e., the source domain, as the input to the deep neural network; Step 3: Adaptively extract the region of interest from the event frames represented by the two-channel event representation, and select the rectangular region with the most events as the region of interest; Step 4: Randomly generate several positional offset variants for each region of interest sample, and construct a self-supervised event learning loss function. This drives the variants at different locations of the same sample to tend to be consistent in the feature subspace; Step 5: Using unlabeled parallel data collected from different camera locations, i.e. the target domain, construct a cross-supervised learning loss function based on cosine similarity comparison learning. Align feature representations of the same health state in the source and target domains; Step 6: Using the overall loss function To optimize the objective, a deep neural network fault diagnosis model is iteratively trained, where, The cross-entropy classification loss is used for labeled training data. and These are the penalty weight coefficients for self-supervised loss and cross-supervised loss, respectively. Step 7: Input the test event data of the target domain into the trained network, output the mechanical health status prediction result, and complete the fault diagnosis.

3. The method according to claim 2, characterized in that, Step 1 specifically involves the event camera asynchronously detecting brightness changes in each pixel, with each event represented as a four-dimensional vector. ,in and For pixel coordinates, For timestamps, The polarity of brightness change For positive polarity events, This is a negative polarity event; during the monitoring time All events collected internally constitute the event stream. , This represents the total number of events.

4. The method according to claim 3, characterized in that, Step 2 specifically involves dividing the event stream into multiple samples, each sample containing a fixed number of... A series of events; the first The two-channel representation of each sample is as follows: ,in and Pixel-level cumulative count representation frames for positive and negative polarity events, respectively; each sample corresponds to a health status label. , constitute the training dataset The source domain is the target domain, and the test dataset is the target domain. From different camera positions.

5. The method according to claim 4, characterized in that, Step 3 specifically involves: setting the width of the region of interest to be... Height is Pixel; the The region of interest for each sample is denoted as ,in and The coordinates of the bottom left pixel of the region of interest; by traversing all... The rectangular block selects the region of interest based on the number of events.

6. The method according to claim 5, characterized in that, Step 4 specifically involves: assuming direction and The range of movement in each direction is respectively and Randomly generate for each original sample in each training round. There are several variants, among which , From the interval Randomly select an integer from within. From the interval Random integers are selected within the range of the original frame, and all variants are limited to the range of the original frame; the self-supervised loss function is defined as: This represents what is learned from the feature extraction module. High-level characteristics, Represents the center vector; The center vector is defined as: In the formula, This indicates the number of additional samples generated. For feature extractors, , .

7. The method according to claim 6, characterized in that, Step 5 specifically involves: parallel datasets Includes unlabeled event data collected from different camera locations under the same unknown health condition; Parallel samples in the target domain , For a given unlabeled health state, the contrastive loss is defined as: Cosine similarity is calculated as follows: In the formula, cos() represents the cosine similarity. and Representing the source and target domains respectively. High-level features of parallel samples within a category; This is the temperature scaling factor; The total loss of cross-supervised learning is defined as: 。 8. The method according to claim 7, characterized in that, Step 6 specifically involves defining the overall loss function as follows: In the formula, Cross-entropy classification loss, and To penalize the weight coefficients, the network parameters are iteratively updated using gradient descent. In the formula, For network parameters, The learning rate is used; iteration continues until the model converges or the maximum number of training epochs is reached.

9. The method according to claim 8, characterized in that, The deep neural network comprises two modules: a feature extractor and a fault detector. The feature extractor consists of two convolutional layers followed by a max-pooling layer. The fault detector includes fully connected layers. The output layer of 1 neuron and the Softmax classification function This represents the number of mechanical health status categories.

Images