Equipment vibration measurement device and method based on machine vision and laser radar

Abstract

The invention provides an equipment vibration measurement device based on machine vision and a laser radar. The equipment vibration measurement device comprises a collaborative sensing system, a visual vibration inversion module, a multi-modal feature fusion and fault knowledge base module and an edge end lightweight intelligent diagnosis module. The collaborative sensing system is used for collecting a video stream of a target area and providing an absolute displacement reference; the visual vibration inversion module is used for processing the video stream to generate a vibration spectrogram; the multi-modal feature fusion and fault knowledge base module is used for constructing a multi-dimensional feature vector and a feature knowledge base; and the edge end lightweight intelligent diagnosis module is used for identifying a fault type and outputting a diagnosis result and confidence. According to the invention, non-contact, high-precision and full-automatic vibration state sensing and fault diagnosis are realized, the core target is to completely get rid of the dependence on a contact sensor, the limitation of a pure vision scheme in a complex industrial environment is overcome, and finally millisecond-level real-time fault identification and early warning are realized on the edge side with limited resources.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of equipment state monitoring and fault diagnosis, and particularly relates to a non-contact equipment vibration measurement device and method based on machine vision and laser radar (laser range finder), which is particularly suitable for real-time monitoring of the vibration state of key equipment in extreme and closed environments such as the nuclear industry, aerospace, etc. and early fault diagnosis. BACKGROUND

[0002] The vibration state of key power components of equipment (such as main circulating pumps, steam generators, hydraulic valve groups, etc.) is an important indicator of its health. Currently, the mainstream vibration monitoring technology mainly relies on contact sensors, such as piezoelectric accelerometers. However, in actual applications, especially in the closed space of the nuclear industry with high temperature, high pressure, and high salt fog, such technology has obvious defects: 1. Difficult to lay and low survival rate: the installation space of the sensor is limited, the cable laying is complex, and the extreme environment easily leads to performance degradation or damage of the sensor itself.

[0003] 2. Limited coverage: single-point measurement cannot fully reflect the overall vibration state of large equipment, and there are many blind spots and hidden parts.

[0004] 3. Early warning lag: traditional manual inspection and fixed-point monitoring are inefficient, and cannot realize real-time perception of the state and early warning of sudden failures.

[0005] In recent years, non-contact visual measurement technology has developed. For example, the Simcenter VVM technology developed by Siemens, based on phase-based motion amplification, can realize nanoscale micro-vibration visual measurement, and can be deeply integrated with CAE simulation to form a "test-simulation" closed loop. However, the VVM and similar technical solutions still have the following limitations: 1. Precision and reliability are greatly affected by the environment: changes in light, background interference, smoke, etc. can seriously affect the measurement accuracy.

[0006] 2. Difficult to achieve absolute measurement: visual methods usually measure relative displacement, which has limitations in quantitative analysis.

[0007] 3. Poor ability to distinguish specific faults: for faults with similar frequency characteristics (such as "misalignment" and "loose"), visual data alone is prone to misjudgment.

[0008] Therefore, the existing technical solutions are difficult to achieve high-reliability, real-time, and intelligent vibration fault diagnosis in extreme and closed environments such as the nuclear industry. SUMMARY

[0009] To address the shortcomings of existing vibration monitoring technologies (including contact sensors and high-end vision measurement systems such as VVM) in terms of adaptability to extreme environments, real-time performance, and intelligent diagnostic capabilities, this invention aims to provide a device and method for measuring equipment vibration based on machine vision and lidar. This device aims to achieve non-contact, high-precision, and fully automated vibration state perception and fault diagnosis. Its core objective is to completely eliminate reliance on contact sensors and overcome the limitations of pure vision solutions in complex industrial environments, ultimately achieving millisecond-level real-time fault identification and early warning at resource-constrained edge computing environments.

[0010] The specific plan is as follows: A device for measuring equipment vibration based on machine vision and lidar, characterized in that it comprises: The collaborative sensing system consists of a high-speed camera and a laser rangefinder, which are synchronously controlled through a network protocol. They are used to acquire video streams of the target area and provide an absolute displacement reference. The visual vibration inversion module processes the video stream acquired by the high-speed camera and, based on phase analysis, motion amplification and visual vibration inversion, converts the temporal changes in the video stream into the inverted vibration signal of the device, generating a vibration spectrum. The multimodal feature fusion and fault knowledge base module receives vibration frequency signals and high-frequency displacement data from the laser rangefinder, aligns and fuses the temporal data using a dynamic time warping algorithm, extracts envelope features, and constructs a multidimensional feature vector and feature knowledge base. The edge-end lightweight intelligent diagnostic module receives multidimensional feature vectors and the vibration spectrum generated from the video stream, identifies the fault type, and outputs diagnostic results and confidence levels.

[0011] Furthermore, the laser rangefinder is an XYZ three-axis laser rangefinder with a sampling rate of ≥4KHz for each axis; the frame rate of the high-speed camera is ≥1KHz.

[0012] Furthermore, the visual vibration inversion module enhances the micro-vibration signal through a sub-pixel-level motion amplification algorithm; extracts the luminance component in the HSV color space, performs a windowed Fourier transform on the temporal pixel matrix, and generates a vibration spectrum.

[0013] Furthermore, the multimodal feature fusion and fault knowledge base module extracts envelope features through Hilbert transform to construct a multidimensional feature vector containing fundamental frequency, harmonic energy ratio, sub-frequency energy, sideband energy, and vibration intensity; at the same time, it uses principal component analysis to reduce dimensionality and eliminate redundant information to construct a feature knowledge base containing typical faults.

[0014] Furthermore, the edge-end lightweight intelligent diagnostic module is a dual-channel hybrid network structure, including: a CNN branch for processing vibration spectrograms and extracting spatial features; an SVM branch for processing structured feature vectors; an attention fusion layer for dynamically weighting and fusing the spatial features extracted by the CNN branch and the structured feature vectors processed by the SVM branch; and a classifier for outputting fault type and confidence based on the fused features.

[0015] Furthermore, the fusion weight αᵢ during dynamic weighted fusion in the attention fusion layer is calculated according to the following formula: ; in, ; f i It is the i-th feature from the CNN branch or the SVM branch; These are weights calculated using an attention network; vᵀ is a weight vector of the hidden layer that maps the hidden layer state to a final scalar score; b is the bias term.

[0016] This invention also provides a method for measuring device vibration based on machine vision and lidar. The method includes: S1, acquiring video stream and laser displacement signal of the target device; S2, performing visual vibration inversion processing and windowed Fourier transform on the acquired video stream to obtain a vibration spectrum and the inverted vibration signal of the device; S3, performing spatiotemporal alignment and feature fusion on the inverted vibration signal and laser displacement signal to generate a multidimensional feature vector; S4, inputting the generated multidimensional feature vector and vibration spectrum into a lightweight dual-channel intelligent diagnostic model for fault diagnosis, and outputting the fault type and confidence level.

[0017] Furthermore, in step S3, a dynamic time warping algorithm is used to perform spatiotemporal alignment and feature fusion of the inverted vibration signal and the laser displacement signal, as follows: ; in, It is a local distance metric between sequences Q and C at points i and j, and a normalized path. It satisfies boundary, monotonic, and continuity constraints; Q and C are the visual vibration sequence and the laser displacement sequence, respectively; DTW(Q, C) is the minimum cumulative distance between two sequences.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] 1. Achieved highly robust sensing under extreme environments.

[0020] By combining and complementing vision and laser technologies, we have achieved both full-field measurement capabilities and an absolute accuracy benchmark, effectively overcoming the limitations of single sensors in high-temperature, high-pressure, and high-interference environments.

[0021] 2. Achieved a leap from "offline analysis" to "real-time diagnosis".

[0022] By designing an extremely lightweight model for edge computing, the "collection-post-processing-analysis" process, which traditional VVM and other technologies require several hours to complete, is shortened to online real-time diagnosis within 50 milliseconds, meeting the timeliness requirements for fault early warning.

[0023] 3. Improved the accuracy and automation level of fault diagnosis.

[0024] By deeply integrating physical mechanisms and AI algorithms, a dedicated fault feature knowledge base has been built, making the diagnostic process interpretable and the results more reliable. It has achieved autonomous and accurate identification and classification of four typical faults with an accuracy rate of ≥85%. Attached Figure Description

[0025] Figure 1 This is a block diagram of the testing device in this invention; Figure 2 This is a flowchart of the testing method in this invention. Detailed Implementation

[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The specific implementation methods of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1:

[0028] like Figure 1 As shown, this embodiment provides a device vibration measurement device based on machine vision and lidar, including a collaborative sensing system, a visual vibration inversion module, a multimodal feature fusion and fault knowledge base module, and a lightweight intelligent diagnostic module at the edge.

[0029] The collaborative sensing system consists of a high-speed camera and an XYZ three-axis laser rangefinder, which are synchronously controlled through a network protocol. They are used to collect video streams of the target area and provide an absolute displacement reference. The frame rate of the high-speed camera is ≥1KHz, and the sampling rate of each axis of the XYZ three-axis laser rangefinder is ≥4KHz.

[0030] The visual vibration inversion module processes the video stream captured by the high-speed camera. It enhances the micro-vibration signal using a sub-pixel-level motion amplification algorithm, extracts the luminance component in the HSV color space, performs a windowed Fourier transform on the temporal pixel matrix, and converts the temporal variations in the video stream into the device's inverted vibration signal, generating a vibration spectrum. The multimodal feature fusion and fault knowledge base module receives vibration frequency signals and high-frequency displacement data from the laser rangefinder. It extracts envelope features using Hilbert transform, constructs a multidimensional feature vector containing the fundamental frequency, harmonic energy ratio, sub-frequency energy, sideband energy, and vibration intensity, aligns and fuses the temporal data using a dynamic time warping algorithm, extracts the envelope features, and uses principal component analysis to reduce dimensionality and eliminate redundant information, constructing a feature knowledge base containing typical faults. The lightweight intelligent diagnostic module at the edge employs a dual-channel hybrid network structure. It receives multi-dimensional feature vectors and vibration spectrograms generated from the video stream, identifies the fault type, and outputs diagnostic results and confidence levels. This includes: a CNN branch for processing the vibration spectrogram and extracting spatial features; an SVM branch for processing structured feature vectors; an attention fusion layer for dynamically weighted fusion of the spatial features extracted by the CNN branch and the structured feature vectors processed by the SVM branch; and a classifier that outputs the fault type and confidence level based on the fused features. The fusion weight αᵢ in the attention fusion layer during dynamic weighted fusion is calculated according to the following formula:

[0031] ;

[0032] in, .

[0033] In the formula, f i It is the i-th feature from the CNN branch or the SVM branch.

[0034] The weights are calculated using an attention network, representing the importance of the feature to the current diagnostic task.

[0035] vᵀ is a weight vector of the hidden layer that maps the hidden layer state to a final scalar score.

[0036] b is a bias term, whose main function is to increase the flexibility and fitting ability of the model.

[0037] Example 2: As Figure 2 As shown, this embodiment provides a corresponding measurement method based on the measuring device provided in Embodiment 1. The method includes: S1. Acquire the video stream and laser displacement signal of the target device; S2. Perform visual vibration inversion processing and window Fourier transform on the acquired video stream to obtain the vibration spectrum and the inverted vibration signal of the device; S3. The dynamic time warping algorithm is used to perform spatiotemporal alignment and feature fusion of the inverted vibration signal and the laser displacement signal to generate a multidimensional feature vector, as follows: ;

[0038] in, It is a local distance metric between sequences Q and C at points i and j, and a normalized path. It satisfies boundary, monotonic, and continuity constraints.

[0039] Q and C represent the visual vibration sequence and the laser displacement sequence, respectively.

[0040] DTW(Q, C) is the minimum cumulative distance between two sequences, finding the optimal alignment path.

[0041] S4. Input the generated multidimensional feature vector and vibration spectrum into the lightweight dual-channel intelligent diagnostic model for fault diagnosis, and output the fault type and confidence level. Example 3: This example is based on the diagnostic method provided in Example 2, diagnosing key components such as the pump body and bearing housing of the main circulation pump. A collaborative sensing system is deployed near these key components. A high-speed camera (1kHz frame rate) is aimed at the pump body to capture its global vibration pattern; an XYZ three-axis laser rangefinder (4kHz per axis) is precisely aimed at specific points such as the coupling and anchor bolts to measure the absolute displacement in key directions. Hardware synchronization of the camera and laser rangefinder is achieved through an industrial switch equipped with the PTP precision clock protocol, ensuring the consistency of all data acquisition timescales, with synchronization errors controlled within 1 millisecond.

[0042] S1. Acquire the video stream and laser displacement signal of the target device. S2. Perform visual vibration inversion processing and windowed Fourier transform on the acquired video stream to obtain the vibration spectrum and the inverted vibration signal of the device. S21. Video stream acquisition and motion amplification.

[0043] The edge computing unit receives the video stream from the high-speed camera and immediately invokes the visual vibration inversion module. First, it runs a phase-based motion amplification algorithm to amplify the micron-level vibrations on the pump surface to a level visible to the human eye. S22, Frequency Domain Analysis and Spectrum Generation

[0044] In the HSV color space, the V (luminance) channel of each frame is extracted. A windowed Fourier transform is performed on the sequence of pixel luminance changes over time for a specific region of interest (ROI) to generate a vibration spectrum map of that region, visually displaying the main frequency components of the vibration. S3. A dynamic time warping algorithm is used to perform spatiotemporal alignment and feature fusion of the inverted vibration signal and the laser displacement signal, generating a multidimensional feature vector. S31. Spatiotemporal alignment of multi-source data.

[0045] The multimodal feature fusion module synchronously reads the high-frequency displacement data from the laser rangefinder and uses a dynamic time warping algorithm to precisely align it with the vibration signal retrieved visually on the time axis.

[0046] S32, Fault Feature Vector Extraction

[0047] From the fused data, the energy amplitudes and sideband features of the fundamental frequency (1X), second harmonic (2X), and 0.5 harmonic (0.5X) are extracted to form a structured multidimensional feature vector. S33, Dual-channel intelligent diagnostic reasoning.

[0048] The feature vector and the visually generated spectrogram are input into a pre-trained lightweight dual-channel intelligent diagnostic model. The CNN branch automatically learns deep spatial patterns from the spectrogram; the SVM branch quickly classifies the structured feature vector; and the attention mechanism dynamically adjusts the weights of the two branch results based on the current data features to complete the final decision.

[0049] To align visual vibration data with laser displacement signals, this invention employs a dynamic time warping algorithm. The goal is to find the warped path that minimizes the cumulative distance between the two sequences, as follows: S4. Input the generated multidimensional feature vector and vibration spectrum into the lightweight dual-channel intelligent diagnostic model for fault diagnosis, and output the fault type and confidence level. The model completes inference within 40 milliseconds and outputs the diagnostic result: "Fault type: Mechanical loosening; Confidence level: 92%". This result, along with the original vibration spectrum and real-time video footage, is displayed on the edge monitoring screen and triggers an audible and visual alarm to notify maintenance personnel to handle the situation promptly.

[0050] The model employs an attention mechanism to dynamically fuse image features from CNN channels and structured features from SVM channels. The fusion weight αᵢ is calculated as follows:

[0051] ;

[0052] in, Ultimately, the fusion features The mechanism enables the model to focus on the key information most relevant to the fault, improving diagnostic efficiency and accuracy.

[0053] The above embodiments are for illustrative purposes only and are not intended to limit the scope of this invention. Although this invention has been described in detail with reference to the embodiments, those skilled in the art should understand that various combinations, modifications, or equivalent substitutions of the technical solutions of this invention do not depart from the spirit and scope of the technical solutions of this invention and should be covered within the scope of the claims of this invention.

Claims

1. A machine vision and lidar based apparatus vibration measurement device, comprising: The method comprises the following steps: A cooperative sensing system comprising a high-speed camera and a laser range finder, which are synchronously controlled through a network protocol, is used to collect video streams of a target area and provide an absolute displacement reference; A visual vibration inversion module is used to process the video streams collected by the high-speed camera, and based on phase analysis motion amplification and visual vibration inversion, the time sequence changes in the video streams are converted into inversion vibration signals of the equipment, and a vibration frequency spectrum is generated; A multi-modal feature fusion and fault knowledge base module is used to receive the vibration frequency signals and high-frequency displacement data of the laser range finder, align the time sequences and fuse them using a dynamic time warping algorithm, extract envelope features, construct a multi-dimensional feature vector and a feature knowledge base; 2.The device vibration measurement apparatus based on machine vision and laser radar according to claim 1, wherein An edge lightweight intelligent diagnosis module is used to receive the multi-dimensional feature vector and the vibration frequency spectrum generated from the video streams, identify the fault type, and output the diagnosis result and the confidence level. 3.The device vibration measurement apparatus based on machine vision and laser radar according to claim 2, wherein, The laser range finder is an XYZ three-axis laser range finder, and the sampling rate of each axis laser range finder is ≥4KHz; the frame rate of the high-speed camera is ≥1KHz.

4. The machine vision and lidar based apparatus vibration measurement device of claim 3, wherein, The visual vibration inversion module enhances the micro-vibration signal through a sub-pixel level motion amplification algorithm; in the HSV color space, the brightness component is extracted, and the windowed Fourier transform is performed on the time sequence pixel matrix to generate a vibration frequency spectrum. 5.The device vibration measurement apparatus based on machine vision and laser radar according to claim 4, wherein, The multi-modal feature fusion and fault knowledge base module extracts envelope features through Hilbert transform, constructs a multi-dimensional feature vector containing fundamental frequency, harmonic energy ratio, frequency division energy, sideband energy and vibration intensity, and uses principal component analysis for dimension reduction to eliminate redundant information and construct a feature knowledge base containing typical faults. 6.The device vibration measurement apparatus based on machine vision and laser radar according to claim 5, wherein, The edge lightweight intelligent diagnosis module is a dual-channel hybrid network structure, which comprises: a CNN branch for processing the vibration frequency spectrum and extracting spatial features; an SVM branch for processing the structured feature vector; an attention fusion layer for dynamically weighting and fusing the spatial features extracted by the CNN branch and the structured feature vector processed by the SVM branch; and a classifier for outputting the fault type and the confidence level based on the fused features. ； wherein ; f i is the i-th feature from the CNN branch or the SVM branch; are weights computed by an attention network; The fusion weight αᵢ of the attention fusion layer during dynamic weighting fusion is calculated as follows: vᵀ is a weight vector of the hidden layer, which maps the hidden layer state to a final scalar score; 7. A machine vision and lidar based device vibration measurement method, the measurement is based on any of the measurement devices of claims 1-6, characterized in that, b is a bias term. 8.The method of claim 7, wherein, The method comprises the following steps: S1, collecting video streams and laser displacement signals of a target equipment; S2, performing visual vibration inversion processing and windowed Fourier transform on the collected video streams to obtain a vibration frequency spectrum and inversion vibration signals of the equipment; S3, performing time and space alignment and feature fusion on the inversion vibration signals and the laser displacement signals to generate a multi-dimensional feature vector; and S4, inputting the generated multi-dimensional feature vector and the vibration frequency spectrum into a lightweight dual-channel intelligent diagnosis model to perform fault diagnosis and output the fault type and the confidence level. ； wherein, is a local distance measure of sequences Q and C at points i and j, regularized path satisfies the boundary, monotonicity and continuity constraints; In step S3, the dynamic time warping algorithm is used to perform time and space alignment and feature fusion on the inversion vibration signals and the laser displacement signals as follows: Q and C are the visual vibration sequence and the laser displacement sequence, respectively; DTW(Q, C) is the minimum cumulative distance between the two sequences.

Images