A fault analysis method and system based on infrared and vibration detection fusion

Abstract

The embodiment of the application provides a fault analysis method and system based on infrared and vibration detection fusion, the method comprises the following steps: synchronously collecting infrared radiation data, vibration original data and visible light video stream of a target device, generating temperature matrix data and vibration features; obtaining the pixel coordinates of the world coordinates of the vibration sensor in the visible light video stream; generating a vibration nephogram of the vibration area, and superimposing color coding and dynamic vibration rendering onto the visible light video stream; superimposing the temperature matrix data onto the visible light video stream; calculating the multi-dimensional features of the superimposed multi-modal data, including the time domain correlation coefficient of vibration and temperature, the frequency domain-temperature domain coherence function, and the spatial coincidence degree of vibration, temperature and vision; generating a preliminary diagnosis conclusion based on the multi-dimensional features; and comprehensively constructing a diagnosis basis vector based on the multi-dimensional features, and then determining a comprehensive detection result of the target device.

Description

Technical Field

[0001] This invention relates to the field of equipment fault detection technology, and in particular to a fault analysis method and system based on the fusion of infrared and vibration detection. Background Technology

[0002] Diesel engines, steam turbines, and electric motors are rotary power machines that convert the thermal / electrical energy of steam into mechanical energy. They rotate throughout the entire operation process and operate in a relatively harsh environment. They often stop operating due to excessively high temperatures or loosening during rotation. The catastrophic consequence is that the power output fails, causing the entire operating system to shut down.

[0003] Currently, existing detection methods typically employ either infrared thermal imaging or vibration detection as a single method. However, because the core relationship between vibration anomalies and temperature changes is one of "interrelation and mutual causation," a single detection method cannot determine the root cause of the fault. This is because most vibration anomalies are accompanied by a temperature increase in specific areas (vibration leads to friction, increasing energy consumption and ultimately generating heat), and some temperature anomalies (such as uneven thermal expansion) can also trigger vibration (such as component deformation leading to abnormal clearances and ultimately vibration anomalies). Furthermore, the synchronous change of both (such as a sudden rise or fall in vibration amplitude and bearing temperature simultaneously) is crucial for determining the root cause of the fault. Therefore, the detection results are often inaccurate. Summary of the Invention

[0004] To address the problems existing in the prior art, embodiments of the present invention provide a fault analysis method and system based on the fusion of infrared and vibration detection.

[0005] This invention provides a fault analysis method based on the fusion of infrared and vibration detection, the method comprising:

[0006] Simultaneously acquire infrared radiation data, raw vibration data, and visible light video stream of the target device; perform temperature inversion on the infrared radiation data to generate temperature matrix data; and perform time-frequency analysis on the raw vibration data to extract vibration features.

[0007] Set the world coordinates of the vibration sensor and combine them with the visible light video stream to calculate the attitude of the shooting module. Combine the intrinsic parameters of the shooting module and the attitude to obtain the pixel coordinates projected by the world coordinates in the visible light video stream.

[0008] Based on the vibration data of the pixel coordinates, a vibration cloud map of the vibration area is established. The vibration cloud map is color-coded based on the vibration data. Based on the vibration data of the pixel coordinates, dynamic vibration rendering is determined. The color coding and dynamic vibration rendering are superimposed on the visible light video stream.

[0009] The temperature matrix data is image registered with the visible light video stream, the temperature matrix data is superimposed on the visible light video stream, and dynamic temperature rendering is performed on the temperature matrix data.

[0010] The multidimensional features of the superimposed multimodal data are calculated, including the temporal correlation coefficient of vibration and temperature, the coherence function of frequency domain-temperature domain, and the spatial overlap of vibration, temperature and vision. A preliminary diagnostic conclusion is generated based on the multidimensional features.

[0011] Based on the multi-dimensional features, a diagnostic basis vector is constructed, thereby determining the comprehensive detection result of the target device.

[0012] In one embodiment, the method further includes:

[0013] Extract the vibration and temperature characteristics of the target equipment at the same location during the same period, and calculate the rolling correlation coefficient between the two, including:

[0014]

[0015] in, T represents the vibration characteristic, and T represents the temperature characteristic. Due to time lag, , is a vibration characteristic sequence and temperature feature sequence Covariance between Vibration characteristics The standard deviation within the time window. Temperature characteristics The standard deviation within the time window. The range is from -1 to 1;

[0016] based on The numerical output shows the correlation between vibration and temperature at the corresponding part of the target device, and outputs a preliminary diagnostic conclusion based on the correlation.

[0017] In one embodiment, the method further includes:

[0018] Extract the raw acceleration signal corresponding to the original vibration data of the target device at the same location during the same period, and extract the impact-related envelope signal from it, including:

[0019] ,

[0020] in Represents the Hilbert transform. The original acceleration signal, It is the envelope signal;

[0021] The envelope spectrum is obtained by performing a Fourier transform on the envelope signal.

[0022] Acquire the temperature time-series signal of the same location during the same period, calculate the Fourier transform to obtain the temperature fluctuation spectrum, and calculate the coherence function between the temperature fluctuation spectrum and the envelope signal, including:

[0023] ,

[0024] in, For coherence functions, The self-power spectral density of the envelope signal, The auto-power spectral density of the temperature time series signal; It is the cross-power spectral density of the envelope signal and the temperature time series signal;

[0025] By comparing the cross-power spectral density with the preset threshold, as well as the envelope spectrum and temperature fluctuation spectrum with the background noise, a preliminary diagnostic conclusion is output based on the comparison results.

[0026] In one embodiment, the method further includes:

[0027] The vibration hotspots, temperature hotspots, and visually abnormal areas of the target device are extracted during the same period, and the spatial overlap of the three is calculated. Based on the spatial overlap, a preliminary diagnostic conclusion is output.

[0028] In one embodiment, the method further includes:

[0029] By combining multi-dimensional features and the target device's metadata, a standardized diagnostic basis vector is encapsulated, and the diagnostic basis vector is uploaded to the cloud to generate a comprehensive detection result for the target device.

[0030] The cloud database is updated through self-learning based on the diagnostic criteria vector.

[0031] In one embodiment, the method further includes:

[0032] Based on the installation position of the vibration sensor on the target device, the corresponding world coordinates are determined;

[0033] Based on the vibration sensor on the target device, visual algorithm analysis is performed on each frame of the visible light video stream to determine the position of the vibration sensor in the image. The image position is compared with world coordinates to calculate the attitude of the shooting module in reverse. The attitude includes rotation matrix and translation vector.

[0034] In one embodiment, the method further includes:

[0035] A visualization area is constructed around the pixel coordinates, and the propagation and attenuation of vibrations in the device structure are simulated, including:

[0036] ,

[0037] in, The vibration intensity at pixel coordinates. It is the geodesic distance from a point to the pixel reference point at the pixel coordinates. The attenuation coefficient is... The intensity of vibration at distance sensor d;

[0038] Centered on the pixel reference point, according to the distance The vibration region is divided, and the vibration influence intensity of all pixels in the vibration region is normalized to form a vibration cloud map.

[0039] This invention provides a fault analysis system based on the fusion of infrared and vibration detection, the system comprising:

[0040] The acquisition module is used to simultaneously acquire infrared radiation data, raw vibration data, and visible light video stream of the target device, perform temperature inversion on the infrared radiation data to generate temperature matrix data, and perform time-frequency analysis on the raw vibration data to extract vibration features.

[0041] The coordinate module is used to set the world coordinates of the vibration sensor, and calculate the attitude of the shooting module in combination with the visible light video stream. Combined with the intrinsic parameters and attitude of the shooting module, the pixel coordinates projected by the world coordinates in the visible light video stream are obtained.

[0042] The vibration module is used to establish a vibration cloud map of the vibration area based on the vibration data of the pixel coordinates, color-encode the vibration cloud map based on the vibration data, determine dynamic vibration rendering based on the vibration data of the pixel coordinates, and superimpose the color encoding and dynamic vibration rendering onto the visible light video stream.

[0043] The temperature module is used to perform image registration between the temperature matrix data and the visible light video stream, to overlay the temperature matrix data onto the visible light video stream, and to perform dynamic temperature rendering on the temperature matrix data.

[0044] The feature module is used to calculate the multi-dimensional features of the superimposed multimodal data, including the temporal correlation coefficient of vibration and temperature, the coherence function of frequency domain-temperature domain, and the spatial overlap of vibration, temperature and vision. Based on the multi-dimensional features, a preliminary diagnostic conclusion is generated.

[0045] The detection module is used to construct a diagnostic basis vector based on the multi-dimensional features, and then determine the comprehensive detection result of the target device.

[0046] This invention provides an electronic device, including a processor and a memory;

[0047] The processor is connected to the memory;

[0048] The memory is used to store executable program code;

[0049] The processor runs a program corresponding to the executable program code stored in the memory to perform the methods described in one or more embodiments.

[0050] This invention provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described fault analysis method based on the fusion of infrared and vibration detection.

[0051] In view of the above, in one or more embodiments of this specification, infrared radiation data, raw vibration data, and visible light video stream of the target device are simultaneously acquired; temperature inversion is performed on the infrared radiation data to generate temperature matrix data; time-frequency analysis is performed on the raw vibration data to extract vibration features; world coordinates of the vibration sensor are set, and the attitude of the shooting module is calculated in combination with the visible light video stream; pixel coordinates projected onto the visible light video stream are obtained by combining the intrinsic parameters and attitude of the shooting module; based on the vibration data of the pixel coordinates, a vibration cloud map of the vibration area is established; the vibration cloud map is color-coded based on the vibration data, and based on... Vibration data at pixel coordinates is used to determine dynamic vibration rendering. Color encoding and dynamic vibration rendering are superimposed onto the visible light video stream. Temperature matrix data is image registered with the visible light video stream, superimposed onto the visible light video stream, and dynamic temperature rendering is performed on the temperature matrix data. Multi-dimensional features of the superimposed multimodal data are calculated, including the temporal correlation coefficient between vibration and temperature, the frequency-temperature domain coherence function, and the spatial overlap of vibration, temperature, and vision. Preliminary diagnostic conclusions are generated based on these multi-dimensional features. A diagnostic basis vector is constructed based on the comprehensive multi-dimensional features to determine the comprehensive detection result of the target device. This breaks down the data silos of vibration, temperature, and vision in traditional fault analysis, achieving precise data fusion at the pixel level. Furthermore, based on data correlation and model analysis, the causal relationship of data components is determined, thereby analyzing the device's detection results and avoiding the problem of inaccurate cause analysis and multiple repairs caused by single-angle analysis. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 This is a flowchart of a fault analysis method based on the fusion of infrared and vibration detection, provided in one embodiment of this specification.

[0054] Figure 2 This is a schematic diagram of a fault analysis system based on the fusion of infrared and vibration detection, provided in one embodiment of this specification.

[0055] Figure 3 This is a schematic diagram of the structure of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0056] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed merely to enable those skilled in the art to better understand and implement the subject matter described herein, and are not intended to limit the scope, applicability, or examples set forth in the claims. The function and arrangement of the elements discussed may be changed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the various examples. For example, the described methods may be performed in a different order than described, and steps may be added, omitted, or combined. Furthermore, features described in some examples may be combined in other examples.

[0057] As used herein, the term "comprising" and its variations are open terms meaning "including but not limited to". The term "based on" means "at least partially based on". The terms "one embodiment" and "an embodiment" mean "at least one embodiment". The term "another embodiment" means "at least one other embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other definitions, whether explicit or implicit, may be included below. Unless explicitly indicated by the context, the definition of a term shall remain consistent throughout the specification.

[0058] like Figure 1 As shown, this embodiment of the invention provides a fault analysis method based on the fusion of infrared and vibration detection, including:

[0059] Step S102: Simultaneously collect infrared radiation data, raw vibration data, and visible light video stream of the target device; perform temperature inversion on the infrared radiation data to generate temperature matrix data; and perform time-frequency analysis on the raw vibration data to extract vibration features.

[0060] Specifically, using an infrared thermal imaging detector and corresponding lens capable of covering the target device, continuous raw infrared radiation data of the target device (such as a diesel engine or electric motor) is acquired. During data acquisition, it is crucial to ensure that each frame of raw data is accompanied by its precise time. The acquired infrared radiation data is then processed using AGC, noise reduction, DDE, and pseudo-color processing to generate a data stream directly playable on a display. The ambient temperature (as well as parameters such as relative humidity, measurement distance, atmospheric transmittance, and target emissivity) is acquired, and combined with a temperature inversion algorithm, the raw infrared radiation data of each pixel is calculated into its corresponding absolute temperature value. A coordinate mapping relationship is then established to ensure a strict one-to-one correspondence between the processed temperature matrix data (each pixel representing a temperature value) and the original data frames and the pseudo-color video frames played on the display, both in time and space.

[0061] For vibration data, raw vibration data (raw acceleration signals) are collected at key diagnostic locations (such as the horizontal and vertical directions of the bearing housing, the drive end and non-drive end of the motor) at a specific sampling frequency using vibration sensors (such as IEPE type accelerometers). The collected raw (acceleration) time-domain waveform data also has a corresponding precise timestamp. The raw vibration data is then preprocessed, including operations such as DC component removal, anti-aliasing filtering, and digital filtering. Time-domain integration, frequency-domain analysis, and feature calculations are then performed to obtain vibration characteristics, including the time-domain waveforms and spectrum of vibration frequency, velocity, and displacement.

[0062] In addition, when collecting temperature and vibration data, and using a visible light camera to acquire a visible light video stream, and when recording the video stream of the target device through a camera, it is necessary to ensure that the clock sources of the visible light camera, infrared thermal imager, and vibration data acquisition card are highly synchronized, so as to achieve time stamp synchronization of the data of the three.

[0063] Step S104: Set the world coordinates of the vibration sensor, and calculate the attitude of the shooting module in combination with the visible light video stream. Combine the intrinsic parameters of the shooting module and the attitude to obtain the pixel coordinates projected by the world coordinates in the visible light video stream.

[0064] Specifically, the installation positions of the vibration sensors deployed on the target device are obtained, and the world coordinates of the corresponding positions in the world coordinate system (the virtual three-dimensional space of the target device itself) are recorded. Then, each frame of the visible light video stream is analyzed in real time, including pre-determining high-contrast visual markers (such as sensor 1) on the surface of the target device, determining the position and world coordinates of the visual markers in the image using computer vision algorithms, and calculating the pose of the current shooting module (the pose includes rotation matrix and translation vector, defining the transformation relationship from the world coordinate system to the camera viewpoint).

[0065] Furthermore, assuming that "vibration sensor #1" has already been labeled, its world coordinates, along with the attitude and intrinsic parameters of the imaging module (including the camera's own intrinsic parameter matrix K (focal length, distortion, etc.)), can be used to calculate the sensor's pixel position in the current image using a projection formula. This formula can include:

[0066] ,

[0067] Where (u, v) are pixel coordinates, K is the camera intrinsic parameter matrix, R is the rotation matrix, and T is the translation vector. The world coordinates of the sensor.

[0068] Step S106: Based on the vibration data of the pixel coordinates, establish a vibration cloud map of the vibration area, color-encode the vibration cloud map based on the vibration data, determine dynamic vibration rendering based on the vibration data of the pixel coordinates, and superimpose the color encoding and dynamic vibration rendering onto the visible light video stream.

[0069] Specifically, the area affected by vibration on the target device is determined based on the pixel location of the vibration, which requires circumferential positioning around the sensor's reference pixel. Construct a visualization area. Simulate the propagation and attenuation of vibrations within the equipment structure, and the vibration intensity... With distance It decreases as the value increases. The formula is as follows:

[0070] ,

[0071] in, It is the vibration intensity at the sensor ( ), It is the distance from a point in the image to the reference point. The geodetic distance (on the surface model of the equipment), It is the attenuation coefficient, which is related to the equipment materials and structure, and can be determined through experiments or simulations. The vibration intensity at distance sensor d is the distance sensor d.

[0072] In pixels Centered on, based on distance Divide the area into corresponding regions. Within each region, calculate the value for each pixel. This refers to the "degree of impact". It involves all pixels... The values ​​are normalized to form a vibration cloud map. The higher the value, the greater the intensity of the vibration.

[0073] Additionally, vibration feature values ​​can be mapped to specific visual attributes. For example, vibration data based on pixel coordinates can be color-coded. Mapping rules can be pre-set to map the vibration values ​​(such as effective velocity values) currently measured by the sensor to corresponding colors. Examples include: Normal - blue, Caution - green, Warning - yellow, Danger - red, etc., followed by color coding.

[0074] Furthermore, dynamic vibration rendering can be used to map sensor coordinates. The vibration characteristic values ​​are plotted as dynamic visual motion. That is, to intuitively display the vibration frequency, at a reference point... Draw a halo around the area, with a radius of... It changes periodically with time t:

[0075] ,

[0076] in, It is the basic radius; It is the dominant vibrational frequency (extracted from the FFT spectrum); It is a scaling factor, related to the displacement amplitude. The vibrations are directly proportional. The higher the frequency f, the faster the halo pulsates; the greater the vibration amplitude, the greater the amplitude of the halo pulsation. This dynamically visualizes the key but abstract information of frequency. A rapidly flashing large halo intuitively represents "high-frequency, intense vibration"; a slowly pulsating small halo represents "low-frequency, micro-vibration." Color encoding and dynamic vibration rendering are then overlaid onto video frames of the visible light video stream. For example, the bottom layer is the original visible light video, the middle layer is a semi-transparent vibration cloud map, and the top layer is a sensor marker with color encoding and a dynamic halo. This allows the system to directly display the following information: which part is vibrating (location), how severe the vibration is (color), how fast the vibration is (halo pulsation frequency), and how large an area the vibration affects (semi-transparent cloud map), providing users with a clear and concise understanding of the vibration situation.

[0077] Step S108: Perform image registration between the temperature matrix data and the visible light video stream, overlay the temperature matrix data onto the visible light video stream, and perform dynamic temperature rendering on the temperature matrix data.

[0078] Specifically, the temperature matrix data is image registered with the visible light video stream, and each frame of synchronously acquired infrared and visible light images is processed. This includes perspective transformation (based on the relative position and orientation between the two sets of images), interpolation and magnification (the infrared image has low resolution, so it is deformed and "filled" to fit the size of the visible light image), and cropping and alignment. Then, a pixel-level relationship is established between the two, assigning a temperature attribute to each pixel in the visible light image. For the temperature attribute, visualization rendering can also be performed, i.e., determining the temperature mapping rules, such as low temperature - blue, medium temperature - green, high temperature - red, etc. The temperature rendering result is then overlaid on the visible light image. Isotherms can also be drawn to identify temperature gradients. The final information synthesis image layer order can be: bottom layer, visible light video; middle layer 1, semi-transparent temperature heatmap; middle layer 2, vibration influence area cloud map; top layer, vibration sensor identifier, dynamic halo, etc. A mapping relationship of "pixel → temperature → vibration intensity" is established.

[0079] Step S110: Calculate the multi-dimensional features of the superimposed multimodal data, including the temporal correlation coefficient of vibration and temperature, the coherence function of frequency domain-temperature domain, and the spatial overlap of vibration, temperature, and vision. Generate a preliminary diagnostic conclusion based on the multi-dimensional features.

[0080] Specifically, the multi-dimensional features of the superimposed multimodal data (temperature, vibration, and visual visible light timestamp alignment) are calculated. These include:

[0081] 1. Time-domain correlation coefficient between vibration and temperature. This involves analyzing how vibration and temperature change over time, and the temporal relationship between these changes. It focuses on the trends of physical quantities over time and their interrelationships to assess the rate of fault evolution and provide early warning. The steps include:

[0082] Extract vibration and temperature data of the target equipment components within the same time period. Simultaneously plot its vibration characteristics (e.g., ...). The curves showing the change of temperature characteristics (e.g., T) over time. Calculate the rolling correlation coefficient between the two. .

[0083] ,

[0084] in It is a time delay, meaning that the temperature signal is considered when calculating the correlation. Relative to vibration signal The amount of time that is brought forward or delayed on the timeline. , indicating vibration sequence Columns and temperature series The covariance between them. Indicating the effective value sequence of vibration velocity The standard deviations listed within the analysis time window. Representing temperature sequence The standard deviation within the analysis time window The range is from -1 to 1, and the larger the absolute value, the stronger the correlation.

[0085] based on The numerical values ​​determine the relationship between the two. exist A strong positive correlation is observed when the value is 0, indicating that vibration and temperature rise intensify synchronously, a typical sign of mechanical friction (such as bearing wear, dry friction). If exist A value greater than 0 indicates a strong positive correlation, suggesting that temperature changes lag behind vibrations, possibly due to thermal inertia. If... exist A value less than 0 indicates a strong positive correlation, suggesting that temperature changes precede vibration, which may be caused by thermal deformation.

[0086] 2. Frequency-Temperature Domain Coherence Function. Establishing a quantitative correlation between high-frequency mechanical vibration and low-frequency temperature changes, thereby binding the characteristic frequencies in the vibration spectrum with thermal effects, is an important means of diagnosing periodic faults in rotating machinery (such as pitting of bearings and gears).

[0087] Because the original vibration signal contains many frequency components, a Hilbert transform is used to extract the periodic impact from the original acceleration signal. Extract the impact-related envelope signal :

[0088] ,

[0089] in This represents the Hilbert transform. The original acceleration signal and the envelope signal. It includes information on the amplitude and repetition frequency of the impact, while filtering out high-frequency carrier waves (such as gear meshing frequencies).

[0090] For envelope signal Perform an FFT Fourier transform to obtain the envelope spectrum. :

[0091] ,

[0092] In envelope spectrum In the context of bearing or gear failure characteristic frequencies (like It will appear as clear spectral lines.

[0093] Then, the periodic fluctuation components in the temperature signal are analyzed, and the temperature time series signal is analyzed. Perform an FFT to obtain the temperature fluctuation spectrum. :

[0094] ,

[0095] Because the time constant of thermal inertia is much larger than the oscillation period, The energy is mainly concentrated in the low-frequency range.

[0096] To quantitatively demonstrate temperature fluctuations It is caused by vibration and impact Caused by this, calculate the coherence function of the two in the frequency domain. It indicates that at frequency Above, the degree of linear correlation between the two signals:

[0097] ,

[0098] in, It is an envelope signal The self-power spectral density; It is a temperature timing signal The self-power spectral density; yes and The cross-power spectral density.

[0099] if >Threshold (e.g., 0.8), and and If the amplitudes of the vibrations are significantly higher than the background noise, the system can provide a high-confidence diagnostic conclusion, such as "A periodic thermal modulation synchronized with the vibration and shock frequency was detected, confirming the existence and persistence of the fault, which is a bearing outer ring fault."

[0100] 3. Spatial overlap of vibration, temperature and vision: Utilizing the advantage of "complete alignment of vibration, temperature and visual images", the spatial relationship of anomalies can be analyzed on a two-dimensional plane, thereby accurately locating the source of the fault.

[0101] First, the system automatically detects and labels: Vibration hotspots: Vibration energy ( For example, a point set significantly higher than the surrounding area. Temperature hotspots: Sets of points whose temperature is significantly higher than the surrounding area. Visually anomalous regions: Sets of points discovered through image recognition (such as edge detection and texture analysis). (such as cracks, rust, oil stains).

[0102] Then calculate the spatial overlap between these point sets:

[0103] ,

[0104] in , , Only when all three are highly overlapping, If the value approaches 1, it indicates that the fault source is located in the overlapping area. Additionally, if there is only high vibration with normal temperature and appearance, it may indicate early loosening. If there is high vibration and high temperature but a normal appearance, it may indicate internal wear. In summary, by conducting in-depth analysis from three different perspectives (time domain, frequency domain, and spatial domain), preliminary diagnostic conclusions are generated from different angles to identify the root cause of the fault.

[0105] Step S112: Construct a diagnostic basis vector based on the multi-dimensional features, and then determine the detection result of the target device.

[0106] Furthermore, the complete analytical basis for the preliminary diagnostic conclusions derived from the above steps is encapsulated into a standardized diagnostic basis vector. This vector includes the time-domain correlation coefficient, the time delay corresponding to the maximum correlation coefficient, the coherence function value at the fault characteristic frequency, multimodal overlap, spatial pattern matching score, and overall confidence level. Additionally, the encapsulated diagnostic basis vector may include metadata such as: equipment model, serial number, operating hours, current operating conditions (load, speed, ambient temperature), material properties (thermal conductivity, coefficient of thermal expansion), and maintenance history. This is then combined with shared experience data in the cloud. This is because different targets and materials have varying impacts on vibration and temperature; therefore, the judgment criteria need to be uploaded to the cloud for integration with past experience data for assessment, and the final diagnostic result is sent to the device. If the device lacks remote transmission capabilities, the final result is determined based on built-in experience data. Furthermore, the algorithm analysis library will automatically learn and update the judgment factors, allowing the cloud database to self-learn and be used for subsequent fault analysis and diagnosis.

[0107] This invention provides a fault analysis method based on the fusion of infrared and vibration detection. It simultaneously acquires infrared radiation data, raw vibration data, and visible light video stream from the target device. Temperature inversion is performed on the infrared radiation data to generate a temperature matrix. Time-frequency analysis is performed on the raw vibration data to extract vibration features. World coordinates of the vibration sensor are set, and the attitude of the imaging module is calculated in conjunction with the visible light video stream. Based on the intrinsic parameters and attitude of the imaging module, the pixel coordinates projected onto the visible light video stream are obtained. A vibration cloud map of the vibration region is established based on the vibration data using these pixel coordinates. The vibration cloud map is then color-coded based on the vibration data. Based on vibration data using pixel coordinates, dynamic vibration rendering is determined, and color encoding and dynamic vibration rendering are superimposed onto the visible light video stream. Temperature matrix data is image registered with the visible light video stream, superimposed onto the visible light video stream, and dynamic temperature rendering is applied to the temperature matrix data. Multi-dimensional features of the superimposed multimodal data are calculated, including the temporal correlation coefficient between vibration and temperature, the frequency-temperature domain coherence function, and the spatial overlap of vibration, temperature, and visual data. Preliminary diagnostic conclusions are generated based on these multi-dimensional features. A diagnostic basis vector is then constructed based on these multi-dimensional features to determine the comprehensive detection result of the target device. This approach breaks down the data silos of vibration, temperature, and visual data in traditional fault analysis, achieving precise data fusion at the pixel level. Furthermore, based on data correlation and model analysis, the causal relationships between data components are determined, leading to analysis of the device's detection results. This avoids the problem of inaccurate cause analysis and multiple repairs due to single-angle analysis.

[0108] Please see Figure 2 , Figure 2 This is a schematic diagram of a fault analysis system based on the fusion of infrared and vibration detection provided in an embodiment of this application. Figure 2 As shown, the system includes:

[0109] The acquisition module S202 is used to simultaneously acquire infrared radiation data, raw vibration data and visible light video stream of the target device, perform temperature inversion on the infrared radiation data to generate temperature matrix data, and perform time-frequency analysis on the raw vibration data to extract vibration features.

[0110] The coordinate module S204 is used to set the world coordinates of the vibration sensor, and calculate the attitude of the shooting module in combination with the visible light video stream. Combined with the intrinsic parameters and attitude of the shooting module, the pixel coordinates projected by the world coordinates in the visible light video stream are obtained.

[0111] The vibration module S206 is used to establish a vibration cloud map of the vibration area based on the vibration data of the pixel coordinates, color-encode the vibration cloud map based on the vibration data, determine dynamic vibration rendering based on the vibration data of the pixel coordinates, and superimpose the color encoding and dynamic vibration rendering onto the visible light video stream.

[0112] Temperature module S208 is used to perform image registration between the temperature matrix data and the visible light video stream, superimpose the temperature matrix data onto the visible light video stream, and perform dynamic temperature rendering on the temperature matrix data.

[0113] The feature module S210 is used to calculate the multi-dimensional features of the superimposed multimodal data, including the temporal correlation coefficient of vibration and temperature, the coherence function of frequency domain-temperature domain, and the spatial overlap of vibration, temperature and vision, and to generate a preliminary diagnostic conclusion based on the multi-dimensional features.

[0114] The detection module S212 is used to construct a diagnostic basis vector based on the multi-dimensional features, and then determine the comprehensive detection result of the target device.

[0115] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0116] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0117] See Figure 3 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 3 As shown, the electronic device 300 may include: at least one processor 301, at least one network interface 304, user interface 303, memory 305, and at least one communication bus 302.

[0118] The communication bus 302 is used to enable communication between these components.

[0119] The user interface 303 may include a display screen and a camera. Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0120] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0121] The processor 301 may include one or more processing cores. The processor 301 connects to various parts within the electronic device 300 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, and by calling data stored in the memory 305. Optionally, the processor 301 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 301 may integrate one or a combination of several of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 301 and may be implemented as a separate chip.

[0122] The memory 305 may include random access memory (RAM) or read-only memory. Optionally, the memory 305 may include a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 305 may also be at least one storage device located remotely from the aforementioned processor 301. Figure 3 As shown, the memory 305, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0123] exist Figure 3 In the illustrated electronic device 300, the user interface 303 is mainly used to provide an input interface for the user and acquire user input data; while the processor 301 can be used to call the image-based interactive application stored in the memory 305, and specifically perform the following operations: synchronously acquire infrared radiation data, raw vibration data, and visible light video stream of the target device; perform temperature inversion on the infrared radiation data to generate temperature matrix data; perform time-frequency analysis on the raw vibration data to extract vibration features; set the world coordinates of the vibration sensor, and calculate the attitude of the shooting module in combination with the visible light video stream; combine the intrinsic parameters of the shooting module and the attitude to obtain the pixel coordinates projected onto the visible light video stream; based on the vibration data of the pixel coordinates, construct... A vibration cloud map of the vibration area is established. The vibration cloud map is color-coded based on the vibration data, and dynamic vibration rendering is determined based on the vibration data at pixel coordinates. The color coding and dynamic vibration rendering are superimposed on the visible light video stream. The temperature matrix data is image registered with the visible light video stream, and the temperature matrix data is superimposed on the visible light video stream. Dynamic temperature rendering is performed on the temperature matrix data. The multi-dimensional features of the superimposed multimodal data are calculated, including the temporal correlation coefficient between vibration and temperature, the coherence function of the frequency domain and temperature domain, and the spatial overlap of vibration, temperature, and vision. A preliminary diagnostic conclusion is generated based on the multi-dimensional features. A diagnostic basis vector is constructed based on the multi-dimensional features to determine the comprehensive detection result of the target device.

[0124] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0125] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0126] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0131] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0132] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

Claims

1. A fault analysis method based on the fusion of infrared and vibration detection, the method comprising: Simultaneously acquire infrared radiation data, raw vibration data, and visible light video stream of the target device; perform temperature inversion on the infrared radiation data to generate temperature matrix data; and perform time-frequency analysis on the raw vibration data to extract vibration features. Set the world coordinates of the vibration sensor and combine them with the visible light video stream to calculate the attitude of the shooting module. Combine the intrinsic parameters of the shooting module and the attitude to obtain the pixel coordinates projected by the world coordinates in the visible light video stream. Based on the vibration data of the pixel coordinates, a vibration cloud map of the vibration area is established. The vibration cloud map is color-coded based on the vibration data. Based on the vibration data of the pixel coordinates, dynamic vibration rendering is determined. The color coding and dynamic vibration rendering are superimposed on the visible light video stream. The temperature matrix data is image registered with the visible light video stream, the temperature matrix data is superimposed on the visible light video stream, and dynamic temperature rendering is performed on the temperature matrix data. The multidimensional features of the superimposed multimodal data are calculated, including the temporal correlation coefficient of vibration and temperature, the coherence function of frequency domain-temperature domain, and the spatial overlap of vibration, temperature and vision. A preliminary diagnostic conclusion is generated based on the multidimensional features. Based on the multi-dimensional features, a diagnostic basis vector is constructed, thereby determining the comprehensive detection result of the target device.

2. The method according to claim 1, characterized in that, The method further includes: Extract the vibration and temperature characteristics of the target equipment at the same location during the same period, and calculate the rolling correlation coefficient between the two, including: ， in, T represents the vibration characteristic, and T represents the temperature characteristic. Due to time lag, , is a vibration characteristic sequence and temperature feature sequence Covariance between Vibration characteristics The standard deviation within the time window. Temperature characteristics The standard deviation within the time window. The range is from -1 to 1; based on The numerical output shows the correlation between vibration and temperature at the corresponding part of the target device, and outputs a preliminary diagnostic conclusion based on the correlation.

3. The method according to claim 1, characterized in that, The method further includes: Extract the raw acceleration signal corresponding to the original vibration data of the target device at the same location during the same period, and extract the impact-related envelope signal from it, including: ， in Represents the Hilbert transform. The original acceleration signal, It is the envelope signal; The envelope spectrum is obtained by performing a Fourier transform on the envelope signal. Acquire the temperature time-series signal of the same location during the same period, calculate the Fourier transform to obtain the temperature fluctuation spectrum, and calculate the coherence function between the temperature fluctuation spectrum and the envelope signal, including: ， in, For coherence functions, The self-power spectral density of the envelope signal, The auto-power spectral density of the temperature time series signal; It is the cross-power spectral density of the envelope signal and the temperature time series signal; By comparing the cross-power spectral density with the preset threshold, as well as the envelope spectrum and temperature fluctuation spectrum with the background noise, a preliminary diagnostic conclusion is output based on the comparison results.

4. The method according to claim 1, characterized in that, The method further includes: The vibration hotspots, temperature hotspots, and visually abnormal areas of the target device are extracted during the same period, and the spatial overlap of the three is calculated. Based on the spatial overlap, a preliminary diagnostic conclusion is output.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: By combining multi-dimensional features and the target device's metadata, a standardized diagnostic basis vector is encapsulated, and the diagnostic basis vector is uploaded to the cloud to generate a comprehensive detection result for the target device. The cloud database is updated through self-learning based on the diagnostic criteria vector.

6. The method according to claim 1, characterized in that, The process of setting the world coordinates of the vibration sensor and calculating the attitude of the shooting module in conjunction with the visible light video stream includes: Based on the installation position of the vibration sensor on the target device, the corresponding world coordinates are determined; Based on the vibration sensor on the target device, visual algorithm analysis is performed on each frame of the visible light video stream to determine the position of the vibration sensor in the image. The image position is compared with world coordinates to calculate the attitude of the shooting module in reverse. The attitude includes rotation matrix and translation vector.

7. The method according to claim 1, characterized in that, The process of establishing a vibration cloud map of the vibration region based on the vibration data of the pixel coordinates includes: A visualization area is constructed around the pixel coordinates, and the propagation and attenuation of vibrations in the device structure are simulated, including: ， in, The vibration intensity at pixel coordinates. It is the geodesic distance from a point to the pixel reference point at the pixel coordinates. The attenuation coefficient is... The intensity of vibration at distance sensor d; Centered on the pixel reference point, according to the distance The vibration region is divided, and the vibration influence intensity of all pixels in the vibration region is normalized to form a vibration cloud map.

8. A fault analysis system based on the fusion of infrared and vibration detection, characterized in that, The system includes; The acquisition module is used to simultaneously acquire infrared radiation data, raw vibration data, and visible light video stream of the target device, perform temperature inversion on the infrared radiation data to generate temperature matrix data, and perform time-frequency analysis on the raw vibration data to extract vibration features. The coordinate module is used to set the world coordinates of the vibration sensor, and calculate the attitude of the shooting module in combination with the visible light video stream. Combined with the intrinsic parameters and attitude of the shooting module, the pixel coordinates projected by the world coordinates in the visible light video stream are obtained. The vibration module is used to establish a vibration cloud map of the vibration area based on the vibration data of the pixel coordinates, color-encode the vibration cloud map based on the vibration data, determine dynamic vibration rendering based on the vibration data of the pixel coordinates, and superimpose the color encoding and dynamic vibration rendering onto the visible light video stream. The temperature module is used to perform image registration between the temperature matrix data and the visible light video stream, to overlay the temperature matrix data onto the visible light video stream, and to perform dynamic temperature rendering on the temperature matrix data. The feature module is used to calculate the multi-dimensional features of the superimposed multimodal data, including the temporal correlation coefficient of vibration and temperature, the coherence function of frequency domain-temperature domain, and the spatial overlap of vibration, temperature and vision. Based on the multi-dimensional features, a preliminary diagnostic conclusion is generated. The detection module is used to construct a diagnostic basis vector based on the multi-dimensional features, and then determine the comprehensive detection result of the target device.

9. An electronic device, comprising a processor and a memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-7.

Images