Rotating machinery fault signal separation and diagnosis method based on audio-visual information fusion

Through the method of audio-visual information fusion, combined with non-negative matrix decomposition and generalized cross-correlation algorithm, the noise interference and equipment invasiveness problems in the signal positioning and separation of rotary mechanical faults are solved, and the accurate positioning and diagnosis of rotary mechanical faults is achieved, which reduces the Cost and calculation overhead.

CN118939980APending Publication Date: 2024-11-12NORTHEASTERN UNIV CHINA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411320774.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art has problems with noise interference and equipment in the positioning and separation of rotating mechanical fault signals, especially in a multi-sound source environment, and traditional methods are expensive and have high calculation overhead.

Method used

Using a method based on audio-visual information fusion, combining non-negative matrix decomposition and generalized cross-correlation algorithm, signals are collected through a dual-channel microphone array and camera-assisted sound source positioning, the sound source angle and delay are calculated, and the mask matrix is ​​generated for signal separation. Fault characteristic frequencies were extracted by wavelet transform and Hilbert envelope spectrum analysis.

Benefits of technology

It realizes accurate positioning of fault sources in a multi-sound source environment, reduces equipment costs and calculation overhead, and improves the separation effect and diagnostic accuracy of fault signals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939980A_ABST
    Figure CN118939980A_ABST
Patent Text Reader

Abstract

According to the rotating machine fault signal separation and diagnosis method based on audio-visual information fusion, a sound source localization method based on visual information assistance is combined with non-negative matrix factorization to realize localization of a rotating machine fault signal source, and separation and feature extraction of fault signals. According to the sound source localization method based on visual information assistance, the defects of a traditional GCC sound source localization algorithm can be overcome under the condition that the angle difference between multiple sound sources is small, the angle of a target fault source is estimated more accurately, the time difference of source signals reaching different array elements is obtained, and therefore accurate localization of the fault source is achieved. Experiments show that the signal separation and diagnosis method provided by the invention has a good separation effect on fault signals, and can make an accurate judgment on fault types.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of rotating machinery fault signal diagnosis, and relates to a rotating machinery fault signal separation and diagnosis method based on audio-visual information fusion. Background Art

[0002] With the advancement of science and technology, the trend of large-scale, precision and systematization of mechanical equipment is becoming more and more obvious, and the integration of rotating parts such as engines, gears and bearings is also increasing exponentially. The coordinated work of a large number of rotating parts ensures a high level of production efficiency, but it also increases the probability of equipment failure. These parts are very easy to wear and deform due to various factors. If not solved in time, multi-point failures will have a huge impact on the work of the assembly line, thereby reducing production efficiency, causing economic losses, and even threatening the lives of personnel. Therefore, fault monitoring of machinery and equipment is particularly important. Accurate and rapid fault diagnosis can minimize property losses and safety risks.

[0003] Most traditional fault monitoring methods collect, process and analyze vibration signals of mechanical equipment. In order to collect the vibration signals of components, the sensor needs to be installed on a plane close to and in contact with the target component, which is invasive. This often requires physical modifications such as disassembly and installation of mechanical equipment, which may cause secondary damage to the components. At present, mechanical fault monitoring methods based on acoustic signal analysis use microphones to collect acoustic characteristics of mechanical equipment during operation to achieve local or global diagnosis of faults. Compared with traditional methods based on vibration signals, methods based on acoustic signals can achieve non-contact, non-invasive and non-invasive measurements without disassembling the equipment, fundamentally solving the problem that sensors are difficult to fix on the surface of the equipment for various reasons, which also ensures that the installation and monitoring process will not affect the normal operation of the equipment. In addition, for industrial production applications, this type of method has better adaptability to the environment and is also convenient for signal acquisition and fault monitoring under harsh conditions such as high temperature, high pressure, and radiation.

[0004] The working environment of mechanical equipment is always full of various sounds, some of which are noises unrelated to the operation of the equipment, and the sound signals generated by a piece of equipment when it is running are often mixed from the sound signals emitted by multiple components when they are working. Therefore, in order to locate and analyze the fault, it is necessary to separate the source signal of interest from the background noise and other interference source signals. Blind source separation is one of the important technologies to solve the above problems. Its mathematical model originated from the "cocktail party" effect, that is, separating the sound signal of interest from the mixed signal of multiple sound sources in a noisy environment. "Blind" is mainly reflected in two aspects: the mathematical model of the source signal cannot be predicted, and the channel parameters of the signal transmission are also unknown.

[0005] Non-negative Matrix Factorization (NMF) is a commonly used algorithm in the field of blind source separation. It decomposes the initial matrix into a set of non-negative dictionary matrices and coefficient matrices, avoiding the problem of positive and negative cancellation in the feature extraction process, and providing a non-destructive representation of the spectrogram of mixed acoustic signals in industrial environments. Under certain conditions, the decomposition result of NMF is unique and does not require other orthogonality or independence assumptions. In order to mine the rich spatiotemporal structure and nonlinear relationships in audio, some studies have combined NMF with neural networks to expand it into a deep structure, which has achieved a huge improvement in performance. Zhang et al. used NMF in the field of fault monitoring and proposed a fault monitoring method that combines structured joint sparse orthogonal NMF (SJSONMF) with multivariate analysis (MVA). The regularization, sparsity and orthogonality constraints of the graph are introduced into the classic NMF, which has stronger discrimination ability, reduces the redundancy from noise and nonlinear data, and improves the interpretability of faults. Although the improved methods based on deep learning have better separation performance than traditional methods, they are too dependent on training data. For actual application scenarios that are very different from the training data collection environment, the separation effect may be difficult to predict. For mixed signals of complex audio, traditional NMF can encode the original signal across multiple dictionary atoms, and then the dictionary atoms need to be grouped to achieve separation. Most methods require manual supervision based on prior knowledge or information in this process, such as spatial positioning information.

[0006] Sound source localization is the main technology for providing spatial information. Inspired by the working principle of human binaural hearing, some methods use dual-channel signal processing technology, while other methods use arrays with more microphones (more than two) for audio collection. Dual-channel signal processing technology is mainly used for audio signal separation, but it is less used in the field of fault monitoring. Microphone arrays are mainly used for acoustic imaging. By using beamforming technology, the beam shape of the array is guided to different spatial positions, thereby achieving spatial filtering, that is, distinguishing the sound source according to the location of the sound source. At present, microphone arrays and beamforming technology have been widely used in mechanical fault monitoring, and the number and layout of array elements are adjusted according to actual needs. However, array-based signal processing methods will incur large equipment and computing costs, and the exploration of binaural methods has become an important research direction. Wood, Sean UN and others proposed a dual-channel generalized cross-correlation (GCC) sound source localization algorithm. However, when the angle difference of multiple sound sources is small, the time delays of source signals at different positions relative to different channels are small and close. In the delay curve generated by the GCC algorithm, the real spectral peaks may merge and be hidden among many pseudo peaks, making it impossible to accurately locate the sound source. Summary of the invention

[0007] In order to solve the above technical problems, the purpose of the present invention is to provide a method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion.

[0008] A method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion of the present invention comprises:

[0009] Step 1: Collect mixed audio signals when the rotating machinery is working, obtain dual-channel mixed audio signals, and collect sound source images;

[0010] Step 2: extracting the time-frequency information of the dual-channel mixed audio signal to obtain the amplitude spectrum and phase spectrum of the dual-channel mixed audio signal;

[0011] Step 3: Perform non-negative matrix decomposition on the amplitude spectrum of the dual-channel mixed audio signal to obtain a dictionary matrix and a coefficient matrix;

[0012] Step 4: Use the dictionary matrix to update the frequency weighting function in the generalized cross-correlation algorithm to obtain the GCC-NMF joint function;

[0013] Step 5: Calculate the sound source angle based on the collected sound source image, and calculate the time delay of the acoustic signals from different sound sources reaching the two array elements based on the geometric position relationship between the array elements and the sound source;

[0014] Step 6: Obtain the mask matrix of the sound source based on the GCC-NMF joint function and time delay;

[0015] Step 7: Multiply the corresponding elements of the mask matrix and the coefficient matrix to obtain a new coefficient matrix, and then multiply the dictionary matrix by the new coefficient matrix to complete the inverse process of non-negative matrix decomposition to obtain the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation, and then obtain the time domain audio signal belonging to each sound source after separation through ISTFT;

[0016] Step 8: Use wavelet transform to perform multi-resolution decomposition and reconstruction on the separated time domain audio signals belonging to each sound source, and perform Hilbert envelope spectrum analysis on the first layer of detail signals to extract the fault characteristic frequency and realize fault diagnosis.

[0017] The present invention provides a method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion, which has the following beneficial effects:

[0018] (1) The present invention proposes a sound source localization method based on visual information assistance, which can overcome the shortcomings of the traditional GCC sound source localization algorithm when the angle difference between multiple sound sources is small, and more accurately estimate the angle of the target fault source, and obtain the time difference between the source signal reaching different array elements, thereby realizing the location of the fault source.

[0019] (2) The sound source localization method assisted by visual information is combined with non-negative matrix decomposition, and a signal separation and diagnosis method based on audio-visual information fusion is proposed to realize the localization of the fault signal source of rotating machinery, the separation of fault signals and feature extraction. Experiments show that this method has a good effect on the separation of fault signals and can make an accurate judgment on the fault type.

[0020] (3) Using acoustic signals instead of vibration signals can achieve a non-invasive signal acquisition process, fundamentally solving the problem that the sensor is difficult to fix on the surface of the equipment due to various reasons.

[0021] (4) The dual-channel signal processing method can reduce the cost of sampling equipment and save computing overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of a method for separating and diagnosing fault signals of rotating machinery based on audio-visual information fusion of the present invention;

[0023] Figure 2 It is a schematic diagram of a bird's-eye view above the camera in a sound source localization method based on visual information assistance;

[0024] Figure 3 It is a schematic diagram of a sound source image in a sound source localization method based on visual information assistance;

[0025] Figure 4This is the Hilbert envelope spectrum of the fan end bearing inner race fault audio;

[0026] Figure 5 is the Hilbert envelope spectrum of the mixed signal;

[0027] Figure 6 It is the Hilbert envelope spectrum of the fault source signal separated based on the existing GCC sound source localization algorithm;

[0028] Figure 7 It is the Hilbert envelope spectrum of the fault source signal separated by the sound source localization method based on visual information assistance of the present invention. DETAILED DESCRIPTION

[0029] like Figure 1 As shown, a method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion of the present invention comprises:

[0030] Step 1: Collect the mixed audio signal when the rotating machinery is working, obtain a dual-channel mixed audio signal, and collect the sound source image, specifically:

[0031] Step 1.1: Use a dual-channel microphone array to collect mixed audio signals generated by different components of the rotating machinery when it is working, and obtain a dual-channel mixed audio signal.

[0032] Step 1.2: At the same time, use the camera set at the center of the microphone array to capture the sound source image.

[0033] Step 2: Extract the time-frequency information of the dual-channel mixed audio signal to obtain the amplitude spectrum and phase spectrum of the dual-channel mixed audio signal, specifically:

[0034] Step 2.1: Perform discrete short-time Fourier transform on the left and right channel signals of the dual-channel mixed audio signal to obtain the time-frequency information of a single channel signal:

[0035]

[0036] Where f and t represent frequency and time respectively, k is a temporary variable required for the summation operation, c represents the index of the two channels, which takes a value of 1 or 2, s represents a discrete single-channel mixed audio signal, including two mixed audio signals s1 and s2, g represents the window function, and S is a complex matrix representing the time-frequency information of a single-channel mixed audio signal, including S 1,f×t and S 2,f×t .

[0037] Step 2.2: Decompose the time-frequency information into amplitude spectrum and phase spectrum:

[0038]

[0039] Where c represents the index of the two channels, which takes the value of 1 or 2, and V represents the amplitude spectrum of the mixed audio signal of a single channel, including V 1,f×t and V 2,f×t , φ represents the phase spectrum of a single channel mixed audio signal, including φ 1,f×t and φ 2,f×t .

[0040] Step 2.3: Splice the amplitude spectra of the mixed audio signals of the two channels along the time axis to obtain the amplitude spectrum V of the two-channel mixed audio signal f×t .

[0041] Step 3: Perform non-negative matrix decomposition on the amplitude spectrum of the dual-channel mixed audio signal to obtain the dictionary matrix and coefficient matrix, specifically:

[0042] Step 3.1: Randomly initialize the dictionary matrix W f×d And the coefficient matrix H d×t , the coefficient matrix consists of the coefficient matrix H of the two channels 1,d×t and H 2,d×t Splicing along the time axis obtains the initialization parameter α and the number of iterations. The function of the parameter α is to control the influence of the sparsity of the coefficient matrix on the reconstruction error.

[0043] Step 3.2: Iteratively update the coefficient matrix:

[0044]

[0045] Where d represents the number of dictionary atoms, f and t represent frequency and time respectively, and α is a parameter that controls the effect of coefficient matrix sparsity on reconstruction error. represents the coefficient matrix after the i-th iteration, represents the coefficient matrix after the i-1th iteration, Represents the dictionary matrix after the i-1th iteration, the “⊙” symbol represents the multiplication operation of the corresponding elements of the two matrices, and “T” represents the matrix transpose operation.

[0046] Step 3.3: Iteratively update the dictionary matrix:

[0047]

[0048] in, Represents the dictionary matrix after the i-th iteration.

[0049] Step 3.4: Normalize each column of the dictionary matrix, i.e., the dictionary atom, and adjust the coefficient matrix accordingly to ensure the equivalence of the decomposition.

[0050] Step 3.5: Determine whether the number of iterations reaches the set value. If so, end the iteration; otherwise, return to step 3.2 and repeat the iteration process.

[0051] Step 4: Use the dictionary matrix to update the frequency weighting function in the generalized cross-correlation algorithm to obtain the GCC-NMF joint function, which is:

[0052] Step 4.1: For a two-channel mixed audio signal, define its GCC function as:

[0053]

[0054] Among them, G(τ,t) represents the GCC function of the dual-channel mixed audio signal, t represents time, f represents frequency, τ represents delay, S 1,f×t and S 2,f×t They represent the time-frequency information of the two channels of the dual-channel mixed audio signal respectively, which is a complex matrix. ψ(f, t) represents the time-varying frequency weighting function, and “*” represents the conjugate operation.

[0055] Step 4.2: According to the properties of non-negative matrix decomposition, each column of the dictionary matrix can be interpreted as a dictionary atom of the amplitude spectrum of the two-channel mixed audio signal. The dictionary atom itself is a non-negative function of frequency. Therefore, the dictionary atom is normalized and a new frequency weighting function is defined:

[0056]

[0057] Where d represents the number of dictionary atoms, t represents time, f represents frequency, V 1,f×t and V 2,f×t Represents the amplitude spectrum of the two channels of the dual-channel mixed audio signal, W f×d Represents a dictionary matrix.

[0058] Step 4.3: Substitute the newly defined frequency weighting function into the GCC function definition formula to obtain the GCC-NMF joint function formula:

[0059]

[0060] Among them, G NMF (d,τ,t) represents the GCC-NMG joint function of the two-channel mixed audio signal, S 1,f×t and S 2,f×t Represent the time-frequency information of the two channels of the dual-channel mixed audio signal, which is a complex matrix; ψ NMF (d,f,t) represents the newly defined frequency weighting function.

[0061] Step 5: According to the collected sound source image, the sound source angle is calculated based on the sound source localization method assisted by visual information; then, combined with the geometric position relationship between the array elements and the sound source, the time delay of the acoustic signals of different sound sources reaching the two array elements is calculated. Figure 2 , Figure 3 The auxiliary map is calculated for the sound source localization method based on visual information assistance. The step 5 is specifically as follows:

[0062] Step 5.1: Set the size of the sound source image to w×h. For a reference object within the shooting range, use its geometric center in the image as the reference point, measure the actual angle θ0 between the reference point and the camera, and record the horizontal coordinate value x0 of the reference point in the sound source image. The vertical distance between the reference point in the sound source image and the image center axis is:

[0063]

[0064] Step 5.2: According to the trigonometric function relationship, calculate the vertical distance between the camera and the reference point in the sound source image:

[0065]

[0066] Step 5.3: Record the horizontal coordinate value x of the sound source to be measured in the sound source image b , b represents the index of the sound source, and the vertical distance between the sound source to be measured and the central axis of the image is:

[0067]

[0068] Step 5.4: Based on the trigonometric relationship again, derive the actual angle θ between the sound source to be measured and the camera b for:

[0069]

[0070] Step 5.5: Since the camera is set at the center of the two array elements of the microphone array, combined with the geometric position relationship between the array elements and the sound source, the time delay τ of the acoustic signals from different sound sources reaching the two array elements is calculated. b .

[0071] Step 6: According to the GCC-NMF joint function and time delay, the mask matrix of the sound source is obtained, which is:

[0072] Step 6.1: The time delay τ of the acoustic signals from different sound sources reaching the two array elements b Substitute them into the GCC-NMF joint function one by one, and get G NMF (d,τ b ,t).

[0073] Step 6.2: Define a binary mask matrix M for each sound sourceb,d×t , attribute the element at each position to make G NMF (d,τ b ,t) The sound source with the largest function value, that is, the value of the element at this position in the mask matrix is ​​obtained by the function argmax b G NMF (d,τ b ,t) It can be concluded that the elements of the mask matrix are either 0 or 1, and at the same position, only one of the mask matrices belonging to different sound sources has an element value of 1 at that position, that is, the sum of the mask matrices of all sound sources is equal to the full 1 matrix, so the mask generation rule is expressed by a mathematical expression:

[0074]

[0075] Among them, M b,dt is the mask matrix M of the bth sound source b,d×t an element of .

[0076] Step 7: Multiply the corresponding elements of the mask matrix and the coefficient matrix to obtain a new coefficient matrix, and then multiply the dictionary matrix by the new coefficient matrix to complete the inverse process of non-negative matrix decomposition, and obtain the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation. Then, the time domain audio signal belonging to each sound source after separation is obtained through ISTFT, which is specifically:

[0077] Step 7.1: Multiply the mask matrix and the coefficient matrix by their corresponding elements to obtain a new coefficient matrix, and then multiply the dictionary matrix by the new coefficient matrix to complete the inverse process of non-negative matrix decomposition, and obtain the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation. The calculation method is:

[0078]

[0079] Among them, b represents the index of the sound source, c represents the index of the channel, c takes the value of 1 or 2, d represents the number of dictionary atoms, t represents time, and f represents frequency. Represents the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation.

[0080] Step 7.2: With the phase spectrum φ c,f×t By combining the time-frequency information of the audio signal belonging to each channel of each sound source after separation, the time-frequency information is obtained, and then the ISTFT is used to transform it into the time-domain audio signal belonging to each sound source after separation, so as to realize the separation of the mixed audio signal.

[0081] Step 8: Use wavelet transform to perform multi-resolution decomposition and reconstruction on the separated time domain audio signals belonging to each sound source, and perform Hilbert envelope spectrum analysis on the first layer detail signal to extract the fault characteristic frequency and realize fault diagnosis, specifically:

[0082] Step 8.1: Use wavelet transform to perform multi-resolution decomposition and reconstruction on the separated time domain audio signals belonging to each sound source to obtain the first layer of detail signals.

[0083] Step 8.2: Perform Hilbert transform on the first layer detail signal, and then perform power spectrum analysis to obtain the Hilbert envelope spectrum of the first layer detail signal.

[0084] Step 8.3: Perform peak detection on the Hilbert envelope spectrum of the first layer detail signal. The frequency corresponding to the maximum amplitude is the fault characteristic frequency. Based on the fault characteristic frequency, it can be determined whether a fault occurs at the sound source and what kind of fault occurs.

[0085] In order to prove the effectiveness of the rotating machinery fault signal separation and diagnosis method based on audio-visual information fusion proposed in the present invention, the following experiment is designed. Three Bluetooth speakers are used to simulate the three components on the rotating machinery, and the normal working audio of the drive end bearing (source 1), the normal working audio of the fan end bearing (source 2) and the fan end bearing fault audio (source 3) are played respectively. The fan end fault is the inner ring fault of the bearing. The three sound sources are played at the same time, and the acoustic signal is collected by a microphone array. Two array elements are selected as the research object, and the corresponding dual-channel signals are retained. 10s of mixed audio is cut out and normalized as the input audio data. During the process of acoustic signal collection, the camera at the center of the array is used to capture the sound source image information so that the sound source can be marked and located later. After the separation is completed, the wavelet transform and Hilbert transform are used to extract the characteristic frequencies of the mixed signal and the separated signal respectively to verify the effectiveness of the fault source signal separation.

[0086] First, the characteristic frequency analysis of the fan end bearing inner ring fault audio is performed, and the results are as follows: Figure 4 As shown, it can be seen that the characteristic frequency of the inner race fault at this location is 29.83 Hz.

[0087] When collecting mixed signals, in order to better demonstrate the separation effect, the sound intensity of sound source 2 is increased so that the fault source signal is "submerged" in the mixed signal. The characteristic frequency analysis of the mixed signal is performed, and the results are as follows: Figure 5 As shown, it can be seen that the amplitude of the characteristic frequency of the fault signal is very small and much smaller than the characteristic frequency of the mixed signal. It needs to be separated before the fault type can be diagnosed.

[0088] The time delay of the sound source is estimated by using the existing GCC sound source localization algorithm and the sound source localization method based on visual information assistance of the present invention. The spatial information and the GCC-NMF joint function are combined to generate a mask matrix. The mask matrix is ​​multiplied by the coefficient matrix and the frequency domain is separated by NMF inverse transformation. Finally, the separated time domain audio is generated by STFT inverse transformation. The characteristic frequency analysis of the fault source signals separated by the two methods is performed respectively. The results are as follows: Figure 6 and Figure 7 shown.

[0089] Depend on Figure 6 and Figure 7 It can be seen that the horizontal coordinate of the maximum amplitude of the Hilbert envelope spectrum of the fault source signal separated by the GCC sound source localization algorithm is not the fault characteristic frequency. The horizontal coordinate of the maximum amplitude of the Hilbert envelope spectrum of the fault source signal separated by the sound source localization method based on visual information assistance of the present invention is 29.83Hz, that is, the fault characteristic frequency, and compared with the mixed signal, the amplitude at the characteristic frequency is expanded to about 8 times, indicating that the rotating machinery fault signal separation and diagnosis method based on audio-visual information fusion adopted by the present invention is effective, and can overcome the defects of the separation method based on the GCC sound source localization algorithm when the sound source angle difference is small, separate the fault signal from the mixed signal, and realize accurate diagnosis of the fault.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the concept of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for separating and diagnosing fault signals of rotating machinery based on audio-visual information fusion, characterized in that: include: Step 1: Collect mixed audio signals when the rotating machinery is working, obtain dual-channel mixed audio signals, and collect sound source images; Step 2: extracting the time-frequency information of the dual-channel mixed audio signal to obtain the amplitude spectrum and phase spectrum of the dual-channel mixed audio signal; Step 3: Perform non-negative matrix decomposition on the amplitude spectrum of the dual-channel mixed audio signal to obtain a dictionary matrix and a coefficient matrix; Step 4: Use the dictionary matrix to update the frequency weighting function in the generalized cross-correlation algorithm to obtain the GCC-NMF joint function; Step 5: Calculate the sound source angle based on the collected sound source image, and calculate the time delay of the acoustic signals from different sound sources reaching the two array elements based on the geometric position relationship between the array elements and the sound source; Step 6: Obtain the mask matrix of the sound source based on the GCC-NMF joint function and time delay; Step 7: Multiply the corresponding elements of the mask matrix and the coefficient matrix to obtain a new coefficient matrix, and then multiply the dictionary matrix by the new coefficient matrix to complete the inverse process of non-negative matrix decomposition to obtain the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation, and then obtain the time domain audio signal belonging to each sound source after separation through ISTFT; Step 8: Use wavelet transform to perform multi-resolution decomposition and reconstruction on the separated time domain audio signals belonging to each sound source, and perform Hilbert envelope spectrum analysis on the first layer of detail signals to extract the fault characteristic frequency and realize fault diagnosis.

2. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 1, characterized in that: The step 1 is specifically as follows: Step 1.1: Use a dual-channel microphone array to collect mixed audio signals generated by different components of the rotating machinery when it is working, and obtain a dual-channel mixed audio signal; Step 1.2: At the same time, use the camera set at the center of the microphone array to capture the sound source image.

3. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1: Perform discrete short-time Fourier transform on the left and right channel signals of the dual-channel mixed audio signal to obtain the time-frequency information of a single channel signal: Where f and t represent frequency and time respectively, k is a temporary variable required for the summation operation, c represents the index of the two channels, which takes a value of 1 or 2, s represents a discrete single-channel mixed audio signal, including two mixed audio signals s1 and s2, g represents the window function, and S is a complex matrix representing the time-frequency information of a single-channel mixed audio signal, including S 1,f×t and S 2,f×t ; Step 2.2: Decompose the time-frequency information into amplitude spectrum and phase spectrum: Where c represents the index of the two channels, which takes the value of 1 or 2, and V represents the amplitude spectrum of the mixed audio signal of a single channel, including V 1,f×t and V 2,f×t , φ represents the phase spectrum of a single channel mixed audio signal, including φ 1,f×t and φ 2,f×t ; Step 2.3: Splice the amplitude spectra of the mixed audio signals of the two channels along the time axis to obtain the amplitude spectrum V of the two-channel mixed audio signal f×t .

4. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 1, characterized in that: The step 3 is specifically as follows: Step 3.1: Randomly initialize the dictionary matrix W f×d And the coefficient matrix H d×t , the coefficient matrix consists of the coefficient matrix H of the two channels 1,d×t and H 2,d×t Splicing along the time axis, initializing the parameter α and the number of iterations. The function of the parameter α is to control the influence of the sparsity of the coefficient matrix on the reconstruction error; Step 3.2: Iteratively update the coefficient matrix: Where d represents the number of dictionary atoms, f and t represent frequency and time respectively, and α is a parameter that controls the effect of coefficient matrix sparsity on reconstruction error. represents the coefficient matrix after the i-th iteration, represents the coefficient matrix after the i-1th iteration, represents the dictionary matrix after the i-1th iteration, the "⊙" symbol represents the multiplication operation of the corresponding elements of the two matrices, and "T" represents the matrix transposition operation; Step 3.3: Iteratively update the dictionary matrix: in, Represents the dictionary matrix after the i-th iteration; Step 3.4: Normalize each column of the dictionary matrix, i.e., the dictionary atom, and adjust the coefficient matrix accordingly to ensure the equivalence of the decomposition; Step 3.5: Determine whether the number of iterations reaches the set value. If so, end the iteration; otherwise, return to step 3.2 and repeat the iteration process.

5. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 4, characterized in that: The step 4 is specifically as follows: Step 4.1: For a two-channel mixed audio signal, define its GCC function as: Among them, G(τ,t) represents the GCC function of the dual-channel mixed audio signal, t represents time, f represents frequency, τ represents delay, S 1,f×t and S 2,f×t Represent the time-frequency information of the two channels of the dual-channel mixed audio signal, which is a complex matrix. ψ(f,t) represents the time-varying frequency weighting function, and "*" represents the conjugate operation. Step 4.2: According to the properties of non-negative matrix decomposition, each column of the dictionary matrix can be interpreted as a dictionary atom of the amplitude spectrum of the two-channel mixed audio signal. The dictionary atom itself is a non-negative function of frequency. Therefore, the dictionary atom is normalized and a new frequency weighting function is defined: Where d represents the number of dictionary atoms, t represents time, f represents frequency, V 1,f×t and V 2,f×t Represents the amplitude spectrum of the two channels of the dual-channel mixed audio signal, W f×d represents the dictionary matrix; Step 4.3: Substitute the newly defined frequency weighting function into the GCC function definition formula to obtain the GCC-NMF joint function formula: Among them, G NMF (d,τ,t) represents the GCC-NMG joint function of the two-channel mixed audio signal, S 1,f×t and S 2,f×t Represent the time-frequency information of the two channels of the dual-channel mixed audio signal, which is a complex matrix; ψ NMF (d,f,t) represents the newly defined frequency weighting function.

6. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 5, characterized in that: The step 5 is specifically as follows: Step 5.1: Set the size of the sound source image to w×h. For a reference object within the shooting range, use its geometric center in the image as the reference point, measure the actual angle θ0 between the reference point and the camera, and record the horizontal coordinate value x0 of the reference point in the sound source image. The vertical distance between the reference point in the sound source image and the image center axis is: Step 5.2: According to the trigonometric function relationship, calculate the vertical distance between the camera and the reference point in the sound source image: Step 5.3: Record the horizontal coordinate value x of the sound source to be measured in the sound source image b , b represents the index of the sound source, and the vertical distance between the sound source to be measured and the central axis of the image is: Step 5.4: Based on the trigonometric relationship again, derive the actual angle θ between the sound source to be measured and the camera b for: Step 5.5: Since the camera is set at the center of the two array elements of the microphone array, combined with the geometric position relationship between the array elements and the sound source, the time delay τ of the acoustic signals from different sound sources reaching the two array elements is calculated. b .

7. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 6, characterized in that: The step 6 is specifically as follows: Step 6.1: The time delay τ of the acoustic signals from different sound sources reaching the two array elements b Substitute them into the GCC-NMF joint function one by one, and get G NMF (d,τ b ,t); Step 6.2: Define a binary mask matrix M for each sound source b,d×t , attribute the element at each position to make G NMF (d,τ b ,t) The sound source with the largest function value, that is, the value of the element at this position in the mask matrix is ​​obtained by the function argmax b G NMF (d,τ b ,t) It can be concluded that the elements of the mask matrix are either 0 or 1, and at the same position, there is only one matrix in the mask matrices belonging to different sound sources whose element value at that position is 1, that is, the sum of the mask matrices of all sound sources is equal to the full 1 matrix, so the mask generation rule is expressed by mathematical expression as follows: Among them, M b,dt is the mask matrix M of the bth sound source b,d×t an element of .

8. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 1, characterized in that: The step 7 is specifically as follows: Step 7.1: Multiply the mask matrix and the coefficient matrix by their corresponding elements to obtain a new coefficient matrix, and then multiply the dictionary matrix by the new coefficient matrix to complete the inverse process of non-negative matrix decomposition, and obtain the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation. The calculation method is: Among them, b represents the index of the sound source, c represents the index of the channel, c takes the value of 1 or 2, d represents the number of dictionary atoms, t represents time, and f represents frequency. Represents the amplitude spectrum of the audio signal belonging to each channel of each sound source after separation; Step 7.2: With the phase spectrum φ c,f×t By combining the time-frequency information of the audio signal belonging to each channel of each sound source after separation, the time-frequency information is obtained, and then the ISTFT is used to transform it into the time-domain audio signal belonging to each sound source after separation, so as to realize the separation of the mixed audio signal.

9. The method for separating and diagnosing rotating machinery fault signals based on audio-visual information fusion according to claim 1, characterized in that: The step 8 is specifically as follows: Step 8.1: Use wavelet transform to perform multi-resolution decomposition and reconstruction on the separated time domain audio signal belonging to each sound source to obtain the first layer detail signal; Step 8.2: Perform Hilbert transform on the first layer detail signal, and then perform power spectrum analysis to obtain the Hilbert envelope spectrum of the first layer detail signal; Step 8.3: Perform peak detection on the Hilbert envelope spectrum of the first layer detail signal. The frequency corresponding to the maximum amplitude is the fault characteristic frequency. Based on the fault characteristic frequency, it can be determined whether a fault occurs at the sound source and what kind of fault occurs.