Wind turbine generator fault diagnosis method based on audio and video identification

Abstract

The invention relates to a wind turbine generator fault diagnosis method based on audio and video recognition, which comprises the following steps of: firstly, installing a pan-tilt camera and a microphone array at a specific position, calibrating the position of each component, and simultaneously monitoring by adopting two modes: in the first mode, carrying out audio and video fault diagnosis and round inspection on large components in a cabin, and according to the position of each large component, carrying out fault diagnosis and round inspection on the large components; performing specific large component video and area enhanced audio data acquisition, and using audio and video algorithms for timing diagnosis; in the second mode, for abnormal sound recognition, spatial position information of a sound source is obtained by means of a cooperative acquisition mechanism of multiple microphones and a sound source positioning technology of a sound array, accurate resolving of the sound source direction is completed, sound position positioning of a large part in a closed space in a cabin of the wind turbine generator is achieved, and a pan-tilt camera is called to shoot the abnormal sound position; and fault diagnosis is carried out, so that operation and maintenance personnel can carry out processing in time to avoid great economic loss.

Description

Technical Field

[0001] This invention relates to a diagnostic method for major component faults in wind turbine nacelles based on audio and video recognition, belonging to the field of wind turbine monitoring technology. Background Technology

[0002] As the core equipment for new energy power generation, the operating status of wind turbine generators directly determines power generation efficiency and operation and maintenance costs. Key components such as bolts, main bearings, gearboxes, generators, and yaw systems within the nacelle are prone to failure under long-term high-load operation. Bolt loss can lead to insufficient strength in the connection between the blades and the hub, while abnormal noises from various rotating parts are typical early signs of failure. Failure to diagnose and address these faults in a timely manner can lead to escalation, power outages, and significant economic losses.

[0003] Existing fault diagnosis technologies for wind turbine generators mainly include vibration monitoring, oil analysis, and manual inspection. Vibration monitoring requires the installation of vibration sensors on key components, which presents problems such as difficult installation and complex wiring. Furthermore, the sensors are susceptible to interference from the generator's operating vibrations, resulting in low accuracy in identifying low-frequency abnormal noises. Oil analysis can only diagnose faults related to the lubrication system, limiting its applicability, and the diagnostic results are subject to lag. Manual inspection relies on the experience of maintenance personnel, is highly subjective, inefficient, and struggles to achieve real-time monitoring and accurate fault location, especially for multi-source abnormal noises within the enclosed space of the nacelle, where identification is extremely difficult.

[0004] Microphone array technology, with its spatial resolution capability for acoustic signals, is widely used in the field of sound source localization. Introducing this technology into wind turbine generator fault diagnosis allows for the non-contact acquisition of fault audio signals, avoiding direct contact between traditional sensors and turbine components, thus reducing installation and maintenance costs. However, current technologies lack dedicated audio diagnostic solutions for various types of abnormal noise faults in wind turbine generators. These solutions suffer from difficulties in separating abnormal noise signals from background noise, low accuracy in fault type identification, and insufficient localization precision. Simple audio detection solutions also suffer from the inability of on-site personnel to remotely confirm the situation, making them unsuitable for practical operation and maintenance needs. Therefore, a diagnostic technology capable of accurately identifying and locating multiple fault types in wind turbine generators is urgently needed. Summary of the Invention

[0005] To address the shortcomings of the prior art, this invention provides a wind turbine fault diagnosis method based on audio and video recognition. This method can diagnose faults in large components within the nacelle. By optimizing the microphone array layout, improving the audio signal processing algorithm, and employing an audio-visual linkage scheme, it achieves real-time identification, accurate location, and precise diagnosis of various types of faults, such as loose bolts in the hub and abnormal noises in the main bearing. Furthermore, it can trigger alarms for sudden events inside the nacelle, assisting on-site maintenance personnel in remotely confirming the urgency of the situation using a gimbal camera and audio, enabling timely handling and preventing significant economic losses.

[0006] The technical solution adopted to achieve the above-mentioned objectives of this invention is as follows:

[0007] A wind turbine fault diagnosis method based on audio and video recognition includes the following steps: S1 mounts a microphone array and a gimbal camera on the inner wall of the nacelle roof to collect and cover the sound and images of various large components inside the wind turbine nacelle, including the control cabinet, generator, gearbox, yaw system, and main bearing. S2 constructs a three-dimensional coordinate system based on the positions of various major components in the cabin and the installation positions of the microphone array and gimbal camera. The x-axis is the length direction of the cabin, the y-axis is the width direction of the cabin, and the z-axis is the height direction of the cabin. The spatial position coordinates of each major component, microphone array, and gimbal camera are further calibrated. The S3 control center periodically controls the microphone array and gimbal camera to collect audio and video images of various large components inside the cabin; at the same time, the control center synchronously controls the microphone array to monitor for abnormal noises in the cabin in real time. When no abnormal noises are detected, S3.1-S3.3 are executed in a loop; when an abnormal noise is detected, the execution of S3.1-S3.3 is stopped and the process directly proceeds to S4. S3.1 When periodically acquiring audio and video images of various large components inside the cabin, the horizontal rotation angle P and vertical pitch angle T of the gimbal camera are calculated based on the spatial coordinates of each large component. The gimbal camera is then controlled to rotate to face the corresponding large component according to the P / T value and acquire video images. Simultaneously, the microphone array is controlled to start the beamforming algorithm, so that the receiving main lobe of the microphone array points to the spatial coordinates of each large component, and the audio signal from the direction of the corresponding large component is directionally enhanced and acquired, while suppressing noise interference from other directions. S3.2 After beamforming is completed, residual noise is further filtered to retain pure audio data of each major component; S3.3 performs fusion analysis on the collected video and audio data, and comprehensively judges the operating status of each major component based on image diagnostic algorithms and audio diagnostic algorithms. It also performs cyclical inspections of each major component in the engine room and generates daily diagnostic status reports for maintenance personnel to confirm. When the microphone array detects abnormal noises inside the cabin, S4 first uses the multi-channel audio signals collected by the microphone array to calculate the time difference of the sound source of the corresponding large component using the TDOA algorithm, and then infers its spatial coordinates. Based on the spatial coordinates derived from the previous step, S5 calculates the horizontal rotation angle P' and vertical pitch angle T' of the gimbal camera, and controls the gimbal camera to rotate to face the corresponding large component according to the P' / T' value to acquire video images; Based on the spatial coordinates inverted from S4, S6 generates visual information representing the spatial intensity distribution of the sound field, and overlays this visual information onto the video image acquired in S5 to form a fault location indication for audio-visual fusion. S7 outputs the fault location indication data from the audio-visual fusion process in the previous step, which is used for fault alarm and remote manual confirmation; and then re-enters S3.

[0008] Furthermore, the microphone array in S1 adopts a linear distribution, circular distribution, planar distribution, three-dimensional array, or distributed array method, so as to meet the requirement of obtaining the sound source location based on the microphone array; the gimbal camera is deployed in the center of the microphone array, and the gimbal camera is located at the origin of the three-dimensional coordinate system.

[0009] Furthermore, the microphone array in S1 adopts a planar distribution, specifically a cross shape, wherein four microphones are arranged in the x-axis direction, each spaced r meters apart, and four microphones are arranged in the y-axis direction, each spaced r meters apart; the x-axis direction is the length direction of the cabin, and the y-axis direction is the width direction of the cabin.

[0010] Furthermore, the calculation formulas for the horizontal rotation angle P and the vertical pitch angle T in S3.1 are as follows: ; In the formula, -xm, ym, and -zm are the three-dimensional coordinates of each major component in the three-dimensional coordinate system.

[0011] Furthermore, the specific steps of the beamforming algorithm in S3.1 are as follows: S3.1.1 First, calculate the phase delay of each microphone: Calculate the phase delay τ_i based on the straight-line distance from the target sound source to different microphones. Each microphone needs to be delayed by τ_i time in order for the target signals to be synchronously superimposed. S3.1.2 Cross-shaped array weighted beamforming: The delay-summation beamforming algorithm is adopted to perform delay compensation, weighted amplification, and superposition summation on the signal of each microphone. The weight allocation method is as follows: microphones closer to the target direction are given higher weights, and microphones farther away from the target direction are given lower weights, thereby strengthening the signal in the target direction. The beam pointing angle is specifically set as the coordinate of the main lobe pointing to the target. The main lobe width is controlled between 10° and 20° to cover the target area and reduce side lobe interference.

[0012] Furthermore, the specific steps for filtering residual noise in S3.2 are as follows: S3.2.1 Adaptive noise estimation: A real-time noise model is established using non-target direction signals collected by a microphone array to dynamically estimate the frequency and amplitude characteristics of the noise; S3.2.2 Frequency Domain Noise Suppression: The beamformed audio signal is converted to the frequency domain, and the spectral differences between the target signal and the noise model are compared; adaptive attenuation is performed on frequency bands with high noise content, and the sound characteristic frequency bands corresponding to faults in large components are preserved and enhanced; S3.2.3 Signal Reconstruction: The processed frequency domain signal is inversely converted back to the time domain, and the enhanced audio signal of the large component is output for subsequent fault feature extraction.

[0013] Furthermore, the specific method of S4 is as follows: S4.1 Signal Synchronization and Time Difference Extraction: Audio is collected synchronously by 8 microphones. With the center origin S (0,0,0) as a reference, the timestamps of abnormal noises from large components arriving at each microphone are extracted and recorded as t1~t8. S4.2 Grouping and Time Difference Calculation: For the four microphones sx1-sx4 along the X-axis, calculate the time difference from the abnormal noise source to sx1 and sx4, and sx2 and sx3. Combine this with the spacing between sx1-sx4 to deduce the X-coordinate of the abnormal noise source. For the four microphones sy1-sy4 along the Y-axis, calculate the time difference from sy1 and sy4, and sy2 and sy3. Combine this with the spacing between sy1-sy4 to deduce the Y-coordinate of the abnormal noise source. S4.3 3D Supplement: Combining the time difference between the arrival times of the abnormal noise source and the microphones in the X and Y axis directions, the Z-axis coordinate of the abnormal noise source is calculated to finally output the spatial location coordinates of the abnormal noise source.

[0014] Furthermore, after back-deriving the spatial coordinates of the abnormal noise source in S4, the distance of each microphone from the abnormal noise source is further calculated based on the calibrated spatial coordinates of each major component. A beamforming algorithm is selected to perform delay compensation and weighted summation on each microphone channel, so that the signals from the direction of the sound source are superimposed in phase, and the signals from the direction of the non-sound source cancel each other out in phase. At the same time, the signals from the direction of the non-sound source are regarded as noise signals, and their spectral characteristics are statistically analyzed in real time. Combined with adaptive filtering or spectral subtraction, the spectral components of the noise signal are subtracted in the frequency domain, and the effective signal from the direction of the sound source is retained.

[0015] Furthermore, in S6, the sound pressure intensity distribution of the three-dimensional spatial field is first calculated through the microphone array, then converted into two-dimensional pixel coordinates of the gimbal camera image, and finally a visual label is superimposed to accurately mark the location of the audio source, thereby assisting in marking the abnormal noise location detected by the microphone array on the image captured by the gimbal camera.

[0016] Furthermore, in S6, the sound pressure intensity distribution is mapped to a color space, and different colors are set to correspond to different intensities of sound pressure energy distribution. Low sound pressure areas are represented by blue or green, medium sound pressure areas by yellow, and high sound pressure areas by red or purple.

[0017] Compared with the prior art, the technical solution provided by the present invention has the following advantages:

[0018] 1) This invention uses an audio and video solution to identify and detect the status of large components and equipment in the cabin. The microphone array and gimbal camera are deployed on the top of the cabin, which can cover various places including the hub-cabin connection, main shaft, gearbox, generator, cabin control cabinet, yaw system, etc. The gimbal camera is deployed in the center of the microphone array.

[0019] 2) The diagnostic method provided by this invention has two modes. Mode 1, which involves rotating large components, can perform daily status monitoring. Based on the position of the large components, the rotating scan enables diagnosis of images captured by the gimbal and provides daily diagnosis of the large components through sound enhancement, filtering, and other functions.

[0020] 3) The abnormal noise recognition in Mode 2 can monitor sound data in real time, quickly locate, collect and diagnose abnormal noise data that are not features in the database, notify maintenance personnel for manual confirmation, and provide visual labels to accurately mark the location of the audio to assist maintenance personnel in diagnosis. Attached Figure Description

[0021] Figure 1 The overall flowchart of the wind turbine fault diagnosis method based on audio and video recognition provided by the present invention;

[0022] Figure 2 This is a schematic diagram of the installation of the microphone array and gimbal camera in the embodiment;

[0023] Figure 3 This is a top view of the various major components, microphone array, and gimbal camera in a three-dimensional coordinate system in the embodiment.

[0024] Figure 4 This is a front view of the various major components, microphone array, and gimbal camera in a three-dimensional coordinate system in the embodiment.

[0025] Figure 5 This is a left view of the various major components, microphone array, and gimbal camera in the three-dimensional coordinate system in the embodiment.

[0026] Figure 6 This is a schematic diagram illustrating the mapping of sound pressure intensity distribution to chromatographic space and the transformation of two-dimensional pixel coordinates in the embodiment.

[0027] Figure 7 This is a schematic diagram illustrating the location of abnormal noises marked on images captured by a gimbal camera in an embodiment.

[0028] In the diagram: 1-Nacelle, 2-Chassis, 3-Control cabinet, 4-Generator, 5-Yaw system, 6-Microphone array, 7-Gimbal camera, 8-Gearbox, 9-Hub, 10-Blade. Detailed Implementation

[0029] The overall process of the wind turbine fault diagnosis method based on audio and video recognition provided by this invention is as follows: Figure 1 As shown in the accompanying drawings and embodiments, the present invention will now be described in detail.

[0030] S1 Since this invention is based on audio and video recognition, the microphone array 6 and the gimbal camera 7 need to be installed on the inner wall of the top of the cabin 1, such as... Figure 2 As shown, this allows for the collection and coverage of sound and visual data from various major components within the nacelle 1 of the wind turbine, as referenced. Figure 2 The major components are all mounted on the chassis 2 inside the engine room 1, and mainly include a control cabinet 3, a generator 4, a gearbox 8, a yaw system 5, and a main bearing. The top of the main bearing is connected to a hub 9, and the hub 9 is bolted to the blades 10. This invention mainly focuses on monitoring and fault diagnosis of the above-mentioned major components. The microphone array 6 can be arranged linearly, in a ring, in a planar, three-dimensional, or distributed manner, depending on whether the sound source location can be obtained from the microphone array. The gimbal camera 7 is deployed in the center of the microphone array, and the gimbal camera is located at the origin of the three-dimensional coordinate system.

[0031] S2 constructs a three-dimensional coordinate system based on the positions of various major components within the cabin, as well as the installation locations of the microphone array and gimbal camera, such as... Figure 3 As shown, the x-axis represents the length of the cabin, the y-axis represents the width of the cabin, and the z-axis represents the height of the cabin. The spatial coordinates of each major component, as well as the microphone array and gimbal camera, are further defined. The microphone array is planarly distributed, specifically in a cross shape, as shown below. Figure 3As shown, four microphones are positioned along the x-axis, each spaced r meters apart, and four microphones are positioned along the y-axis, each spaced r meters apart. The x-axis represents the length of the cabin, and the y-axis represents the width of the cabin. The calibrated spatial positions of each major component are determined based on the locations of the large components within the cabin, as well as the installation positions of the microphone array and the gimbal camera. For example, the gimbal camera is located at (0, 0, 0), microphones sx1 (1.5r, 0, 0), sx2 (0.5r, 0, 0), sx3 (-0.5r, 0, 0), sx4 (-1.5r, 0, 0), sy1 (0, 1.5r, 0), sy2 (0, 0.5r, 0), sy3 (0, -0.5r, 0), sy4 (0, -1.5r, 0), and the generator is located at (-xm, ym, -zm).

[0032] like Figure 4 As shown, the four microphones in the x-direction are linearly distributed. Due to the limited space in the cabin, adjusting the appropriate microphone spacing *r* allows the microphones to satisfy the near-field sound source model. The Time Difference of Arrival (TDOA) localization method calculates the distance difference based on the time delay difference of the sound source arriving at different microphones, and then uses the distance difference and the spatial geometric position of the microphones to confirm the sound source location. This method consists of two steps: TDOA estimation (estimating the time difference of signal arrival at each microphone) and TDOA localization (using geometric relationships to determine the sound source location). This method can clearly define the sound source's position in the x-direction of the cabin. Similarly, based on... Figure 5 It can pinpoint the exact location of the sound source in the y-direction of the cabin. When various components inside the cabin emit sound, audio data can be collected using a microphone array to identify the component causing the sound.

[0033] After completing the preparatory work in S1 and S2, S3 can simultaneously perform two monitoring modes. In the first mode, the control center periodically controls the microphone array and pan-tilt camera to collect audio and video images of various large components inside the cabin, and executes S3.1-S3.3 in a loop.

[0034] In the second mode, the control center synchronously controls the microphone array to monitor abnormal noises in the cabin in real time. Abnormal noises include many types, such as the sound of bolts falling and colliding, the abnormal noise of hub pitch bearing bolts falling, the sound of main shaft rotation jamming, the sound of gearbox jamming, the sound of generator jamming, and the abnormal noise of yaw jamming.

[0035] If no abnormal noise is detected, execute S3.1-S3.3 in a loop; if an abnormal noise is detected, stop executing S3.1-S3.3 and proceed directly to S4.

[0036] S3.1-S3.3 are the core steps of the large component inspection, which will be explained in detail below.

[0037] S3.1 When periodically acquiring audio and video images of various large components inside the cabin, the horizontal rotation angle P and vertical pitch angle T of the gimbal camera are calculated based on the spatial coordinates of each component. The gimbal camera is then controlled to rotate to face the corresponding component according to the P / T values ​​and acquire video images. Taking the generator as an example, after triggering the set acquisition time, the gimbal camera is controlled to rotate to the generator position and record video. The calculation formulas for the horizontal rotation angle P and vertical pitch angle T are as follows:

[0038] ; In the formula, -xm, ym, and -zm are the three-dimensional coordinates of the generator in the three-dimensional coordinate system, which are calibrated in S2.

[0039] The microphone array is synchronously controlled to initiate a beamforming algorithm, causing the main lobe of the microphone array to point towards the spatial coordinates of each large component. This enables directional enhancement and acquisition of audio signals from the corresponding large component direction, while suppressing noise interference from other directions. The specific steps of the beamforming algorithm are as follows:

[0040] S3.1.1 First, calculate the phase delay of each microphone: Calculate the phase delay τ_i based on the straight-line distance from the target sound source to each microphone. Taking sx1(1.5r, 0, 0) as an example, the distance... The phase delay τ_i = d_i / c (where c is the speed of sound ≈ 343m / s) means that each microphone needs to be delayed by τ_i time in order for the target signals to be synchronously superimposed.

[0041] S3.1.2 Cross-shaped array weighted beamforming: The delay-summation beamforming algorithm is adopted to perform delay compensation, weighted amplification, and superposition summation on the signal of each microphone. The weight allocation method is as follows: microphones closer to the target direction (e.g., for the generator, the X-axis coordinate = -xm is the negative direction, and sx3 and sx4 have higher weights than sx1 and sx2; the Y-axis coordinate = ym is the positive direction, and sy1 and sy2 have higher weights than sy3 and sy4) are given higher weights (e.g., 0.8~1.0), and microphones farther away have lower weights (e.g., 0.2~0.5), thereby strengthening the signal in the target direction; the beam pointing angle is specifically set as the coordinates (-xm, ym, -zm) of the main lobe pointing to the target (taking the generator as an example), and the main lobe width is controlled at 10°~20° to cover the target area and reduce side lobe interference.

[0042] S3.2 After beamforming is completed, residual noise is further filtered to retain the pure audio data of each major component; the specific steps for filtering residual noise are as follows:

[0043] S3.2.1 Adaptive noise estimation: By collecting non-target direction signals (such as cabin ambient wind noise and vibration noise from other components) through a microphone array, a real-time noise model is established to dynamically estimate the frequency and amplitude characteristics of the noise;

[0044] S3.2.2 Frequency Domain Noise Suppression: The beamformed audio signal is converted to the frequency domain (via FFT), and the spectral differences between the target signal and the noise model are compared. Adaptive attenuation is performed on frequency bands with high noise content (such as low-frequency wind noise and high-frequency electromagnetic interference), while the sound characteristic frequency bands corresponding to the failure of large components (such as the abnormal noise of gearbox failure in 100~1000Hz and the abnormal noise of bearing failure in 1000~5000Hz) are preserved and enhanced.

[0045] S3.2.3 Signal Reconstruction: The processed frequency domain signal is inversely converted back to the time domain, and the enhanced audio signal of the large component is output for subsequent fault feature extraction.

[0046] S3.3 performs fusion analysis on the collected video and audio data. Based on image and audio diagnostic algorithms, it comprehensively judges the operating status of each major component and cyclically inspects each major component in the engine room. Taking the generator as an example: image technology is used to diagnose and identify the position of the generator drive end bearing to determine if there is a bearing misalignment problem; image frame difference method is used to determine the overall longitudinal displacement of the generator to determine if there is a jitter problem. Audio diagnostic algorithms are used to determine if there is any abnormal noise from bearing jamming in the generator. After the inspection, a daily diagnostic status report for each major component is generated for maintenance personnel to confirm. It should be noted that there are many mature and applied image and audio diagnostic algorithms in this field. This patent directly adopts mature algorithms from existing technologies, therefore, this part is not described in detail.

[0047] Step S4 is the follow-up processing step under mode two when the microphone array detects abnormal noise in the cabin. First, the multi-channel audio signal collected by the microphone array is used to calculate the time difference of the sound source of the corresponding large component using the TDOA algorithm, and then its spatial coordinates are deduced.

[0048] Due to the large number of components and the complexity of the ambient sound, after identifying the location of the noise, a directional audio enhancement and noise removal method can be implemented using "beamforming + noise suppression algorithm" to filter the sound of specific components. Taking the abnormal noise source of the generator as an example, the specific steps are as follows: S4.1 Signal synchronization and time difference extraction: Eight microphones synchronously collect audio. Taking the center origin (0, 0, 0) as a reference, the timestamps of the abnormal noise from the large components reaching each microphone are extracted and recorded as t1~t8;

[0049] S4.2 Grouping and Time Difference Calculation: For the four microphones sx1-sx4 along the X-axis, calculate the time difference between the abnormal noise source and sx1 and sx4, and between sx2 and sx3. Combine this with the spacing between sx1-sx4 (e.g., the spacing between sx1 and sx4 is 3r) to inversely deduce the X-coordinate of the abnormal noise source. For the four microphones sy1-sy4 along the Y-axis, calculate the time difference between sy1 and sy4, and between sy2 and sy3. Combine this with the spacing between sy1-sy4 to inversely deduce the Y-coordinate of the abnormal noise source.

[0050] S4.3 3D Supplement: Combining the time difference between the arrival times of the abnormal noise source and the microphones in the X and Y axis directions, the Z-axis coordinate of the abnormal noise source is calculated to finally output the spatial position coordinates (-xm, ym, -zm) of the abnormal noise source (generator).

[0051] After reversing the spatial coordinates (-xm, ym, -zm) of the abnormal noise source, the distance of each microphone from the abnormal noise source is further calculated based on the calibrated spatial coordinates of each major component. Beamforming algorithms (such as Delay-and-Sum, MVDR, LCMV, etc.) are selected to perform delay compensation and weighted summation on each microphone channel, ensuring that signals from the sound source direction are superimposed in phase (energy enhancement) and signals from non-sound source directions are phase-cancelled (energy attenuation). Simultaneously, signals from non-sound source directions are treated as noise signals, and their spectral characteristics (such as amplitude and frequency distribution) are statistically analyzed in real time. Combined with adaptive filtering (such as NLMS, RLS) or spectral subtraction, the spectral components of the noise signal are subtracted in the frequency domain, retaining the effective signal from the sound source direction. The detailed processing flow in this step is similar to that in S3.1 and will not be repeated here.

[0052] Based on the spatial coordinates derived from the previous step, S5 calculates the horizontal rotation angle P' and vertical pitch angle T' of the PTZ camera. It then controls the PTZ camera to rotate to face the corresponding large component based on the P' / T' value for video image acquisition. The calculation method for the P' / T' value is consistent with that in S3.1. Simultaneously, an image diagnostic algorithm provides a basic diagnosis. In cases of severe abnormal noise, data is pushed to the field-end display interface, allowing maintenance personnel to remotely control the PTZ camera to confirm the situation and formulate subsequent handling plans.

[0053] Based on the spatial coordinates derived from S4, S6 generates visual information representing the spatial intensity distribution of the sound field, and overlays this visual information onto the video image acquired in S5 to form a fault location indication for audio-visual fusion.

[0054] In practice, the sound pressure intensity distribution in the three-dimensional spatial field is first calculated using a microphone array, and then converted into two-dimensional pixel coordinates in the pan-tilt camera image, such as... Figure 6As shown, finally, visual labels are superimposed to accurately mark the location of the abnormal noise, thereby assisting in marking the location of the abnormal noise detected by the microphone array on the image captured by the gimbal camera.

[0055] By mapping sound pressure level (in dB SPL) to a color space, different colors correspond to different intensities of sound wave energy distribution: for example, low sound pressure areas are typically represented by blue or green, medium intensity by yellow, and high intensity by red or purple. This color coding scheme not only intuitively reflects the intensity distribution of noise sources, but also helps engineers quickly locate abnormal noise sources through real-time overlay of pseudo-color images and optical images.

[0056] S7 outputs the fault location indication data from the audio-video fusion process in the previous step, which is used for fault alarm and remote manual confirmation. Figure 7 This diagram shows the location of abnormal noise sources in the engine room monitored during actual operation, superimposed with video images. The larger colored block at the bottom represents the gearbox, and the smaller colored block at the top represents the cooling fan. This diagram visually illustrates the specific location and intensity of the abnormal noise, facilitating identification by the work team.

[0057] After completing the relevant steps in Mode 2 above, re-enter S3, and the control center sends a command to continue executing the timed polling task in Mode 1.

[0058] The embodiments described above are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A method for fault diagnosis of wind turbine generators based on audio and video recognition, characterized in that... Includes the following steps: S1 mounts a microphone array and a gimbal camera on the inner wall of the nacelle roof to collect and cover the sound and images of various large components inside the wind turbine nacelle, including the control cabinet, generator, gearbox, yaw system, and main bearing. S2 constructs a three-dimensional coordinate system based on the positions of various major components in the cabin and the installation positions of the microphone array and gimbal camera. The x-axis is the length direction of the cabin, the y-axis is the width direction of the cabin, and the z-axis is the height direction of the cabin. The spatial position coordinates of each major component, microphone array, and gimbal camera are further calibrated. The S3 control center periodically controls the microphone array and gimbal camera to collect audio and video images of various large components inside the cabin; at the same time, the control center synchronously controls the microphone array to monitor for abnormal noises in the cabin in real time. When no abnormal noises are detected, S3.1-S3.3 are executed in a loop; when an abnormal noise is detected, the execution of S3.1-S3.3 is stopped and the process directly proceeds to S4. S3.1 When periodically acquiring audio and video images of various large components inside the cabin, the horizontal rotation angle P and vertical pitch angle T of the gimbal camera are calculated based on the spatial coordinates of each large component. The gimbal camera is then controlled to rotate to face the corresponding large component according to the P / T value and acquire video images. Simultaneously, the microphone array is controlled to start the beamforming algorithm, so that the receiving main lobe of the microphone array points to the spatial coordinates of each large component, and the audio signal from the direction of the corresponding large component is directionally enhanced and acquired, while suppressing noise interference from other directions. S3.2 After beamforming is completed, residual noise is further filtered to retain pure audio data of each major component; S3.3 performs fusion analysis on the collected video and audio data, and comprehensively judges the operating status of each major component based on image diagnostic algorithms and audio diagnostic algorithms. It also performs cyclical inspections of each major component in the engine room and generates daily diagnostic status reports for maintenance personnel to confirm. When the microphone array detects abnormal noises inside the cabin, S4 first uses the multi-channel audio signals collected by the microphone array to calculate the time difference of the sound source of the corresponding large component using the TDOA algorithm, and then infers its spatial coordinates. Based on the spatial coordinates derived from the previous step, S5 calculates the horizontal rotation angle P' and vertical pitch angle T' of the gimbal camera, and controls the gimbal camera to rotate to face the corresponding large component according to the P' / T' value to acquire video images; Based on the spatial coordinates inverted from S4, S6 generates visual information representing the spatial intensity distribution of the sound field, and overlays this visual information onto the video image acquired in S5 to form a fault location indication for audio-visual fusion. S7 outputs the fault location indication data from the audio-visual fusion process in the previous step, which is used for fault alarm and remote manual confirmation; and then re-enters S3.

2. The wind turbine fault diagnosis method based on audio and video recognition according to claim 1, characterized in that: In S1, the microphone array can be linearly distributed, circularly distributed, planarly distributed, three-dimensional array, or distributed array, depending on whether the sound source location can be obtained from the microphone array. The gimbal camera is deployed in the center of the microphone array and is located at the origin of the three-dimensional coordinate system.

3. The wind turbine fault diagnosis method based on audio and video recognition according to claim 2, characterized in that: The microphone array in S1 is planarly distributed, specifically in a cross shape, with 4 microphones arranged along the x-axis, each spaced r meters apart, and 4 microphones arranged along the y-axis, each spaced r meters apart; the x-axis is the length direction of the cabin, and the y-axis is the width direction of the cabin.

4. The wind turbine fault diagnosis method based on audio and video recognition according to claim 2, characterized in that: The formulas for calculating the horizontal rotation angle P and the vertical pitch angle T in S3.1 are as follows: ; In the formula, -xm, ym, and -zm are the three-dimensional coordinates of each major component in the three-dimensional coordinate system.

5. The wind turbine fault diagnosis method based on audio and video recognition according to claim 3, characterized in that: The specific steps of the beamforming algorithm in S3.1 are as follows: S3.1.1 First, calculate the phase delay of each microphone: Calculate the phase delay τ_i based on the straight-line distance from the target sound source to different microphones. Each microphone needs to be delayed by τ_i time in order for the target signals to be synchronously superimposed. S3.1.2 Cross-shaped array weighted beamforming: The delay-summation beamforming algorithm is adopted to perform delay compensation, weighted amplification, and superposition summation on the signal of each microphone. The weight allocation method is as follows: microphones closer to the target direction are given higher weights, and microphones farther away from the target direction are given lower weights, thereby strengthening the signal in the target direction. The beam pointing angle is specifically set as the coordinate of the main lobe pointing to the target. The main lobe width is controlled between 10° and 20° to cover the target area and reduce side lobe interference.

6. The wind turbine fault diagnosis method based on audio and video recognition according to claim 5, characterized in that: The specific steps for filtering residual noise in S3.2 are as follows: S3.2.1 Adaptive noise estimation: A real-time noise model is established using non-target direction signals collected by a microphone array to dynamically estimate the frequency and amplitude characteristics of the noise; S3.2.2 Frequency Domain Noise Suppression: The beamformed audio signal is converted to the frequency domain, and the spectral differences between the target signal and the noise model are compared; adaptive attenuation is performed on frequency bands with high noise content, and the sound characteristic frequency bands corresponding to faults in large components are preserved and enhanced; S3.2.3 Signal Reconstruction: The processed frequency domain signal is inversely converted back to the time domain, and the enhanced audio signal of the large component is output for subsequent fault feature extraction.

7. The wind turbine fault diagnosis method based on audio and video recognition according to claim 3, characterized in that: The specific method for S4 is as follows: S4.1 Signal Synchronization and Time Difference Extraction: Audio is collected synchronously by 8 microphones. With the center origin S(0,0,0) as a reference, the timestamps of abnormal noises from large components arriving at each microphone are extracted and denoted as t1~t8. S4.2 Grouping and Time Difference Calculation: For the four microphones sx1-sx4 along the X-axis, calculate the time difference from the abnormal noise source to the time between sx1 and sx4, and between sx2 and sx3. Combine this with the distance between sx1-sx4 to deduce the X-coordinate of the abnormal noise source. For the four microphones sy1-sy4 along the Y-axis, calculate the time difference from the time between sy1 and sy4, and between sy2 and sy3. Combine this with the distance between sy1-sy4 to deduce the Y-coordinate of the abnormal noise source. S4.3 3D Supplement: Combining the time difference between the arrival times of the abnormal noise source and the microphones in the X and Y axis directions, the Z-axis coordinate of the abnormal noise source is calculated to finally output the spatial location coordinates of the abnormal noise source.

8. The wind turbine fault diagnosis method based on audio and video recognition according to claim 7, characterized in that: After retrieving the spatial coordinates of the abnormal noise source in S4, the distance of each microphone from the abnormal noise source is further calculated based on the calibrated spatial coordinates of each major component. A beamforming algorithm is then used to compensate for delays and perform weighted summation on each microphone channel, ensuring that signals from the direction of the noise source are superimposed in phase, while signals from directions other than the noise source are phase-cancelled. Simultaneously, signals from directions other than the noise source are treated as noise signals, and their spectral characteristics are analyzed in real time. By combining adaptive filtering or spectral subtraction, the spectral components of the noise signal are subtracted in the frequency domain, while retaining the effective signal in the direction of the sound source.

9. The wind turbine fault diagnosis method based on audio and video recognition according to claim 1, characterized in that: In S6, the sound pressure intensity distribution of the three-dimensional spatial field is first calculated through the microphone array, then converted into two-dimensional pixel coordinates of the gimbal camera image, and finally a visual label is superimposed to accurately mark the location of the audio source, thereby marking the abnormal noise location detected by the microphone array on the image captured by the gimbal camera.

10. The wind turbine fault diagnosis method based on audio and video recognition according to claim 9, characterized in that: In S6, the sound pressure intensity distribution is mapped to a color space, and different colors are set to correspond to different intensities of sound pressure energy distribution. Low sound pressure areas are represented by blue or green, medium sound pressure areas by yellow, and high sound pressure areas by red or purple.

Images