Fault positioning method and device based on microphone array and visual perception fusion, equipment and medium

Abstract

The invention discloses a fault positioning method and device based on microphone array and visual perception fusion, equipment and a medium, and relates to the technical field of industrial automation and intelligent monitoring, and the method comprises the steps: controlling a microphone array to collect an original sound source signal of electromechanical equipment; wherein the microphones in the microphone array are located in different directions; matching the original sound source signals with voiceprint features in a mechanical fault voiceprint sample library so as to screen out fault sound source signals from the original sound source signals; obtaining a time difference of arrival result among the microphones by using a GCC-PHAT algorithm and the fault sound source signal, and obtaining sound source azimuth information of a fault sound source in the electromechanical equipment according to the time difference of arrival result; and fusing the sound source azimuth information and the equipment abnormal area information in the original visual image of the electromechanical equipment so as to carry out fault positioning on the electromechanical equipment. According to the scheme, the fault positioning accuracy of the electromechanical equipment can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial automation and intelligent monitoring, and in particular to a fault positioning method and device based on microphone array and visual perception fusion, equipment and medium. BACKGROUND

[0002] As a core component in the fields of industrial production, power supply and rail transportation, the stable operation of electromechanical equipment is directly related to production efficiency, operation safety and maintenance cost. Once a fault occurs, it may lead to serious consequences such as production line shutdown and safety accidents. Therefore, accurate fault positioning of electromechanical equipment is the key to ensuring its reliable operation.

[0003] At present, fault positioning based on microphone array and visual perception fusion mainly relies on manual inspection combined with simple instrument detection. In addition, in industrial environments, multi-source interference such as electromagnetic noise and mechanical vibration is superimposed. Traditional single-mode detection methods have large positioning errors and cannot meet the operation and maintenance needs of precision equipment.

[0004] In summary, how to improve the accuracy of fault positioning based on microphone array and visual perception fusion is a problem to be solved in the field. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a fault positioning method, device, equipment and medium based on microphone array and visual perception fusion to improve the accuracy of fault positioning based on microphone array and visual perception fusion. The specific scheme is as follows:

[0006] In a first aspect, the present application discloses a fault positioning method based on microphone array and visual perception fusion, comprising:

[0007] controlling a microphone array to collect original sound source signals of an electromechanical equipment; wherein each microphone in the microphone array is located at different azimuths;

[0008] matching the original sound source signals with soundprint features in a mechanical fault soundprint sample library to filter out fault sound source signals from the original sound source signals;

[0009] obtaining time difference of arrival results between each microphone using a GCC-PHAT algorithm and the fault sound source signals, and obtaining sound source azimuth information of a fault sound source in the electromechanical equipment according to the time difference of arrival results;

[0010] fusing the sound source azimuth information and device abnormal region information in the original visual image of the electromechanical equipment to position the fault of the electromechanical equipment.

[0011] Optionally, the matching the original sound source signal with the soundprint features in the mechanical fault soundprint sample library to screen out a fault sound source signal from the original sound source signal comprises:

[0012] filtering noise in the original sound source signal by using a deep noise suppression network to obtain a processed sound source signal;

[0013] matching the processed sound source signal with the soundprint features in the mechanical fault soundprint sample library to screen out a fault sound source signal from the original sound source signal.

[0014] Optionally, the obtaining of the sound source azimuth information of the fault sound source in the electromechanical equipment according to the time difference of arrival result comprises:

[0015] if the distance between the fault sound source signal and the microphone array is greater than a preset far-near field critical value, determining that the sound source positioning model of the fault sound source in the electromechanical equipment is a plane wave model;

[0016] determining the azimuth angle information of the fault sound source in the electromechanical equipment based on the plane wave model and a first preset number of the time difference of arrival results, and combining an azimuth angle calculation formula.

[0017] Optionally, the obtaining of the sound source azimuth information of the fault sound source in the electromechanical equipment according to the time difference of arrival result comprises:

[0018] if the distance between the fault sound source signal and the microphone array is not greater than a preset far-near field critical value, determining that the sound source positioning model of the fault sound source in the electromechanical equipment is a spherical wave model;

[0019] determining the three-dimensional coordinate information of the fault sound source in the electromechanical equipment based on the spherical wave model and a second preset number of the time difference of arrival results, and combining the law of cosines.

[0020] Optionally, the fusing of the sound source azimuth information and the equipment abnormal area information in the original visual image of the electromechanical equipment to locate a fault of the electromechanical equipment comprises:

[0021] obtaining an original visual image of the electromechanical equipment, and performing defect recognition on the original visual image by using a target detection algorithm integrated with a visual Transformer model to obtain equipment abnormal area information in the original visual image;

[0022] fusing the sound source azimuth information and the equipment abnormal area information to locate a fault of the electromechanical equipment.

[0023] Optionally, the fusion of the sound source orientation information and the device abnormal area information is for fault positioning of the electromechanical device, and the fault positioning comprises:

[0024] The sound source orientation information and the device abnormal area information are matched in features after being spatio-temporally aligned by using a contrast learning algorithm, so as to obtain a fault positioning result.

[0025] Optionally, the fault positioning method based on fusion of a microphone array and visual perception further comprises:

[0026] A sound source distribution image is generated based on the sound source orientation information, the sound source distribution image is superimposed on the original visual image, a fault position is marked in the superimposed image based on the fault positioning result, a target device fault image is obtained, and the target device fault image is displayed on a preset display interface.

[0027] A target indicator light corresponding to the fault positioning result is selected from the indicator lights of the microphone array, and the target indicator light is controlled to display in a preset fault mode.

[0028] In a second aspect, the present application discloses a fault positioning device based on fusion of a microphone array and visual perception, comprising:

[0029] A sound source acquisition module is configured to control a microphone array to acquire an original sound source signal of an electromechanical device, wherein each microphone in the microphone array is located at a different orientation.

[0030] A sound source screening module is configured to match the original sound source signal with soundprint features in a mechanical fault soundprint sample library, so as to screen a fault sound source signal from the original sound source signal.

[0031] An orientation acquisition module is configured to acquire a time difference of arrival result between each microphone by using a GCC-PHAT algorithm and the fault sound source signal, and acquire sound source orientation information of a fault sound source in the electromechanical device according to the time difference of arrival result.

[0032] A fault positioning module is configured to fuse the sound source orientation information and device abnormal area information in an original visual image of the electromechanical device, so as to perform fault positioning of the electromechanical device.

[0033] In a third aspect, the present application discloses an electronic device, comprising:

[0034] A memory is configured to save a computer program.

[0035] A processor is configured to execute the computer program to implement the steps of the method for fault positioning based on fusion of microphone array and visual perception.

[0036] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program, wherein the computer program is configured to implement the steps of the method for fault positioning based on fusion of microphone array and visual perception when executed by a processor.

[0037] The present application has the following advantages: the present application controls a microphone array to collect original sound source signals of a mechanical and electrical equipment, wherein each microphone in the microphone array is located at a different direction; the original sound source signals are matched with soundprint features in a mechanical fault soundprint sample library to filter out fault sound source signals from the original sound source signals; a GCC-PHAT algorithm and the fault sound source signals are used to obtain time difference of arrival results between each microphone, and sound source direction information of a fault sound source in the mechanical and electrical equipment is obtained according to the time difference of arrival results; the sound source direction information is fused with equipment abnormal area information in an original visual image of the mechanical and electrical equipment to perform fault positioning on the mechanical and electrical equipment. As can be seen, the present application controls a microphone array at different directions to collect original sound source signals of a mechanical and electrical equipment, can capture acoustic information during equipment operation from multiple directions, ensures comprehensiveness and integrity of the original sound source signals, and provides reliable data basis for subsequent processing; the original sound source signals are matched with soundprint features in a mechanical fault soundprint sample library to filter out fault sound source signals, which can accurately distinguish normal running sound and fault sound, effectively exclude non-fault sound source interference, and improve pertinence and accuracy of fault signal identification; the GCC-PHAT algorithm and the fault sound source signals are used to obtain time difference of arrival between each microphone and obtain sound source direction information, which can improve accuracy of time difference calculation based on high-precision characteristics of the algorithm, and further improve accuracy of fault sound source direction judgment; the sound source direction information is fused with equipment abnormal area information in an original visual image of the mechanical and electrical equipment to perform fault positioning, which can combine acoustic and visual double features, realize complementary verification of multi-modal information, reduce error of single modal positioning, and finally improve accuracy and reliability of mechanical and electrical equipment fault positioning, and ensure that the fault position can be quickly and accurately locked. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0039] Figure 1A flow chart of a fault positioning method based on microphone array and visual perception fusion disclosed in the present application;

[0040] Figure 2 A specific microphone array schematic diagram disclosed in the present application;

[0041] Figure 3 A specific far-field model schematic diagram disclosed in the present application;

[0042] Figure 4 A specific near-field model schematic diagram disclosed in the present application;

[0043] Figure 5 A specific positioning system hardware structure schematic diagram disclosed in the present application;

[0044] Figure 6 A specific fault sound source positioning schematic diagram disclosed in the present application;

[0045] Figure 7 A specific fault positioning device structure schematic diagram based on microphone array and visual perception fusion disclosed in the present application;

[0046] Figure 8 A specific electronic device structure diagram disclosed in the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0048] As a core component in the fields of industrial production, power supply, rail transportation, etc., the stable operation of electromechanical equipment is directly related to production efficiency, operation safety and maintenance cost. Once a fault occurs, it may lead to serious consequences such as production line shutdown and safety accidents. Therefore, accurate fault positioning of electromechanical equipment is the key to ensuring its reliable operation.

[0049] Currently, fault positioning of electromechanical equipment mainly relies on manual inspection combined with simple instrument detection. In addition, in industrial environments, multi-source interferences such as electromagnetic noise and mechanical vibration are superimposed. The positioning error of traditional single-mode detection method is large, which is difficult to meet the operation and maintenance needs of precision equipment.

[0050] Therefore, the present application provides a fault positioning scheme based on microphone array and visual perception fusion to improve the accuracy of fault positioning of electromechanical equipment.

[0051] Reference is made to Figure 1As shown, the embodiment of the present application discloses a fault positioning method based on microphone array and visual perception fusion, comprising:

[0052] Step S11: control the microphone array to collect the original sound source signal of the electromechanical equipment; wherein each microphone in the microphone array is located at different positions.

[0053] In this embodiment, the microphone array includes a plurality of microphones located at different positions, for example Figure 2 As shown in a specific microphone array schematic diagram, the microphone array includes 7 MEMS microphones with a sensitivity of -26dB, wherein the microphone array module is designed based on MSM261S4030H0 digital microphone chip, and the module has high sound recognition sensitivity and high signal-to-noise ratio. It is composed of 6 microphones along the board and 1 central microphone. Further, the array board also includes 12 LEDs (light-emitting diodes), which can be used to visually identify the fault sound source position.

[0054] Control the microphone array (MicArray) to collect the original sound source signal of the electromechanical equipment, the sampling frequency is 16kHz, and the original sound source signal (including time-frequency domain features) is output.

[0055] Step S12: match the original sound source signal with the soundprint features in the mechanical fault soundprint sample library to filter out the fault sound source signal from the original sound source signal.

[0056] In this embodiment, the matching of the original sound source signal with the soundprint features in the mechanical fault soundprint sample library to filter out the fault sound source signal from the original sound source signal comprises: filtering the noise in the original sound source signal by using a deep noise suppression network to obtain a processed sound source signal; and matching the processed sound source signal with the soundprint features in the mechanical fault soundprint sample library to filter out the fault sound source signal from the original sound source signal.

[0057] Pre-control the microphone array to collect 20+ kinds of mechanical fault soundprints (such as bearing wear sound and gear abnormal sound), so as to obtain the soundprint features of various mechanical faults, and save the soundprint features in the mechanical fault soundprint sample library.

[0058] After the original sound source signal is collected, in order to make the subsequent feature matching more accurate, first, the noise in the original sound source signal is filtered, specifically, the noise in the original sound source signal is filtered by using a deep noise suppression network (Deep Noise Suppression Network, DNSN), the DNSN network filters the acoustic signal in a 120dB strong noise environment, and the signal-to-noise ratio is improved to obtain a processed sound source signal. Second, the processed sound source signal is matched with the soundprint features in the mechanical fault soundprint sample library to filter out the fault sound source signal from the original sound source signal. It can be understood that if there is no sound source signal matching the soundprint features in the processed sound source signal, it is determined from the sound angle that the electromechanical equipment has no fault, and if there is a sound source signal matching the soundprint features in the processed sound source signal, it is determined from the sound angle that the electromechanical equipment has a fault, which needs to be fused with the visual angle for subsequent fault positioning.

[0059] It can be understood that the processed sound source signal can include one or more fault sound source signals, for example, the processed sound source signal includes a fault sound source signal 1 matching the soundprint features of bearing wear sound, and the processed sound source signal includes a fault sound source signal 2 matching the soundprint features of gear abnormal sound.

[0060] Step S13: using the GCC-PHAT algorithm and the fault sound source signal to obtain the time difference of arrival result between each microphone, and obtaining the sound source position information of the fault sound source in the electromechanical equipment according to the time difference of arrival result.

[0061] It can be understood that the positions of the microphones are different, so the times when each microphone collects the fault sound source signal are also different, that is, the times when the fault sound source signal reaches each microphone are different. The time difference of arrival result between each microphone is obtained by using the GCC-PHAT (Generalized Cross-Correlation Phase Transform) algorithm and the fault sound source signal. The model of sound source positioning can be simply divided into near-field model and far-field model. If the microphone is close, it conforms to the near-field model, and if it is far away, it conforms to the far-field model.

[0062] In a specific far-field embodiment, the sound source position information of the fault sound source in the electromechanical equipment is obtained according to the time difference of arrival result, including: if the distance between the fault sound source signal and the microphone array is greater than a preset near-far field critical value, it is determined that the sound source positioning model of the fault sound source in the electromechanical equipment is a plane wave model; based on the plane wave model and the first preset number of time difference of arrival results, and combining the azimuth angle calculation formula to determine the azimuth angle information of the fault sound source in the electromechanical equipment.

[0063] If the distance between the fault sound source signal and the microphone array is greater than the preset far-near field critical value, it is determined that the sound source positioning model of the fault sound source in the electromechanical equipment is a plane wave model, i.e., a far field model, wherein the preset far-near field critical value is 0.61 times the wavelength of the sound wave , the azimuth angle information of the fault sound source in the electromechanical equipment is determined using the first preset number of time difference of arrival results and the plane wave model in combination with the azimuth angle calculation formula, for example Figure 3 , a specific far field model diagram is shown, the first preset number is 1, i.e., 1 time difference of arrival result is used, and the azimuth angle calculation formula is , wherein d represents the distance between the microphones, represents the propagation speed of the sound wave in the air, and a represents the azimuth angle of the fault sound source relative to the microphone, i.e., the azimuth angle information of the fault sound source in the electromechanical equipment is determined.

[0064] In a specific near field embodiment, the sound source azimuth information of the fault sound source in the electromechanical equipment is obtained according to the time difference of arrival results, including: if the distance between the fault sound source signal and the microphone array is not greater than the preset far-near field critical value, it is determined that the sound source positioning model of the fault sound source in the electromechanical equipment is a spherical wave model; based on the spherical wave model and the second preset number of time difference of arrival results, and in combination with the cosine law, the three-dimensional coordinate information of the fault sound source in the electromechanical equipment is determined.

[0065] If the distance between the fault sound source signal and the microphone array is not greater than the preset far-near field critical value, i.e., if the distance between the fault sound source signal and the microphone array is not greater than 0.61 times the wavelength of the sound wave (λ), it is determined that the sound source positioning model of the fault sound source in the electromechanical equipment is a spherical wave model, i.e., a near field model, based on the spherical wave model and the second preset number of time difference of arrival results, and in combination with the cosine law, the three-dimensional coordinate information of the fault sound source in the electromechanical equipment is determined, for example Figure 4 , a specific near field model diagram is shown, the second preset number is at least 2, i.e., at least 3 microphones are needed to locate the sound source, and the time difference of arrival results between the three microphones are

[0066] ;

[0067] ;

[0068] , wherein t 12 represents the time difference of arrival result between microphone 1 and microphone 2, and t 13represents the time difference of arrival between microphone 1 and microphone 3, r1, r2, r3 respectively represent the distance between microphone 1, microphone 2, microphone 3 and the fault sound source respectively, v represents the propagation speed of sound in air, according to the geometry of the microphone array and combined with the cosine theorem: r2 2 = r1 2 + d2 + 2r1d cos a1, r3 2 = r 12 + 4d2 + 4r1d cos a1, in this way, r1, r2, r3 can be obtained, and the three-dimensional coordinate information of the fault sound source can be determined.

[0069] Step S14: fuse the sound source orientation information and the equipment abnormal area information in the original visual image of the electromechanical equipment to locate the fault of the electromechanical equipment.

[0070] In the embodiment, the fusion of the sound source orientation information and the equipment abnormal area information in the original visual image of the electromechanical equipment to locate the fault of the electromechanical equipment comprises: acquiring the original visual image of the electromechanical equipment, and using a target detection algorithm of an integrated visual Transformer model to identify defects in the original visual image to obtain equipment abnormal area information in the original visual image; fuse the sound source orientation information and the equipment abnormal area information to locate the fault of the electromechanical equipment.

[0071] The original visual image of the electromechanical equipment is acquired by using an infrared camera. The infrared camera can be used to take a full view of the electromechanical equipment, that is, the acquired original visual image is an overall equipment image of the electromechanical equipment. Alternatively, the infrared camera can be used to take a partial view of the electromechanical equipment based on the sound source position information, that is, the acquired original visual image is a partial equipment image corresponding to the sound source position information. For example, the sound source position information indicates that the fault sound source is at (x1, y2, z3). In this case, the infrared camera is centered at (x1, y2, z3) to take a picture of the electromechanical equipment, thereby acquiring a partial equipment image corresponding to the sound source position information. In other words, after the sound source position is determined, the camera is focused on the position, so that the visual acquisition is more targeted, irrelevant areas are avoided, and the details of the equipment corresponding to the sound source are captured, such as surface cracks at the abnormal sound position of the gear box and thermal imaging features at the bearing wear position. Before the original visual image is subjected to defect recognition by using a target detection algorithm of an integrated Vision Transformer model (i.e., an improved YOLO-OBC model), the original visual image can be preprocessed, such as improving the contrast and reducing the noise in the original visual image. Then, the original visual image after the preprocessing is subjected to defect recognition by using the target detection algorithm of the integrated Vision Transformer model, so as to acquire the equipment abnormal area information in the original visual image, that is, to identify the position of the electromechanical equipment having a fault from the visual dimension, that is, the equipment abnormal area, to identify the abnormal area such as a crack and a leakage, and to output the Bounding Box (bounding box) and thermal map label (i.e., temperature abnormal area) of the equipment abnormal area, thereby obtaining the equipment abnormal area information, such as identifying the bearing surface crack area in the infrared thermal imaging image. Further, the type of equipment surface defect and the severity of the fault can be output.

[0072] In the embodiment, the sound source position information and the equipment abnormal area information are fused to locate the fault of the electromechanical equipment. The sound source position information and the equipment abnormal area information are matched in features after being spatio-temporally aligned by using a contrast learning algorithm, so as to locate the fault of the electromechanical equipment and obtain a fault locating result.

[0073] The sound source direction information of the sound dimension and the device abnormal area information of the visual dimension are fused to locate the fault of the electromechanical device. First, the space-time alignment is performed, and the alignment error between the sound source direction information (time sequence) and the device abnormal area information (spatial image) is controlled to be less than 3 ms by using a contrast learning algorithm. Then, the sound source coordinates (or azimuth) are associated with the visual abnormal area, and it is judged whether the sound source corresponds to the device defect, such as whether the abnormal sound of the gearbox matches the surface crack, so as to locate the fault of the electromechanical device and obtain the fault location result. Further, when the sound source direction information indicates that the A point position in the electromechanical device has a fault and the device abnormal area information indicates that the A point position also has a fault, it can be directly determined that the A point position has a fault. However, when the sound source direction information indicates that the A point position in the electromechanical device has a fault but the device abnormal area information indicates that the A point position does not have a fault, or the sound source direction information indicates that the A point position in the electromechanical device does not have a fault but the device abnormal area information indicates that the A point position has a fault, it can be determined that the A point position has a potential fault.

[0074] It can be understood that the fault location result can be a position where both the sound source direction information and the device abnormal area information indicate that there is a fault, or a position where there is a potential fault. Further, when the sound source direction information or the device abnormal area information indicates that the current position has a potential fault, the LSTM-Transformer model can be used to obtain the fault evolution trend of the current position in the electromechanical device based on the feature matching result, and output the fault location result of the current position in the electromechanical device. The fault location result includes the fault position, type, and risk level.

[0075] In the embodiment, the sound source distribution image is generated based on the sound source direction information, the sound source distribution image is superimposed on the original visual image, and the fault position is marked in the superimposed image based on the fault location result to obtain a target device fault image. The target device fault image is displayed on a preset display interface. The target indicator light corresponding to the fault location result is selected from the indicator lights of the microphone array, and the target indicator light is controlled to display in a preset fault mode.

[0076] The embodiment can also visualize the fault. The fault is visualized on a preset display interface: first, a sound source distribution image is generated based on the sound source direction information, then the sound source distribution image is superimposed on the original visual image, and the fault position is marked in the superimposed image based on the fault positioning result to obtain a target device fault image, and finally the target device fault image is displayed on the preset display interface. Wherein, the sound source image is generated according to the sound source distribution data obtained by the microphone array, the acoustic signal is collected by the microphone array, the direction, coordinate and intensity of the sound source are calculated, and then the fault sound source signal is converted into a visual sound source distribution image. Specifically, the sound source map is converted into a rainbow color. The data panel control indicator light displays according to the fault mode: the target indicator light corresponding to the fault positioning result is selected from the indicator light of the microphone array, wherein each indicator light is located at different directions, and the target indicator light is controlled to display according to the preset fault mode, that is, the LED color is set according to the sound source direction, such as left sound source bright red, right sound source bright green, which directly indicates the sound source direction. Further, the sound and light alarm can be triggered based on the fault positioning result, and the fault positioning result is pushed to the operation and maintenance platform to support automatic shutdown protection of the device (response time <0.3 seconds).

[0077] The beneficial effects of the present application are as follows: the present application controls a microphone array to collect original sound source signals of electromechanical equipment; wherein, each microphone in the microphone array is located at a different orientation; the original sound source signal is matched with the voiceprint features in a mechanical fault voiceprint sample library to filter out the fault sound source signal from the original sound source signal; the GCC-PHAT algorithm and the fault sound source signal are used to obtain the arrival time difference results between each of the microphones, and the sound source orientation information of the fault sound source in the electromechanical equipment is obtained based on the arrival time difference results; the sound source orientation information and the equipment abnormal area information in the original visual image of the electromechanical equipment are fused to locate the fault of the electromechanical equipment. It can be seen that the present application can capture the acoustic information of the equipment during operation in multiple directions by controlling the microphone arrays in different directions to collect the original sound source signals of the electromechanical equipment, ensure the comprehensiveness and integrity of the original sound source signals, and provide a reliable data basis for subsequent processing; by matching the original sound source signals with the voiceprint features in the mechanical fault voiceprint sample library to screen the fault sound source signals, it can accurately distinguish between normal operating sounds and fault sounds, effectively eliminate the interference of non-fault sound sources, and improve the pertinence and accuracy of fault signal identification; using the GCC-PHAT algorithm and the fault sound source signal to obtain the arrival time difference between each microphone and obtain the sound source direction information, it can improve the accuracy of the time difference calculation based on the high-precision characteristics of the algorithm, and thus improve the accuracy of the fault sound source direction judgment; the sound source direction information is integrated with the equipment abnormal area information in the original visual image of the electromechanical equipment for fault location, which can combine the dual characteristics of acoustics and vision to realize the complementary verification of multimodal information, reduce the error of single modal positioning, and ultimately improve the accuracy and reliability of electromechanical equipment fault location, ensuring that the fault location can be locked quickly and accurately.

[0078] Below Figure 5 The present application is described using the hardware structure diagram of a specific positioning system as an example. The fault positioning system based on the fusion of microphone array and visual perception includes a microphone array sound acquisition module, a camera visual acquisition module, an LED screen visual output module, and a core computing module.

[0079] The microphone array sound acquisition module is a highly sensitive audio capture device that can capture sound from multiple directions simultaneously. It is suitable for high-precision sound source positioning and sound tracking. In applications that require voice interaction, MicArray can provide clear audio input and support sound recognition and processing, enabling the product to understand and respond to user voice commands.

[0080] The core operation module can be a K210 development board, which has a computing power of 1TOPS (Tera Operations Per Second) and a power consumption of <5W. K210 is a system-on-chip (SoC) that integrates machine learning modules, making it particularly suitable for processing image recognition and neural network computing tasks. It supports multiple types of AI (Artificial Intelligence) models and can quickly process and analyze received data to provide intelligent decision support for products. The high performance and low power consumption design of the K210 development board make it an ideal choice for processing complex algorithms and running AI applications. K210 uses the RISC-V processor architecture, has three characteristics of audiovisual integration, autonomous IP kernel, and strong programmable ability, supports multi-modal recognition of machine vision and machine hearing, and can be widely used in smart home, smart park, smart energy consumption, and smart farm scenarios. It has a dual-core 64-bit processor with a total computing power of up to 1TOPS, built-in multiple hardware acceleration units, and good power performance, stability, and reliability. Hardware acceleration units such as KPU (Kernel Processing Unit), FPU (Floating Point Unit), and FFT (fast Fourier transform) are also available. Figure 6 As shown in a specific fault sound source positioning diagram, the microphone array sound acquisition module continuously and cyclically acquires the original sound source signal of the electromechanical equipment. The original sound source signal obtained by the core operation module is a black and white bitmap (imga). The fault sound source signal is filtered out, the sound source direction is determined, the angle and sound source intensity variables are initialized, the coordinates of the sound source are calculated, the angle and intensity are adjusted according to the coordinates, the RGB LED color value is set, the sound source coordinates, intensity, and angle are displayed on the LCD, the sound source map is obtained, the map is adjusted to a square, the sound source map is converted to a rainbow color, and the sound source map is displayed on the LCD.

[0081] The camera vision acquisition module is specifically an infrared camera with a resolution of 1920x1080. The camera is a key input device in the product, used to capture high-definition video and image data. In security monitoring, face recognition, or dynamic scene analysis applications, the camera sensor provides the necessary visual input, enabling the K210 development board to perform image recognition and other vision processing tasks.

[0082] The LED screen vision output module is specifically an LED screen. The LED screen compatible with the development board is used to display processing results and user interfaces, providing users with intuitive operation experience and real-time feedback information. The screen can display images, videos, and other graphical interface elements, making the product's interaction more friendly and intuitive.

[0083] The integration of these core hardware components enables the application to perform complex data processing tasks in various environments, including but not limited to voice and image recognition, real-time monitoring, interactive entertainment, and other application scenarios. In addition, the product design also considers scalability and modularity, and in the future, more functions can be added or the performance of existing functions can be improved according to demand.

[0084] From the system, the system can also be divided from the functional point of view, respectively including the main control unit, voice processing module, image monitoring module. Take K210 chip as the main control unit, use its integrated machine vision and machine hearing ability to cooperate with AI decision to realize real-time data processing. In the debugging process, through the pin configuration of K210 development board, system parameter adjustment and firmware upgrade, the main control module realizes the rapid acquisition and processing of multi-channel signals. Tests show that in a typical industrial environment, the system processing delay is controlled within 50ms, effectively meeting the real-time requirements. The voice processing module is based on a microphone array, which realizes high-precision positioning of noise sources in the industrial environment through the GCC-PHAT acoustic positioning algorithm. In actual testing, the system can detect the direction of the sound source within ±90 degrees, with an average positioning error of less than 1 degree. Through multi-point array data fusion and adaptive filtering algorithm, the system significantly improves the noise source identification accuracy, and realizes stable sound detection and direction judgment in noisy environment. The image monitoring module realizes real-time monitoring of key areas in the industrial environment through the camera and K210 vision processing unit. Combined with various image feature extraction algorithms, the image module can capture and analyze the real-time image of the noise source. In the debugging process, by adjusting the image acquisition frequency and optimizing the feature extraction parameters, the recognition accuracy of the system in high light and low light environments is maintained above 85%.

[0085] The system combines MALT technology with real-time linkage of the camera to realize multi-modal fusion of acoustic data and visual data. The noise source positioning data provided by MALT directly drives the camera to adjust the shooting direction and capture real-time pictures, thereby realizing the visualization of noise monitoring. Multi-modal data fusion not only improves the intuitiveness of the monitoring system, but also provides rich information support for subsequent analysis by archiving video and acoustic data of abnormal noise events. This combination enables the system to "hear" and "see" the noise source, greatly improving the monitoring efficiency and accuracy. The core technology builds a triple AI engine: first, based on the improved GCC-PHAT-TDOA algorithm combined with the deep noise suppression network (DNSN), it can still realize multi-source separation and positioning in a 120dB strong noise environment; second, the lightweight YOLO-OBC visual detection model is deployed, which is adapted to the thermodynamic characteristics of different devices through transfer learning; finally, the cross-modal spatio-temporal alignment Transformer (CMT) is innovatively developed to realize the intelligent association of the soundprint spectrum with the visual hot area, and to build a digital twin of the device health status. The system adopts an edge AI computing architecture and carries a neural network acceleration module, so that the inference delay is controlled within 8ms. It adopts modular design, has compact structure, and flexible installation mode, and can adapt to the needs of various industrial scenes. The device can be placed in indoor and outdoor places such as industrial parks, production lines, and workshops, and supports wall installation, ceiling hanging, or lamp pole fixation, etc. Through the cooperative work of multiple nodes, the system can cover a wide area and support wireless network connection to transmit data to the industrial management platform in real time. In addition, the system can be integrated with the existing monitoring system in the industrial scene to realize unified intelligent management and has good expansibility.

[0086] Referring to Figure 7 As shown in the drawings, the embodiments of the present application disclose a fault positioning device based on fusion of microphone array and visual perception, comprising:

[0087] The sound source acquisition module 11 is configured to control the microphone array to acquire the original sound source signal of the electromechanical equipment; wherein each microphone in the microphone array is located at a different direction.

[0088] The sound source screening module 12 is configured to match the original sound source signal with the soundprint features in the mechanical fault soundprint sample library, so as to screen out the fault sound source signal from the original sound source signal.

[0089] The direction acquisition module 13 is configured to acquire the time difference of arrival (TDOA) result between each microphone by using the GCC-PHAT algorithm and the fault sound source signal, and acquire the sound source direction information of the fault sound source in the electromechanical equipment according to the TDOA result.

[0090] The fault positioning module 14 is configured to fuse the sound source direction information and the equipment abnormal area information in the original visual image of the electromechanical equipment to perform fault positioning on the electromechanical equipment.

[0091] The application has the following beneficial effects: The application controls the microphone array to collect the original sound source signal of the electromechanical equipment, wherein each microphone in the microphone array is located at different directions; the original sound source signal is matched with the soundprint features in the mechanical fault soundprint sample library to screen out a fault sound source signal from the original sound source signal; the GCC-PHAT algorithm and the fault sound source signal are used to obtain the time difference of arrival results between each microphone, and the sound source direction information of the fault sound source in the electromechanical equipment is obtained according to the time difference of arrival results; and the sound source direction information is fused with the equipment abnormal area information in the original visual image of the electromechanical equipment to perform fault positioning on the electromechanical equipment. As can be seen, the application controls the microphone array at different directions to collect the original sound source signal of the electromechanical equipment, can capture acoustic information during equipment operation from multiple directions, ensures the comprehensiveness and integrity of the original sound source signal, and provides reliable data basis for subsequent processing; the original sound source signal is matched with the soundprint features in the mechanical fault soundprint sample library to screen out a fault sound source signal, which can accurately distinguish normal running sound and fault sound, effectively exclude non-fault sound source interference, and improve the pertinence and accuracy of fault signal identification; the GCC-PHAT algorithm and the fault sound source signal are used to obtain the time difference of arrival between each microphone and obtain the sound source direction information, which can improve the accuracy of time difference calculation based on the high-precision characteristics of the algorithm, and then improve the accuracy of fault sound source direction judgment; the sound source direction information is fused with the equipment abnormal area information in the original visual image of the electromechanical equipment to perform fault positioning, which can combine the dual characteristics of acoustics and vision, realize complementary verification of multi-modal information, reduce the error of single modal positioning, and finally improve the accuracy and reliability of electromechanical equipment fault positioning, so as to ensure that the fault position can be quickly and accurately locked.

[0092] Further, the embodiment of the application also provides an electronic device. Figure 8 The electronic device 20 shown in the figure is according to an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the application.

[0093] Figure 8 The structure of the electronic device provided by the embodiment of the application is shown in the figure. Specifically, it can include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21, and the related steps in the fault positioning method based on the fusion of the microphone array and visual perception performed by the electronic device disclosed in any of the preceding embodiments are implemented.

[0094] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device; the communication interface 24 is configured to create a data transmission channel between the electronic device and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which is not limited specifically herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which is not limited specifically herein.

[0095] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a GPU (Graphics Processing Unit) that is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 can also include an AI (Artificial Intelligence) processor configured to process machine learning-related computing operations.

[0096] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk, or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222, and data 223, etc. The storage mode can be temporary storage or permanent storage.

[0097] The operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device, so as to realize the operation and processing of the processor 21 on the mass data 223 in the memory 22, and can be Windows, Unix, Linux, etc. In addition to the computer program capable of completing the foregoing any embodiment disclosed by the electronic device to execute the fault positioning method based on the fusion of the microphone array and visual perception, the computer program 222 can further include a computer program capable of completing other specific work. The data 223 can include the data transmitted by the external device received by the electronic device, and can also include the data collected by the self input and output interface 25, etc.

[0098] Further, the application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by the processor to realize the foregoing disclosed fault positioning method based on the fusion of the microphone array and visual perception. For the specific steps of the method, refer to the corresponding contents disclosed in the foregoing embodiments, which will not be described here.

[0099] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between each embodiment, refer to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part is described in the method part.

[0100] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or any combination thereof. To clearly illustrate this interchangeability of hardware and software, various examples have been described herein in terms of their functionality, which has been described generally and symbolically in flow charts. Having thus described the functionality of the examples in terms of a process, it is appreciated that this functionality can be implemented by one or more types of electrical circuits or computer software, which are collectively referred to herein as a "circuit" that can carry out a variety of operations described herein. The circuit can include a variety of different types of general purpose or special purpose circuits, or combinations thereof. In addition, it is further noted that the embodiments disclosed herein can be modified to incorporate any or all of the optional features of the examples previously described, and these features can affect the description of the methods and the claims.

[0101] Finally, it should be noted that, in this document, the terms "comprises", "comprising", or other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. In addition, it should be noted that the terms "first", "second", or the like, are intended to distinguish between similar objects, items, or actions, but are not necessarily intended to convey a relative importance between the identified objects, items, or actions.

[0102] The above describes in detail the fault positioning method, device, equipment and medium based on microphone array and visual perception fusion provided by the present application. The principles and implementation manners of the present application are described by using specific examples. The above example is only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A fault location method based on the fusion of microphone array and visual perception, characterized in that: include: Controlling the microphone array to collect original sound source signals of the electromechanical device; wherein each microphone in the microphone array is located at a different position; Matching the original sound source signal with the voiceprint features in a mechanical fault voiceprint sample library to filter out the fault sound source signal from the original sound source signal; Obtaining arrival time difference results between the microphones using the GCC-PHAT algorithm and the fault sound source signal, and obtaining sound source orientation information of the fault sound source in the electromechanical device based on the arrival time difference results; The sound source orientation information is fused with the abnormal area information of the electromechanical device in the original visual image to locate the fault of the electromechanical device.

2. The fault location method based on microphone array and visual perception fusion according to claim 1 is characterized in that: The matching of the original sound source signal with the voiceprint features in the mechanical fault voiceprint sample library to filter out the fault sound source signal from the original sound source signal includes: Filtering the noise in the original sound source signal using a deep noise suppression network to obtain a processed sound source signal; The processed sound source signal is matched with the soundprint features in the mechanical fault soundprint sample library to filter out the fault sound source signal from the original sound source signal.

3. The fault location method based on microphone array and visual perception fusion according to claim 1 is characterized in that: The obtaining of the sound source orientation information of the fault sound source in the electromechanical equipment according to the arrival time difference result includes: If the distance between the fault sound source signal and the microphone array is greater than a preset near-far field threshold, determining that the sound source localization model of the fault sound source in the electromechanical device is a plane wave model; Based on the plane wave model and a first preset number of arrival time difference results, and in combination with an azimuth angle calculation formula, azimuth angle information of a fault sound source in the electromechanical equipment is determined.

4. The fault location method based on microphone array and visual perception fusion according to claim 1 is characterized in that: The obtaining of the sound source orientation information of the fault sound source in the electromechanical equipment according to the arrival time difference result includes: If the distance between the fault sound source signal and the microphone array is not greater than a preset near-far field threshold, determining that the sound source localization model of the fault sound source in the electromechanical device is a spherical wave model; Based on the spherical wave model and a second preset number of arrival time difference results, and in combination with the law of cosines, the three-dimensional coordinate information of the fault sound source in the electromechanical equipment is determined.

5. The fault location method based on the fusion of microphone array and visual perception according to any one of claims 1 to 4, characterized in that: The step of fusing the sound source orientation information with the abnormal area information of the electromechanical device in the original visual image to locate the fault of the electromechanical device includes: Acquire an original visual image of the electromechanical device, and perform defect recognition on the original visual image using an object detection algorithm integrated with a visual Transformer model to obtain information about abnormal areas of the device in the original visual image; The sound source orientation information and the equipment abnormal area information are integrated to locate the fault of the electromechanical equipment.

6. The fault location method based on microphone array and visual perception fusion according to claim 5 is characterized in that: The fusing of the sound source orientation information and the equipment abnormal area information to locate the fault of the electromechanical equipment includes: After the sound source orientation information and the equipment abnormal area information are aligned in time and space using a contrastive learning algorithm, feature matching is performed on the sound source orientation information and the equipment abnormal area information to locate the fault of the electromechanical equipment and obtain a fault location result.

7. The fault location method based on microphone array and visual perception fusion according to claim 6 is characterized in that: Also includes: generating a sound source distribution image based on the sound source orientation information, superimposing the sound source distribution image with the original visual image, marking the fault location in the superimposed image based on the fault location result to obtain a target device fault image, and displaying the target device fault image in a preset display interface; A target indicator light corresponding to the fault location result is selected from the indicator lights of the microphone array, and the target indicator light is controlled to display according to a preset fault mode.

8. A fault location device based on the fusion of microphone array and visual perception, characterized in that: include: A sound source acquisition module, used to control the microphone array to acquire the original sound source signal of the electromechanical device; wherein each microphone in the microphone array is located at a different position; A sound source screening module, configured to match the original sound source signal with the voiceprint features in a mechanical fault voiceprint sample library to screen out the fault sound source signal from the original sound source signal; a direction acquisition module, configured to obtain arrival time difference results between the microphones using the GCC-PHAT algorithm and the fault sound source signal, and obtain sound source direction information of the fault sound source in the electromechanical device based on the arrival time difference results; The fault location module is used to fuse the sound source orientation information with the abnormal area information of the device in the original visual image of the electromechanical device to locate the fault of the electromechanical device.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the steps of the fault location method based on the fusion of microphone array and visual perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the fault location method based on the fusion of microphone array and visual perception as described in any one of claims 1 to 7 are implemented.

Images