Array type multi-channel acoustic imaging device and abnormal sound source positioning display method

Abstract

The invention discloses an array type multichannel acoustic imaging device and an abnormal sound source positioning display method, and the device comprises a main microphone array which is used for collecting sound signals of a target region; the camera device is used for collecting an image of the target area; the thermal imaging device is used for collecting a thermal image of the target area; the first data processing and analyzing module is used for identifying an abnormal sound signal of the target area and preliminarily estimating the position of an abnormal sound source; acquiring sound field distribution of the abnormal sound source and generating a sound field image by using an acoustic imaging algorithm; the second data processing and analyzing module is used for inputting the change rate of the sound wave parameters of the unit area and the temperature into a second neural network model, obtaining the sound velocity of the unit area in the sound field image, calculating the temperature of the unit area, fusing the sound velocity of the unit area with the temperature of the corresponding unit area in thermal imaging and determining the abnormal temperature; and the display screen is used for displaying the sound field image marked with the abnormal temperature. The abnormal sound source can be effectively distinguished and positioned, and the abnormal temperature can be displayed.

Description

Technical Field

[0001] The present application relates to the technical field of acoustic imaging and sound source localization and display, and in particular to an array-type multi-channel acoustic imaging device and an abnormal sound source localization and display method. Background Art

[0002] In the field of power inspection and fire safety, traditional detection methods mainly rely on manual visual and auditory judgment. Although these methods can meet basic needs to a certain extent, their limitations in complex environments are becoming increasingly prominent. Especially in working environments with high noise and limited vision, traditional methods are difficult to accurately detect hidden dangers, which seriously affects work efficiency and operator safety. With the development of technology, acoustic imaging technology has gradually become an important tool for improving work safety and efficiency, especially for application scenarios that require real-time monitoring and rapid response.

[0003] In order to meet the above challenges, the industry generally uses a single sensor to collect sound signals, but a single sensor cannot form spatial positioning and lacks an intuitive sound source distribution image, which makes it difficult to accurately locate the sound source. In a high-intensity noise environment, a single sensor is easily interfered, resulting in data distortion, reducing the reliability and accuracy of the system. In addition to using a single sensor for positioning, existing acoustic detection also sets up a multi-channel microphone array to collect and locate sound signals, and will simultaneously use thermal imaging instruments to synchronously display the temperature of the detection target to further display more information about abnormal sound sources to facilitate users to complete subsequent maintenance processing. However, in complex scenarios, the accuracy of such abnormal sound source positioning is still insufficient, and the temperature measurement method is relatively single. The temperature measurement results are often not accurate enough, making it difficult for users to obtain accurate information for maintenance processing.

[0004] Therefore, how to effectively distinguish and locate abnormal sound sources in complex environments and display accurate temperature information of abnormal sound sources to improve the efficiency and safety of acoustic detection in practical applications. Summary of the invention

[0005] In order to effectively locate abnormal sound sources and display abnormal sound sources and their temperatures in complex environments, and to improve the accuracy of positioning and temperature measurement, the present application provides an array-type multi-channel acoustic imaging device and an abnormal sound source positioning and display method.

[0006] In a first aspect, the present application provides an array-type multi-channel acoustic imaging device, comprising: An imaging display device, a main microphone array mounted on the imaging display device, a camera device, and a thermal imaging device; The main microphone array is used to collect sound signals from the target area and transmit them to the imaging display device; A camera device, used to collect images of the target area and transmit them to an imaging display device; A thermal imaging device, used to collect thermal images of the target area and transmit them to an imaging display device; The imaging display device includes: a display screen, a first data processing and analysis module and a second data processing and analysis module that are connected: the first data processing and analysis module is used to use a first neural network model to identify abnormal sound signals in a target area, and preliminarily estimate the position of the abnormal sound source through a spatial positioning algorithm; the first neural network model is generated by training and marking abnormal sound signals; the acoustic imaging algorithm is used to obtain the sound field distribution of the preliminarily estimated abnormal sound source, and a sound field image is generated in combination with the collected target area image; the second data processing and analysis module is used to obtain the rate of change of the acoustic wave parameter per unit area in the sound field image; the obtained rate of change of the acoustic wave parameter per unit area is generated; The rate of change of the acoustic wave parameters and the temperature per unit area obtained from the thermal image are input into the second neural network model to obtain the sound velocity per unit area in the sound field image; the second neural network model is generated by training the rate of change of the acoustic wave parameters per unit area in the sound field image of the historical target area, the sound velocity and the temperature per unit area in the thermal image of the historical target area; the temperature per unit area in the sound field image is calculated according to the relationship equation between the sound velocity and the gas temperature, and the final temperature is obtained by fusing it with the temperature per unit area in the thermal image; the final temperature is compared with the preset temperature, an abnormal temperature prompt is generated and annotated on the sound field image, and the sound field image annotated with the abnormal temperature is displayed on the display screen.

[0007] By adopting the above scheme, an array-type multi-channel microphone array is used to collect sound signals, and a neural network model is used to identify abnormal sounds, obtain the initial positioning of the abnormal sound source, and then the acoustic imaging algorithm is used to enhance the collected sound signals to obtain a more accurate sound field image of the abnormal sound source; the temperature of the target area is obtained through a thermal imaging device, and the relationship between the propagation speed of sound waves in the air and the temperature is used to estimate the sound speed at each position in the sound field by measuring the changes in sound field parameters, and the temperature distribution of the target area is indirectly calculated. The two temperature data are fused to obtain a more accurate temperature and display the abnormal temperature in a timely manner.

[0008] Preferably, the first data processing and analysis module is also used to obtain the relative position of each microphone in the main microphone array; according to the preliminary estimated abnormal sound source position, calculate the distance to each microphone in the main microphone array respectively, and cluster the calculated distances; count the number of clusters and compare the average size of each type of distance set, and match the weight values ​​in the same number of preset weight combinations in the order of the average size of each type of distance set as the weighting factors of each microphone in the acoustic imaging process; the preset weight combination has a preset number of weight values ​​whose sum is 1, and the weight values ​​are arranged in descending order.

[0009] By adopting the above scheme, the weight ratio is determined according to the distance between different microphones and the initially estimated abnormal sound source position, thereby completing the signal enhancement in the direction of the initially estimated abnormal sound source, so as to generate a more accurate sound field image.

[0010] Preferably, it also includes: a signal transmitting device installed on the imaging display device; A signal transmitting device is used to transmit test signals of different frequencies; a main microphone array is also used to receive the test signal and output a frequency response; The first data processing and analysis module is also used to analyze the frequency response curve of each microphone to test signals of different frequencies, and to generate a sensitivity adjustment coefficient for each microphone to respond to test signals of different frequencies; in the process of obtaining a preliminary estimate of the sound field distribution of the abnormal sound source using the acoustic imaging algorithm, the main frequency band of the collected abnormal signal is analyzed, and the corresponding sensitivity adjustment coefficient is multiplied according to the frequency of the main frequency band.

[0011] By adopting the above scheme, taking into account the different responsiveness of the microphones in the main microphone array to signals of different frequencies, a test signal is used to determine the responsiveness of the microphones in the main microphone array to signals of different frequencies to generate a sensitivity adjustment coefficient, so that in the actual audio processing process, the signal of each microphone is adjusted accordingly according to its sensitivity adjustment coefficient to obtain a better signal response effect.

[0012] Preferably, the first data processing and analysis module is also used to perform cluster analysis on the abnormal sound signals in the identified target area, compare each type of abnormal signal with the abnormal signals in the preset abnormal signal information library, determine the priority of each type of abnormal signal, generate a corresponding sound field image separately for each type of preliminary estimated abnormal sound source, and generate a recommended viewing order of the sound field images according to the priority order and transmit it to the display screen; the display screen, which is an interactive display screen, is also used to receive and display the sound field image of a type of preliminary estimated abnormal sound source selected by the user according to the recommended viewing order of the sound field image.

[0013] By adopting the above scheme, considering the existence of multiple types of abnormal sound sources, a sound field image corresponding to each type of abnormal sound source is generated and a viewing recommendation order is generated based on the priority of the abnormal signal. Combined with an interactive display screen, users can flexibly adjust the type of abnormal sound source they are concerned about according to actual needs, thereby improving work efficiency and safety.

[0014] Preferably, it also includes: a noise reduction microphone array installed on the imaging display device; A noise reduction microphone array is used to collect noise signals and transmit them to an imaging display device; The first data processing and analysis module is also used to receive the noise signal collected by each microphone in the noise reduction microphone array, use digital signal processing technology to analyze the frequency spectrum, amplitude and phase of the noise signal, and generate a corresponding anti-phase sound wave signal with a phase opposite to the noise signal and matching amplitude; before performing abnormal signal recognition, for each microphone in the main microphone array, select a microphone in the noise reduction microphone array closest to it for microphone matching, and fuse the anti-phase sound wave signal generated by a microphone in the matched noise reduction microphone array with the sound signal collected by a microphone array in the corresponding main microphone array to obtain the sound signal after fusion processing of each microphone in the main microphone array.

[0015] By adopting the above solution, an additional set of noise reduction microphones are configured to capture and analyze background noise. Through digital signal processing technology, sound waves opposite to the background noise are actively generated to achieve noise cancellation at the acoustic level, thereby purifying the original collected audio signal and enhancing the anti-interference performance of the microphone array in noisy environments. This ensures that high-quality audio information can still be obtained in strong noise environments, thereby improving the robustness and accuracy of acoustic imaging.

[0016] Preferably, the first data processing and analysis module is also used to count the number of abnormal sound sources preliminarily estimated, and calculate the density of the abnormal sound sources in combination with the size of the image of the target area; determine the preset density threshold range within which the calculated density is located, and match the preset ratio according to the judgment result, and the density in each preset density threshold range corresponds to a preset ratio; determine the unit area according to the matched preset ratio and the size of the image of the target area.

[0017] By adopting the above solution, the density of abnormal sound sources in the target area is taken into consideration, and the display of temperature and sound field parameters is adjusted based on the density, so that the user can clearly view the information of the abnormal sound sources.

[0018] Preferably, the first data processing and analysis module is also used to perform spectral analysis on the identified abnormal sound signal to determine whether the main frequency band of the sound signal originating from the direction of the initially estimated abnormal sound source belongs to the high frequency band, the medium frequency band or the low frequency band; when it is determined to belong to the high frequency band or the medium frequency band, a delay-sum beamforming algorithm is selected as the acoustic imaging algorithm; when it is determined to belong to the low frequency band, a deconvolution beamforming algorithm is selected as the acoustic imaging algorithm.

[0019] By adopting the above scheme, the frequency of abnormal sound sources in the target area is taken into consideration, and an acoustic imaging algorithm adapted to the sound source frequency is selected to generate a more accurate sound field image.

[0020] In a second aspect, the present application provides a method for locating and displaying an abnormal sound source, comprising: The main microphone array is used to collect the sound signal of the target area and transmit it to the imaging display device; the camera device is used to collect the image of the target area and transmit it to the imaging display device; the thermal imaging device is used to collect the thermal image of the target area and transmit it to the imaging display device; Using the first neural network model in the first data processing and analysis module, the abnormal sound signal in the target area is identified, and the location of the abnormal sound source is preliminarily estimated through a spatial positioning algorithm; The second data processing and analysis module is used to obtain the rate of change of the acoustic wave parameter per unit area in the acoustic field image; the obtained rate of change of the acoustic wave parameter per unit area and the temperature per unit area obtained from the thermal image are input into the second neural network model to obtain the speed of sound per unit area in the acoustic field image; The temperature per unit area in the acoustic field image is calculated according to the relationship equation between the speed of sound and the gas temperature, and the final temperature is obtained by fusing it with the temperature per unit area in the thermal image. The final temperature is compared with the preset temperature, an abnormal temperature prompt is generated and marked on the sound field image, and the sound field image marked with the abnormal temperature is displayed on the display screen.

[0021] By adopting the above scheme, a microphone array is used to collect multi-channel sound signals and a neural network model is used to identify abnormal signals. The location of the abnormal sound source is preliminarily estimated. On this basis, acoustic imaging technology is used to accurately obtain the sound field image. At the same time, a comprehensive temperature measurement method of thermal imaging and acoustic temperature measurement is used to obtain a more accurate temperature and timely determine and display the abnormal temperature, so that users can view the relevant information of the abnormal sound source.

[0022] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method as described above.

[0023] In a fourth aspect, the present application provides a computer device, comprising a memory, a processor, and a program stored and executable on the memory, wherein the program implements the steps of the above method when executed by the processor.

[0024] In summary, this application has the following beneficial effects: 1. The main microphone array is used to collect multi-channel sound signals and the neural network model is used to identify abnormal signals. The microphone position and positioning algorithm of the main microphone array are combined to realize the preliminary positioning of abnormal sound sources. On this basis, the acoustic imaging technology is used to enhance the signal of the abnormal sound source direction derived from the preliminary estimation, and a more accurate sound field image of the abnormal sound source is generated. At the same time, thermal imaging technology and acoustic temperature measurement technology are used to obtain multi-source fused temperature data, obtain more accurate temperature and display abnormal temperature, so that users can accurately obtain and view relevant information about abnormal sound sources. 2. According to the calculated distance between the main microphone and the preliminarily estimated abnormal sound source, the weights of different microphones are dynamically adjusted to enhance the signal in the preliminarily estimated direction of the abnormal sound source and generate a more accurate sound field image; 3. Use the test signal to determine the responsiveness of the microphones in the main microphone array to signals of different frequencies to generate a sensitivity adjustment coefficient, so that in the actual audio processing process, the signal of each microphone can be adjusted accordingly according to the sensitivity adjustment coefficient, so as to collect the signal more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural schematic diagram of the device described in a specific embodiment; Figure 2 Flowchart of the method described in the specific embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] like Figure 1 As shown, an embodiment of the present application discloses an array-type multi-channel acoustic imaging device, comprising: an imaging display device 1, a main microphone array 2 installed on the imaging display device 1, a camera device 3 and a thermal imaging device 4.

[0028] The main microphone array 2 includes a plurality of microphones evenly distributed in a specific shape, such as 16 microphones distributed in a rectangular, square or multi-arm spiral shape, etc. The main microphone array 2 is installed on the imaging display device 1 and transmits data with the imaging display device 1 through a connection interface, so as to collect sound signals in the target area and transmit them to the imaging display device 1.

[0029] The camera device 3 is connected to the imaging display device 1 via an interface, and is used to capture an image of the target area and transmit it to the imaging display device 1 .

[0030] The thermal imaging device 4 is connected to the imaging display device 1 via an interface, and is used to collect thermal images of the target area and transmit them to the imaging display device 1 .

[0031] The imaging display device 1 comprises: a display screen 11, a first data processing and analysis module 12 and a second data processing and analysis module 13 for establishing a connection; The first data processing and analysis module 12 is used to use the first neural network model to identify abnormal sound signals in the target area, and preliminarily estimate the position of the abnormal sound source through a spatial positioning algorithm; use an acoustic imaging algorithm to obtain the sound field distribution of the preliminarily estimated abnormal sound source, and generate a sound field image in combination with the collected target area image.

[0032] Specifically, the first neural network model is generated by training with annotated abnormal sound signals, the input of the model is the sound signal of the target area collected by each microphone, and the output is the recognition result of whether the sound signal of the target area collected by each microphone is abnormal. Taking power line fault detection as an example, the first neural network model is generated by training with a data set consisting of sound signals that historically contain and annotate typical power line faults. In order to more quickly complete the abnormal signals in the sound signals collected by multiple microphones, the sound signals collected by different microphones can be input into multiple first neural network models respectively, so as to respectively identify and obtain the abnormal signals in the sound signals from different directions received by each microphone itself.

[0033] Combined with the spatial positioning algorithm, the relevant collection information of the abnormal signal collected by each microphone is obtained to preliminarily estimate the location of the abnormal sound source; such as the TDOA algorithm, which uses the arrival time difference to perform the positioning algorithm to obtain the location of the abnormal sound source, including the formula:

[0034] The time when the abnormal signal reaches the two microphones in the main microphone array is t1 and t2 respectively, and (i, j) represents the position of the microphone in the main microphone array; C is the propagation speed of the signal; is the distance between the abnormal signal and the two microphones; combined with the calculated , the precise location of the signal source is calculated through triangulation.

[0035] Acoustic imaging algorithms are used to obtain a preliminary estimate of the sound field distribution of abnormal sound sources. Common acoustic imaging algorithms include beamforming algorithms and planar near-field acoustic holography algorithms. Beamforming algorithms include delayed sum beamforming algorithms, cross-spectral beamforming algorithms, and deconvolution beamforming algorithms. When selecting a specific acoustic imaging algorithm to obtain a preliminary estimate of the sound field distribution of abnormal sound sources, it is necessary to consider the near field and far field in advance. The judgment formula is:

[0036] In the formula, To preliminarily estimate the reference distance between the abnormal sound source and the main microphone array; is the main microphone array aperture, is the wavelength of the sound signal; when the distance between the main microphone array and the initially estimated abnormal sound source is greater than the reference distance , it is considered as near field, otherwise it is considered as far field.

[0037] For the near field, you can choose to use delayed sum beamforming algorithm, planar near-field acoustic holography algorithm and other acoustic imaging algorithms to obtain the preliminary estimated sound field related parameter distribution of the abnormal sound source. For example, establish a spatial coordinate system with the geometric center of the main microphone array as the origin. Assuming that the main microphone array has M channels, the coordinates of each microphone are , the initial estimated abnormal sound source (considering that there may be multiple initial estimated abnormal sound sources The plane is the abnormal sound source plane, and the coordinates of the N focal points on the abnormal sound source plane are defined as , according to the geometric relationship of the main microphone array , then the focus The response expression of the beamformer at is:

[0038] In the formula, Is the focus The beamforming response at is the weighting factor of the mth microphone, is the sound pressure value of the mth microphone, c is the propagation speed of the sound wave, which can be converted into the frequency domain as follows:

[0039] In the formula, Is the focus The frequency domain response of the near-field delayed-sum beamforming at is the sound pressure vector received by the mth microphone.

[0040] Solve the time domain or frequency domain responses of the N focal points of the plane where the abnormal sound source is preliminarily estimated as the focusing plane, obtain the sound pressure values ​​of the N focal points of the plane where the abnormal sound source is preliminarily estimated as the focusing plane, draw the sound pressure value distribution of the plane where the abnormal sound source is preliminarily estimated, and generate the sound field image in combination with the collected target area image.

[0041] Alternatively, for the far field, you can choose to use an acoustic imaging algorithm such as a delayed sum beamforming algorithm to obtain a preliminary estimate of the sound field related parameter distribution of the abnormal sound source. For example, establish a spatial coordinate system with the geometric center of the main microphone array as the origin. Assuming that the main microphone is channel M, the coordinates of the microphone are , preliminarily estimate the abnormal sound source (considering that there may be multiple preliminarily estimated abnormal sound sources ) is the plane where the sound source is located. The main microphone array receives the plane wave emitted from each initially estimated abnormal sound source direction. The output of the delay-sum beamformer is:

[0042] In the formula, is the output of the far-field delayed-sum beamforming, is a unit vector, and its direction is opposite to the propagation direction of the plane wave. is the weighting factor of the mth microphone, is the sound pressure value of the mth microphone, c is the propagation speed of the sound wave, which can be converted into the frequency domain as follows:

[0043] In the formula, is the frequency domain response of the far-field delay-sum beamforming, The sound pressure vector received by the mth microphone.

[0044] Solve the time domain or frequency domain response of the N focal points of the plane where the abnormal sound source is preliminarily estimated as the focusing plane, obtain the sound power of the N focal points of the plane where the abnormal sound source is preliminarily estimated as the focusing plane, draw the sound power distribution of the plane where the abnormal sound source is preliminarily estimated, and generate the sound field image in combination with the collected target area image.

[0045] The second data processing and analysis module 13 is used to obtain the rate of change of the sound wave parameters per unit area in the sound field image; input the obtained rate of change of the sound wave parameters per unit area and the temperature of the corresponding unit area obtained according to the thermal image into the second neural network model to obtain the sound speed per unit area in the sound field image; calculate the temperature per unit area in the sound field image according to the relationship equation between the sound speed and the gas temperature, and fuse it with the temperature of the corresponding unit area in the thermal image to obtain the final temperature; compare the final temperature with the preset temperature, generate an abnormal temperature and mark it on the sound field image.

[0046] Specifically, the change rate of the acoustic wave parameters per unit area in the acoustic field image is obtained; wherein the unit area of ​​the acoustic field image can be set according to the image size of the shooting target area, and the change rate formula of the acoustic wave parameters is:

[0047] In the formula, is the maximum acoustic parameter per unit area, is the minimum acoustic wave parameter per unit area, For unit area.

[0048] The acquired rate of change of the acoustic wave parameters per unit area and the temperature per unit area acquired according to the thermal image are input into a second neural network model to acquire the sound velocity per unit area in the sound field image; wherein, the second neural network model is generated by training the rate of change of the acoustic wave parameters per unit area in the sound field image of the historical target area, the sound velocity (average sound velocity) and the temperature per unit area in the thermal image of the historical target area, the input of the model is the rate of change of the acoustic wave parameters per unit area in the sound field image and the temperature (average temperature) per unit area in the thermal image, and the output is the sound velocity per unit area in the sound field image as the estimated sound velocity per unit area in the sound field image; wherein, since the acquired sound field image and thermal image are both images of the same target area (the background image has the same size) and have the same unit area setting, it is determined that the unit area in the sound field image has the corresponding unit area in the thermal image.

[0049] The temperature per unit area in the sound field image is calculated based on the relationship equation between the speed of sound and the gas temperature. The formula is:

[0050] In the formula, is the speed of sound; is the gas adiabatic index; is the ideal gas constant; is the thermodynamic temperature; in a specific medium, Z is a constant; in air, the value of Z is 20.045. The temperature per unit area in the acoustic field image obtained by calculation is fused with the average temperature per unit area in the thermal imaging data, such as obtaining the average of the two temperature values ​​per unit area as the final temperature.

[0051] The final temperature is compared with the preset temperature. If it exceeds the preset temperature, it is considered that there is an abnormal temperature. An abnormal temperature prompt is generated and marked at the corresponding unit area position of the sound field image.

[0052] The display screen 11 is used to display the sound field image marked with abnormal temperature.

[0053] A specific embodiment takes into account that in the process of obtaining the sound field distribution of the preliminarily estimated abnormal sound source by using an acoustic imaging algorithm, the signal toward the preliminarily estimated abnormal sound source is enhanced by beamforming, in order to further ensure that the beam points to the direction of the abnormal sound source so as to subsequently generate a more stable sound field map.

[0054] The first data processing and analysis module 12 is also used to obtain the relative position of each microphone in the main microphone array; according to the preliminary estimated abnormal sound source position, calculate the distance to each microphone in the main microphone array respectively, and cluster the calculated distances; count the number of clusters and compare the average size of each type of distance set after clustering, and match the weight values ​​in the same number of preset weight combinations in the order of the average size of each type of distance set as the weighting factors of each microphone in the acoustic imaging process; the preset weight combination has a preset number of weight values ​​whose sum is 1, and the weight values ​​are arranged in descending order.

[0055] Specifically, a coordinate axis is established with the center of the microphone in the main microphone array as the origin, and the position information corresponding to each microphone is determined according to the relative position of each microphone in the main microphone array. The initially estimated position of the abnormal sound source is located within the established coordinate axis and in the same plane.

[0056] If there are multiple abnormal sound sources that are preliminarily estimated, for each preliminarily estimated abnormal sound source, calculate the distance between the abnormal sound source position and each microphone in the main microphone array, and cluster them based on the calculated distance; count the number of clusters and compare the average size of each type of distance set, and match the weight values ​​in the same number of preset weight combinations matched according to the number of clusters in the order of the average size of each type of distance set. For example: the number of clusters is 3, the average sizes of the three types of distance sets are ranked as 1.12m, 1.32m and 1.57m, and the weight values ​​of the preset weight combinations are 0.45, 0.35 and 0.2, corresponding to the weight factor of several microphones corresponding to the distance set with an average size of 1.12m, which is 0.45, corresponding to the formula: Substitute the formula to calculate the sound field parameter distribution corresponding to the abnormal sound source, and then complete the generation of the subsequent sound field map to obtain a more accurate sound field map.

[0057] In a specific embodiment, considering that each microphone in the main microphone array has different response capabilities to signals of different frequencies, some microphones have better response to high-frequency and high-intensity signals, and poorer response to medium-low-frequency and low-intensity signals, in order to ensure that the abnormal sound source signal in the target area can be well responded to, the adjustment coefficient is set to accurately define and display the abnormal sound source, and the system further includes: A signal transmitting device 5 installed on the imaging display device 1; The signal transmitting device 5 is used to transmit test signals of different frequencies. In order to better implement the test, the signal transmitting device and the main microphone array are both set to be rotatable. When they are rotated to face each other and the entire device is in the initial startup stage, test signals of different frequencies are transmitted.

[0058] The main microphone array 2 is also used to receive the test signal and output the frequency response; The first data processing and analysis module 12 is also used to analyze the frequency response curve of each microphone to test signals of different frequencies. The horizontal axis of the frequency response curve represents the sound frequency, ranging from low frequency (such as 20Hz) to high frequency (such as 20kHz), and the vertical axis shows the gain or attenuation of the microphone to the corresponding frequency, and generates a sensitivity adjustment coefficient for each microphone to respond to test signals of different frequencies, that is, a multiplication factor is set, and the multiplication factor is dynamically adjusted along with the gain or attenuation of the frequency of the test signal response of different frequencies, so that the output test signal response of different frequencies is within a preset frequency response value (such as 1-1.1); in the process of obtaining a preliminary estimate of the sound field distribution of the abnormal sound source by using the acoustic imaging algorithm, the main frequency band of the collected abnormal signal is analyzed, and the sensitivity adjustment coefficient is multiplied according to the frequency correspondence of the main frequency band, so as to generate an accurate sound field map based on a more accurate abnormal signal.

[0059] In addition, an environmental sensor 6 installed on the imaging display device 1 can also be set to collect environmental data; the first data processing and analysis module 12 is also used to analyze the frequency response curve of each microphone to test signals of different frequencies under different environmental data conditions, and generate a sensitivity adjustment coefficient for each microphone to test signals of different frequencies under different environmental conditions; in the process of obtaining a preliminary estimate of the sound field distribution of the abnormal sound source using the acoustic imaging algorithm, the main frequency band of the collected abnormal signal is analyzed according to the currently obtained environmental data, and the corresponding sensitivity adjustment coefficient under the corresponding environmental data and frequency conditions is multiplied, so as to generate an accurate sound field map based on a more accurate abnormal signal.

[0060] In a specific embodiment, considering that the positioning processing of abnormal sound sources in actual power lines or fire rescue application scenarios often requires processing of a single abnormal sound source, in order to meet the needs of users, the system further includes: The first data processing and analysis module 12 is further used to perform cluster analysis on the abnormal sound signals in the identified target area, compare each type of abnormal signal with the abnormal signals in the preset abnormal signal information library, and determine the priority of each type of abnormal signal; wherein the preset abnormal signal information library stores typical abnormal signals and their corresponding priorities, such as: abnormal signals generated by arc discharge in the power line, which have a higher priority than abnormal signals generated by other typical faults in the power line; generate corresponding sound field images for each type of initially estimated abnormal sound source, such as: generate sound field images for abnormal sound sources such as abnormal signals generated by arc discharge; generate a recommended viewing order of the sound field images according to the priority order and transmit them to the display screen; The display screen 11 is an interactive display screen, and is also used to receive a type of preliminary estimated abnormal sound source selected by the user according to the recommended viewing order of the sound field image to generate a sound field image and display the sound field image corresponding to the corresponding type of preliminary estimated abnormal sound source.

[0061] In a specific embodiment, considering that noise is often mixed in the process of collecting abnormal signals, which affects the recognition of the final abnormal signals, thereby causing errors in the positioning of abnormal sound sources, a noise reduction microphone array and an advanced signal processing algorithm are introduced to further improve the anti-interference ability in a strong noise environment. The system also includes: a noise reduction microphone array 7 installed on the imaging display device; The noise reduction microphone array 7 is used to collect noise signals and transmit them to the imaging display device; wherein the noise reduction microphone array 7 is composed of a plurality of small microphones, such as 8 microphones evenly distributed around the periphery of the main microphone array.

[0062] The first data processing and analysis module 12 is also used to receive the noise signal collected by each microphone in the noise reduction microphone array 7, analyze the spectrum, amplitude and phase of the noise signal using digital signal processing technology, and generate an anti-phase sound wave signal with a phase opposite to the noise signal and matching the amplitude; wherein, after receiving the noise signal received by each microphone in the noise reduction microphone array, the noise signal is pre-processed to extract the environmental noise signal; Before abnormal signal recognition, considering that the noise signals collected by microphones at similar positions are the most similar, for each microphone (i, j) in the main microphone array, a microphone in the noise reduction microphone array closest to it is selected. Microphone matching is performed, and the anti-phase sound wave signal generated by a corresponding microphone in the matched noise reduction microphone array is fused with the sound signal collected by a corresponding microphone array in the corresponding main microphone array to obtain the sound signal after fusion processing of each microphone in the main microphone array, that is, to obtain the sound signal after noise cancellation.

[0063] In a specific embodiment, considering that there may be relatively dense abnormal sound sources in the target area, the distribution of the corresponding generated sound field related parameters should not be relatively dense, and the size of the unit area is adjusted accordingly to better display the sound field parameters of the abnormal sound sources. The system also includes: The first data processing and analysis module 12 is further used to count the number of abnormal sound sources preliminarily estimated, and calculate the density of the abnormal sound sources in combination with the size of the image of the target area; determine the preset density threshold range in which the calculated density is located, and according to the determination result, the specific preset threshold range type (such as the first preset threshold range) corresponds to a matching preset ratio (such as the first preset ratio), and the density in each preset density threshold range corresponds to a matching preset ratio; and determine the unit area according to the matching preset ratio and the size of the image of the target area.

[0064] In a specific embodiment, considering that different acoustic imaging technologies have different accuracy of acoustic images generated under different sound source frequencies, in order to obtain a more accurate acoustic image, the system further includes: The first data processing and analysis module 12 is further used to perform spectrum analysis on the identified abnormal sound signal to determine whether the main frequency band of the sound signal originating from the direction of the initially estimated abnormal sound source belongs to the high frequency band, the medium frequency band or the low frequency band; when it is determined to belong to the high frequency band or the medium frequency band, the delay-sum beamforming algorithm is selected as the acoustic imaging algorithm; when it is determined to belong to the low frequency band, the deconvolution beamforming algorithm is selected as the acoustic imaging algorithm.

[0065] like Figure 2 As shown, the embodiment of the present application discloses a method for locating and displaying an abnormal sound source using the above array multi-channel acoustic imaging device, and the specific steps include: S1. Turn on and initialize the array multi-channel acoustic imaging device.

[0066] S2. Use the main microphone array to collect sound signals in the target area and transmit them to the imaging display device.

[0067] S3. Use a camera to capture an image of the target area and transmit it to an imaging display device.

[0068] S4. Collect the thermal image of the target area using the thermal imaging device and transmit it to the imaging display device.

[0069] S5. Using the first neural network model in the first data processing and analysis module, identify abnormal sound signals in the target area, and preliminarily estimate the location of the abnormal sound source through a spatial positioning algorithm.

[0070] S6. Use the second data processing and analysis module to obtain the rate of change of the acoustic wave parameters per unit area in the sound field image; input the obtained rate of change of the acoustic wave parameters per unit area and the temperature per unit area obtained from the thermal image into the second neural network model to obtain the sound velocity per unit area in the sound field image.

[0071] S7. Calculate the temperature per unit area in the sound field image according to the relationship equation between the speed of sound and the gas temperature, and fuse it with the temperature per unit area in the thermal image to obtain the final temperature. Compare the final temperature with the preset temperature, generate an abnormal temperature prompt and mark it on the sound field image.

[0072] S8. Displaying the sound field image with abnormal temperature marked on the display screen.

[0073] The embodiment of the present application also discloses a computer-readable storage medium.

[0074] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute the abnormal sound source localization display method as described above. The computer-readable storage medium includes, for example, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0075] The embodiment of the present application also discloses a computer device.

[0076] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above abnormal sound source localization and display method.

[0077] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. An array-type multi-channel acoustic imaging device, characterized in that: include: An imaging display device, a main microphone array mounted on the imaging display device, a camera device, and a thermal imaging device; The main microphone array is used to collect sound signals from the target area and transmit them to the imaging display device; A camera device, used to collect images of the target area and transmit them to an imaging display device; A thermal imaging device, used to collect thermal images of the target area and transmit them to an imaging display device; The imaging display device includes: a display screen, a first data processing and analysis module and a second data processing and analysis module that are connected to each other: the first data processing and analysis module is used to use a first neural network model to identify abnormal sound signals in a target area and preliminarily estimate the location of the abnormal sound source through a spatial positioning algorithm; the first neural network model is generated by training and marking abnormal sound signals; The acoustic imaging algorithm is used to obtain the sound field distribution of the abnormal sound source that is preliminarily estimated, and the sound field image is generated in combination with the collected target area image; the second data processing and analysis module is used to obtain the rate of change of the sound wave parameters per unit area in the sound field image; the obtained rate of change of the sound wave parameters per unit area and the temperature per unit area obtained according to the thermal image are input into the second neural network model to obtain the sound speed per unit area in the sound field image; the second neural network model is generated by training the rate of change of the sound wave parameters per unit area in the sound field image of the historical target area, the sound speed and the temperature per unit area in the thermal image of the historical target area; the temperature per unit area in the sound field image is calculated according to the relationship equation between the sound speed and the gas temperature, and the final temperature is obtained by fusing it with the temperature per unit area in the thermal image; the final temperature is compared with the preset temperature, an abnormal temperature prompt is generated and annotated on the sound field image, and the sound field image annotated with the abnormal temperature is displayed on the display screen.

2. The array-type multi-channel acoustic imaging device according to claim 1, characterized in that: The first data processing and analysis module is further used to obtain the relative position of each microphone in the main microphone array; calculate the distance to each microphone in the main microphone array according to the preliminary estimated abnormal sound source position, and perform clustering according to the calculated distance; The number of clusters is counted and the average size of each type of distance set is compared, and the weight values ​​in the same number of preset weight combinations are matched in order of the average size of each type of distance set as the weighting factors of each microphone in the acoustic imaging process; the preset weight combination has a preset number of weight values ​​whose sum is 1, and the weight values ​​are arranged in descending order.

3. The array-type multi-channel acoustic imaging device according to claim 1, characterized in that: Also includes: A signal transmitting device installed on the imaging display device; A signal transmitting device, used for transmitting test signals of different frequencies; The main microphone array is also used to receive the test signal and output the frequency response; The first data processing and analysis module is also used to analyze the frequency response curve of each microphone to test signals of different frequencies, and to generate a sensitivity adjustment coefficient for each microphone to respond to test signals of different frequencies; in the process of obtaining a preliminary estimate of the sound field distribution of the abnormal sound source using the acoustic imaging algorithm, the main frequency band of the collected abnormal signal is analyzed, and the corresponding sensitivity adjustment coefficient is multiplied according to the frequency of the main frequency band.

4. The array-type multi-channel acoustic imaging device according to claim 1, characterized in that: The first data processing and analysis module is further used to perform cluster analysis on the abnormal sound signals of the identified target area, compare each type of abnormal signal with the abnormal signals in the preset abnormal signal information library, determine the priority of each type of abnormal signal, generate a corresponding sound field image for each type of initially estimated abnormal sound source, and generate a recommended viewing order of the sound field image according to the priority order and transmit it to the display screen; The display screen is an interactive display screen, and is also used to receive and display a type of sound field image of a preliminary estimated abnormal sound source selected by the user according to the recommended viewing order of the sound field image.

5. The array-type multi-channel acoustic imaging device according to claim 1, characterized in that: Also includes: A noise-cancelling microphone array mounted on an imaging display device; A noise reduction microphone array is used to collect noise signals and transmit them to an imaging display device; The first data processing and analysis module is also used to receive the noise signal collected by each microphone in the noise reduction microphone array, use digital signal processing technology to analyze the frequency spectrum, amplitude and phase of the noise signal, and generate a corresponding anti-phase sound wave signal with a phase opposite to the noise signal and matching amplitude; before performing abnormal signal recognition, for each microphone in the main microphone array, select a microphone in the noise reduction microphone array closest to it for microphone matching, and fuse the anti-phase sound wave signal generated by a microphone in the matched noise reduction microphone array with the sound signal collected by a microphone array in the corresponding main microphone array to obtain the sound signal after fusion processing of each microphone in the main microphone array.

6. The array-type multi-channel acoustic imaging device according to claim 1, characterized in that: The first data processing and analysis module is also used to count the number of abnormal sound sources preliminarily estimated, and calculate the density of the abnormal sound sources in combination with the size of the image of the target area; determine the preset density threshold range within which the calculated density is located, and match the preset ratio according to the judgment result, and the density in each preset density threshold range corresponds to a preset ratio; determine the unit area according to the matched preset ratio and the size of the image of the target area.

7. The array-type multi-channel acoustic imaging device according to claim 1, characterized in that: The first data processing and analysis module is also used to perform spectrum analysis on the identified abnormal sound signal to determine whether the main frequency band of the sound signal originating from the direction of the initially estimated abnormal sound source belongs to the high frequency band, the medium frequency band or the low frequency band; when it is determined to belong to the high frequency band or the medium frequency band, the delay-sum beamforming algorithm is selected as the acoustic imaging algorithm; when it is determined to belong to the low frequency band, the deconvolution beamforming algorithm is selected as the acoustic imaging algorithm.

8. A method for locating and displaying abnormal sound sources using the array multi-channel acoustic imaging device according to any one of claims 1 to 7, characterized in that: include: The main microphone array is used to collect the sound signal of the target area and transmit it to the imaging display device; Using a camera device to collect images of the target area and transmit them to an imaging display device; Using a thermal imaging device to collect a thermal image of the target area and transmit it to an imaging display device; Using the first neural network model in the first data processing and analysis module, the abnormal sound signal in the target area is identified, and the location of the abnormal sound source is preliminarily estimated through a spatial positioning algorithm; The second data processing and analysis module is used to obtain the rate of change of the acoustic wave parameter per unit area in the acoustic field image; the obtained rate of change of the acoustic wave parameter per unit area and the temperature per unit area obtained from the thermal image are input into the second neural network model to obtain the speed of sound per unit area in the acoustic field image; The temperature per unit area in the acoustic field image is calculated according to the relationship equation between the sound velocity and the gas temperature, and the final temperature is obtained by fusing it with the temperature per unit area in the thermal image. The final temperature is compared with the preset temperature, an abnormal temperature prompt is generated and marked on the sound field image, and the sound field image marked with the abnormal temperature is displayed on the display screen.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to claim 8.

10. A computer device, characterized in that: The computer device comprises a memory, a processor and a program stored and executable on the memory, and the program implements the steps of the method according to claim 8 when executed by the processor.

Images