Sound source imaging calculation method and system, electronic equipment, storage medium and computer program product

Abstract

The invention provides a sound source imaging calculation method and system, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: carrying out the preprocessing of obtained audio data, so as to obtain a center frequency in a scanning frequency range, and the audio data are collected through a microphone array; sound intensity grids are divided within the visible range of the video collection device. And determining an Euclidean distance matrix and an array steering vector from the microphone array to the sound intensity grid. And obtaining an output power spectrum matrix according to the array steering vector and the cross-spectrum matrix. And coloring the sound intensity grids to obtain a thermodynamic diagram. The sound source imaging method is optimized and innovated by utilizing the parallel computing capability of the CUDA technology, the accurate positioning of the sound source is realized, the processing speed and the positioning precision of the CUDA architecture model are improved, and powerful technical support is provided for the development of the sound source imaging field.

Description

Technical Field

[0001] The present application relates to the technical field of sound source imaging, and in particular to a sound source imaging calculation method, system, electronic device, storage medium and computer program product. Background Art

[0002] At present, acoustic imaging technology converts sound signals into image signals to visualize the distribution of spatial sound sources. Multi-channel audio data is collected through a microphone array, and the above audio data at a specified frequency is calculated through a beamforming delay sum algorithm to obtain the sound source distribution information on the plane where the microphone array is located, and the sound source distribution information is fused with the real-life image. The spatial position and source of the sound source are determined on the fused image.

[0003] Sound source imaging technology has been widely used in transportation, noise detection, industrial anomaly detection and other fields. Sound source imaging technology can directly and accurately reflect the location and intensity of the sound source, and provide relevant technical support for noise control, fault diagnosis and other aspects.

[0004] The inventors found that the conventional beamforming delay summation algorithm directly performs delay summation processing on the collected audio data, uses beamforming and time difference to invert the sound source distribution, and outputs an image-based result. Since the multi-channel microphone array will generate self-noise, the accuracy of sound source identification is reduced. At the same time, there are a large number of matrix convolution operations in the beamforming delay summation algorithm, and the traditional central processing unit architecture mainly performs serial calculations, resulting in problems such as slow calculation speed and long processing time, and cannot meet real-time requirements. Summary of the invention

[0005] The present application provides a sound source imaging calculation method, system, electronic device and storage medium, which are used to solve the problems of long serial calculation time, complex calculation model and difficulty in meeting real-time requirements of traditional central processing units.

[0006] According to one aspect of the present application, the present application provides a method, comprising: preprocessing the acquired audio data to obtain a center frequency within a scanning frequency range, wherein the audio data is collected by a microphone array. Dividing a sound intensity grid within a visible range of a video collection device. Determining a Euclidean distance matrix and an array steering vector from the microphone array to the sound intensity grid. Obtaining an output power spectrum matrix based on the array steering vector and the cross-spectral matrix. Obtaining a heat map by coloring the sound intensity grid.

[0007] According to some embodiments of the present application, before the above-mentioned step of preprocessing the acquired audio data to obtain the center frequency within the scanning frequency range, the method also includes: setting the video collection device at the center of the microphone array and establishing a spatial rectangular coordinate system with the center of the microphone array as the origin.

[0008] According to some embodiments of the present application, the above-mentioned step of preprocessing the acquired audio data to obtain the center frequency within the scanning frequency range may include: dividing the acquired audio data into frames according to a preset frame length to obtain framed audio data; multiplying each frame of the framed audio data with a preset window function to obtain windowed audio data; converting the windowed audio data into frequency domain audio data through discrete Fourier transform; scanning the amplitude of the frequency domain audio data within the scanning frequency range to obtain the center frequency.

[0009] According to some embodiments of the present application, the above-mentioned step of dividing the sound intensity grid within the visible range of the video collection device may include: selecting a rectangular area parallel to the plane of the microphone array in a preset direction of the microphone array; dividing the rectangular area into a preset number of grid points to obtain a sound intensity grid.

[0010] According to some embodiments of the present application, the above-mentioned steps of determining the Euclidean distance matrix and array steering vector from the microphone array to the sound intensity grid may include: calculating the Euclidean distance between the microphone array elements and the grid points according to the Euclidean distance formula to generate a Euclidean distance matrix; and calculating in parallel the array steering vector at each frequency point within the first frequency range through the CUDA architecture model based on the Euclidean distance matrix and the wavenumber matrix.

[0011] According to some embodiments of the present application, the preprocessed frequency domain audio data is processed in parallel through a CUDA architecture model, and the cross-spectral matrix is ​​determined based on a beamforming delayed sum algorithm.

[0012] According to some embodiments of the present application, the above-mentioned step of obtaining the output power spectrum matrix based on the array steering vector and the cross-spectrum matrix may include: based on the array steering vector and the cross-spectrum matrix, parallelly calculating the output power of each grid point through the CUDA architecture model to generate the output power spectrum matrix.

[0013] According to some embodiments of the present application, the step of obtaining a heat map by coloring the sound intensity grid may include: coloring each grid point of the sound intensity grid according to the output power size of the output power spectrum matrix to generate a heat map.

[0014] According to some embodiments of the present application, the acquired video data is overlaid with the heat map to obtain a real-time video stream; wherein the video data is collected by a video collection device.

[0015] According to one aspect of the present application, the present application also provides a system, which includes a data acquisition module, a data processing module, a matrix calculation module and an image processing module. The data acquisition module is based on a video collection device and a microphone array, and the data acquisition module collects video data and audio data simultaneously and in real time. The data processing module is used to pre-process the acquired audio data; and divide the sound intensity grid within the visible range of the video collection device. The matrix calculation module is used to determine the Euclidean distance matrix and array steering vector from the microphone array to the sound intensity grid, and obtain the output power spectrum matrix according to the array steering vector and the cross-spectral matrix. The image processing module is used to obtain a thermal map by coloring the sound intensity grid.

[0016] According to one aspect of the present application, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors can implement the high-speed calculation method for sound source imaging as described above.

[0017] According to one aspect of the present application, a non-volatile computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the high-speed calculation method for sound source imaging as described above can be implemented.

[0018] According to one aspect of the present application, a computer program product is also provided, comprising: a computer program stored on a computer-readable storage medium; the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the high-speed calculation method for sound source imaging as described above.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] The present application realizes parallel processing of audio data and improves the processing speed and accuracy of sound source imaging through GPU (Graphics Processing Unit) based on CUDA (Compute Unified Device Architecture). BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram showing a flow chart of a sound source imaging calculation method 1 according to an embodiment of the present application;

[0023] Figure 2 A schematic flow chart showing step S10 in the sound source imaging calculation method 1 according to an embodiment of the present application;

[0024] Figure 3 A schematic flow chart showing step S30 in step 1 of the sound source imaging calculation method according to an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a sound source imaging computing system 2 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0027] The described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of these specific details, or other modes, components, materials, devices, etc. may be adopted. In these cases, known structures, methods, devices, implementations, materials or operations will not be shown or described in detail.

[0028] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0029] The terms "first", "second" and the like in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order.

[0030] The following is a clear and complete description of the technical solution of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] This application can be implemented on any GPU device that supports the CUDA architecture. To meet the requirements of large-scale matrix-vector multiplication, it is recommended that the video memory capacity be above 2GB to reduce the delay caused by data exchange.

[0032] According to one aspect of the present application, the present application provides a sound source imaging calculation method, which can be executed by a computer system.

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0034] Figure 1 A schematic flow chart of a sound source imaging calculation method according to the present application is shown.

[0035] Reference Figure 1 , the flowchart includes steps S10-S60. Exemplarily, the method may be executed by a computer system.

[0036] In step S10, the computer system pre-processes the acquired audio data to obtain a center frequency within a scanning frequency range, wherein the audio data is collected by a microphone array.

[0037] Optionally, before step S10, the method further comprises setting a video collection device at the center of the microphone array, and establishing a spatial rectangular coordinate system with the center of the microphone array as the origin. The video collection device is a visible light camera.

[0038] The visible light camera is usually set at the center of the microphone, but can also be set at other locations. Since the subsequent steps require superimposition of the audio data and the generated heat map, the coincidence of the origins of the two coordinate systems can reduce the amount of calculation for the superposition process. The location of the sound source can be directly observed through the visible light camera, which is conducive to the subsequent generation of visible light images.

[0039] For example, a microphone array is a system composed of a certain number of microphones that samples and filters the spatial characteristics of a sound field. The microphone array uses a group of multi-channel microphones arranged in a certain geometric structure to obtain audio data, and obtains the spatial coordinate information of one or more sound sources by processing and analyzing the obtained multi-channel audio data.

[0040] According to the exemplary embodiment, the microphone array can be divided into: linear array, planar array, and stereo array according to certain geometric structure shapes. In this embodiment, the microphone array is described by taking a planar array as an example. All microphone array elements in the microphone array have the same frequency response and clock cycle.

[0041] Specifically, the microphone array consists of 64 miniature digital microphones arranged in a spiral shape on the same plane. With the center of the microphone array as the origin, the plane where the microphone array is located as the xoy plane, and the front of the visible light camera as the positive direction of the z-axis, a spatial rectangular coordinate system is established; and the coordinate value of each microphone array element is accurately measured. Among them, the front of the visible light camera is the direction where the visible light camera lens is located.

[0042] According to the exemplary embodiment, the microphone array element is a miniature digital microphone. The more microphone array elements there are, the better the positioning effect is. Generally, 64 or 128 microphone array elements are used to form a multi-channel microphone array. The spiral arrangement makes the main lobe more prominent and more suitable for sound source imaging. The number of miniature digital microphones can be set according to the needs of those skilled in the art.

[0043] The above-mentioned multi-channel microphone array and visible light camera respectively collect external voice and video data in real time. Among them, data collection is to convert the acquired audio signal into an electrical signal through the sensor, and send it to the computer system for processing, analysis, storage and display through steps such as signal modulation, sampling, quantization, encoding and transmission.

[0044] According to the example embodiment, since the audio data acquired by the microphone array is a baseband signal that has not been modulated, the above data needs to be preprocessed after the audio data is acquired. Among them, the phase difference of the audio signal collected by the microphone array is determined by the characteristics of the sound source and the microphone array structure. The frequency of the audio signal is generally between 100-4000Hz, and the center frequency changes with the change of the sound source.

[0045] Preprocessing of a continuously collected multi-channel audio data mainly includes framing, windowing and discrete Fourier transform.

[0046] like Figure 2 As shown, step S10 may further include steps S11-S14.

[0047] In step S11, the computer system divides the acquired audio data into frames according to a preset frame length to obtain framed audio data.

[0048] In audio data processing, framing is to divide a continuous signal into multiple segments according to a preset frame length, each segment is called a frame. Within each frame of framed audio data, the signal is stable.

[0049] For example, in order to make a smooth transition between frames, the frame processing generally adopts an overlapping segmentation method, so that there is a certain time overlap between the previous frame and the next frame. The overlapping part is called frame shift; wherein, the frame shift has a certain proportional relationship with the frame length, and generally the value range of the frame shift is less than or equal to 1 / 2 of the frame length; usually the audio frame length range is 10 to 40ms.

[0050] For example, take a segment of continuously sampled audio data and perform frame processing. Assuming that the frame shift is 1 / 2 of the frame length, the first frame contains 1 to 10 sampling points, the next frame contains 5-15 sampling points, and so on, thus obtaining a three-dimensional matrix of M*N*K, where M is the number of frames, N is the number of sampling points in each frame, and K is the number of channels (i.e., the number of microphone array elements).

[0051] According to the exemplary embodiment, the continuous audio data is framed and converted into discrete data blocks, which is beneficial for the subsequent steps to independently process and analyze the framed audio data, such as noise reduction, audio compression, filtering, and frequency domain analysis. The continuity of the signal can be maintained by the overlapping segmentation method, so that each frame has a smooth transition.

[0052] In step S12, the computer system multiplies each frame of the framed audio data by a preset window function to obtain windowed audio data.

[0053] Since the signal is non-periodically truncated by the frame processing, the signal period is not necessarily an integer period, which causes the spectrum to tail in the entire frequency band, resulting in a certain error, called leakage. In order to prevent the time domain truncation effect from causing spectrum leakage, windowing is required before discrete Fourier transform.

[0054] Exemplarily, the windowing process is performed in the time domain to reduce the slope at both ends of the time window so that the two ends of the window edge do not cause a sharp change but smoothly transition to zero, thereby slowly reducing the signal waveform at the frame interception point to zero, thereby reducing the frame truncation effect.

[0055] In step S13, the computer system converts the windowed audio data into frequency domain audio data by discrete Fourier transform.

[0056] According to an exemplary embodiment, the windowed audio data is a time domain signal, and the computer system performs a discrete Fourier transform on the time domain signal to obtain a frequency domain signal.

[0057] Specifically, the discrete Fourier transform can be FFT (Fast Fourier Transform) or DFT (Discrete Fourier Transform). Both FFT and DFT are used to process discrete signals, but FFT simplifies the calculation amount of DFT, shortens the calculation time of discrete Fourier transform, and improves the processing speed.

[0058] In step S14, the computer system scans the amplitude of the frequency domain audio data within the scanning frequency range to obtain the center frequency.

[0059] According to the exemplary embodiment, the amplitude within the frequency range is scanned to obtain the maximum amplitude within the frequency range. The frequency point corresponding to the maximum amplitude is used as the center frequency. The scanning frequency range can be set according to the needs of those skilled in the art, and is not specifically limited in this embodiment.

[0060] By preprocessing the acquired audio data, the noise in the audio data is reduced, and the subsequent processing and calculation of the audio data is facilitated. After obtaining the center frequency, a frequency range with the center frequency as the center and a preset bandwidth as the interval is selected for subsequent matrix calculation, thereby narrowing the calculation range, reducing the redundancy and calculation time of the calculation, and improving the model processing speed.

[0061] In step S20, the computer system divides the sound intensity into grids within the visible range of the video collection device.

[0062] According to an example embodiment, the computer system selects a rectangular area parallel to the plane of the microphone array in a preset direction of the microphone array; the computer system divides the rectangular area into a preset number of grid points to obtain a sound intensity grid.

[0063] The video collection device is a visible light camera. Different models of visible light cameras have different visual ranges, which are not specifically limited in this embodiment.

[0064] Specifically, the preset direction is the positive direction of the Z axis in the above-mentioned spatial rectangular coordinate system; the microphone array plane is the xoy plane in the spatial rectangular coordinate system; the preset number of grid points is M*N grid points, where M and N are positive integers greater than 0.

[0065] In this embodiment, the sound intensity grid is divided within the visible range of the visible light camera. The denser the sound intensity grid is, the higher the accuracy of sound source positioning is, and the greater the amount of calculation is. Therefore, the density of the sound intensity grid division is set according to the needs of technical personnel in this field, and this embodiment does not impose any specific restrictions.

[0066] In step S30 , the computer system determines a Euclidean distance matrix and an array steering vector from the microphone array to the sound intensity grid.

[0067] like Figure 3 As shown, step S30 may also include steps S31-S33.

[0068] In step S31 , the computer system calculates the Euclidean distances between the microphone array elements and the grid points according to the Euclidean distance formula to generate a Euclidean distance matrix.

[0069] The coordinate position of each microphone array element and the coordinate position of each grid point divided by the rectangular area are marked in the spatial rectangular coordinate system established in step S10. Then, the Euclidean distance between the microphone array element and the grid point is calculated according to the Euclidean distance formula.

[0070] Exemplarily, assuming M grid points and N microphone array elements, a dist[M][N] Euclidean distance matrix is ​​formed, where dist[m][n] represents the Euclidean distance from the mth grid point to the nth microphone array element.

[0071] In step S32, the computer system generates a wave number matrix according to the determined sound wave numbers per unit time at different frequency points under the preset medium sound speed.

[0072] Specifically, the wave number formula is used to calculate the number of sound waves per unit time at each frequency point within a frequency range with the center frequency as the center and the preset bandwidth as the interval.

[0073] The wave number formula is k = 2*π*f / c;

[0074] Where f is a one-dimensional array [f1,f2,f3,...,f n ] represents frequency, f1,f2,f3,...,f n represents the frequency points within the frequency range with the center frequency as the center and the preset bandwidth as the interval; c is the speed of sound, which is a constant; k is a one-dimensional array [k1, k2, k3, ..., k n ] represents the number of sound waves, k1, k2, k3, ..., k n Indicates the number of sound waves per unit time at different frequency points.

[0075] The wave number matrix is ​​calculated according to the above wave number formula.

[0076] For example, since the sound speeds in different media are different, the medium is selected according to the needs of those skilled in the art, and no specific limitation is made in this embodiment.

[0077] In step S33, the computer system calculates in parallel the array steering vector at each frequency point within the first preset frequency range based on the Euclidean distance matrix and the wavenumber matrix through the CUDA architecture model.

[0078] Specifically, the first preset frequency range is a frequency range centered on the center frequency and having a preset bandwidth as an interval. There is an array steering vector at each frequency point in the first preset frequency range; wherein the array steering vector is an M*N matrix, where M is the number of grid points and N is the number of microphone array elements.

[0079] The Euclidean distance matrix dist[m][n] obtained in step S31 and the wave number matrix obtained in step S42 are substituted into formula 1 to calculate the array steering vector at the current frequency in parallel through the CUDA architecture model.

[0080] Formula 1: steerVec[m][n] = cos(expArg) – j*sin(expArg);

[0081] Where expArg is the wave number matrix at the current frequency multiplied by the Euclidean distance matrix; steerVec[m][n] is the array steering vector at the current frequency.

[0082] By selecting the first preset frequency range, the computational redundancy of the CUDA architecture model is simplified, and the model processing speed is improved. The array steering vector at each frequency point is calculated in parallel through the CUDA architecture model according to the Euclidean distance matrix and the wave number matrix, thereby improving the accuracy of sound source positioning.

[0083] Optionally, after step S10, the computer system processes the preprocessed frequency domain audio data in parallel through a CUDA architecture model, and determines the cross-spectral matrix based on a beamforming delayed-sum algorithm.

[0084] Based on the frequency domain audio data preprocessed in step S10, a CSM (Cross-Spectral Matrix) matrix is ​​calculated in parallel based on a beamforming delay-sum algorithm through a CUDA architecture model.

[0085] The CSM matrix is ​​an M*N*N three-dimensional matrix, where M is the number of frequency points and N is the number of microphone array elements.

[0086] Exemplarily, an N*N two-dimensional matrix in a CSM matrix is ​​taken as an example for explanation. The CSM matrix has a conjugate property, and all the main diagonal elements of the CSM matrix are zero.

[0087] The lower triangular elements of the CSM matrix are calculated according to Formula 2;

[0088] Formula 2 is: csm[i][j] = ∑f(m,j).conj*f(m,k), k = 0, 1, 2, .., j-1;

[0089] Wherein, f(m,j) represents the value of the discrete Fourier transform result of the j-th microphone array element at the m-th frequency point, and conj represents the conjugate characteristic of the matrix, where the upper triangular elements are conjugate symmetric with the lower triangular elements.

[0090] In this embodiment, the CSM (Cross-Spectral Matrix) matrix has the characteristics that the main diagonal elements are 0 and the upper and lower triangular elements are conjugate symmetric. This characteristic can simplify calculations, reduce calculation time, and improve the parallel computing speed of the CUDA architecture model.

[0091] In step S40, the computer system obtains an output power spectrum matrix according to the array steering vector and the cross-spectrum matrix.

[0092] Optionally, the computer system calculates the output power of each grid point in parallel through a CUDA architecture model according to the array steering vector and the cross-spectrum matrix to generate an output power spectrum matrix.

[0093] The output power matrix is ​​a one-dimensional array whose length is the number of grid points. The value of each component of the array represents the output power value of each grid point. The maximum value of the array is obtained by traversing the entire array. The grid point corresponding to the maximum value is the location of the sound source.

[0094] The output power at the nth grid point is calculated as follows:

[0095]

[0096] Wherein, N is the number of microphone array elements, conj represents the conjugate characteristic, and real represents taking the real part.

[0097] Since the sound intensity grid contains M*N grid points, and each grid point contains multiple frequency points, the values ​​of multiple frequency points are added together as the output power of the grid point. The output powers of all grid points in the sound intensity grid form an output power matrix.

[0098] In this embodiment, the parallel computing speed of the CUDA architecture model is greatly improved through the graphics processor, which improves the model processing speed and positioning accuracy. At the same time, the CSM matrix is ​​calculated based on the beamforming delay sum algorithm; the interference of channel self-noise is effectively suppressed, and the accuracy of sound source identification is improved. The positioning effect in poor environments is also more accurate, with good robustness and applicability.

[0099] In step S50 , the computer system obtains a thermal map by coloring the sound intensity grid.

[0100] According to the output power spectrum matrix obtained in step S40, each grid point of the sound intensity grid is colored according to the output power to generate a thermal map.

[0101] For example, a heat map is a visual data method for displaying matrix or grid data, which uses color to represent the size of data values. The output power spectrum matrix is ​​a two-dimensional matrix. Each cell of the heat map represents a data point. The color of the cell is colored according to the output power value of the output power spectrum matrix. Gradient colors are usually used to represent the size of different values. For example, values ​​with smaller power in the output power spectrum matrix are represented by lighter colors, and values ​​with larger power are represented by darker colors.

[0102] The present application realizes parallel processing of audio data and improves the processing speed and accuracy of sound source imaging through GPU (Graphics Processing Unit) based on CUDA (Compute Unified Device Architecture).

[0103] Optionally, after step S50, the computer system performs superposition processing on the acquired video data and the heat map to obtain a real-time video stream; wherein the video data is collected by a video collection device.

[0104] The heat map obtained in step S50 is overlaid with the image taken by the visible light camera, and the video stream is pushed in real time to realize the visualization of sound source positioning.

[0105] For example, a computer system can use an open source computer vision and machine learning software library (Open Source Computer Vision Library, OpenCV) to implement the superposition of a heat map with each frame of an image captured by a visible light camera, and the heat map is superimposed on the upper layer of each frame with a certain degree of transparency to obtain a real-time video stream. The computer system can push the real-time video stream to a uniform resource locator (Uniform Resource Locator, URL) through the Real Time Streaming Protocol (Real Time Streaming Protocol, RSTP), and other devices (for example, other computer devices) can pull the stream in real time through the URL to obtain a real-time video stream. The computer system can clearly see the source of the sound in the video data superimposed with the heat map, that is, realize the visualization of sound source positioning.

[0106] According to one aspect of the present application, a sound source imaging computing system 2 is also provided. Figure 4 As shown, the sound source imaging calculation system 2 includes a data acquisition module 21 , a data processing module 22 , a matrix calculation module 23 and an image processing module 24 .

[0107] According to an exemplary embodiment, the data acquisition module 21 is based on a video collection device and a microphone array, and the data acquisition module 21 acquires video data and audio data simultaneously and in real time.

[0108] According to an exemplary embodiment, the data processing module 22 is used to pre-process the acquired audio data; the data processing module 22 is also used to divide the sound intensity grid within the visible range of the video collection device.

[0109] According to an exemplary embodiment, the matrix calculation module 23 is used to determine the Euclidean distance matrix and the array steering vector from the microphone array to the sound intensity grid. The matrix calculation module 23 is also used to determine the cross-spectrum matrix based on the beamforming delay sum algorithm. The matrix calculation module 23 is also used to obtain the output power spectrum matrix according to the array steering vector and the cross-spectrum matrix.

[0110] According to an exemplary embodiment, the image processing module 24 is used to obtain a thermal map by coloring the sound intensity grid. The image processing module 24 is also used to overlay the acquired video data with the thermal map to obtain a real-time video stream.

[0111] The present application realizes parallel processing of audio data and improves the processing speed and accuracy of sound source imaging through GPU (Graphics Processing Unit) based on CUDA (Compute Unified Device Architecture).

[0112] According to one aspect of the present application, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors can implement the sound source imaging calculation method as described above.

[0113] According to one aspect of the present application, a non-volatile computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned sound source imaging calculation method can be implemented.

[0114] According to one aspect of the present application, a computer program product is also provided, comprising: a computer program stored on a computer-readable storage medium; the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the sound source imaging calculation method as described above.

[0115] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A sound source imaging calculation method, characterized in that: include: Preprocessing the acquired audio data to obtain a center frequency within a scanning frequency range, wherein the audio data is collected by a microphone array; Dividing the sound intensity grid within the visual range of the video collection device; Determining a Euclidean distance matrix and an array steering vector from the microphone array to the sound intensity grid; Obtaining an output power spectrum matrix according to the array steering vector and the cross-spectrum matrix; A heat map is obtained by coloring the acoustic intensity grid.

2. The method according to claim 1, characterized in that: Before preprocessing the acquired audio data to obtain a center frequency within a scanning frequency range, the method further includes: The video collection device is arranged at the center of the microphone array, and a spatial rectangular coordinate system is established with the center of the microphone array as the origin.

3. The method according to claim 1, characterized in that The preprocessing of the acquired audio data to obtain the center frequency within the scanning frequency range includes: Divide the acquired audio data into frames according to a preset frame length to obtain framed audio data; Multiplying each frame of the framed audio data by a preset window function to obtain windowed audio data; Converting the windowed audio data into frequency domain audio data by discrete Fourier transform; The amplitude of the frequency domain audio data is scanned within the scanning frequency range to obtain the center frequency.

4. The method according to claim 1, characterized in that: The dividing of the sound intensity grid within the visible range of the video collection device comprises: Selecting a rectangular area parallel to the plane of the microphone array in a preset direction of the microphone array; The rectangular area is divided into a preset number of grid points to obtain the sound intensity grid.

5. The method according to claim 4, characterized in that The determining of the Euclidean distance matrix and array steering vector from the microphone array to the sound intensity grid comprises: Calculating the Euclidean distance between the microphone array element and the grid point according to the Euclidean distance formula to generate the Euclidean distance matrix; Generate a wave number matrix according to the determined sound wave number per unit time at different frequency points under the preset medium sound speed; The array steering vector at each frequency point within a first frequency range is calculated in parallel by using the CUDA architecture model according to the Euclidean distance matrix and the wavenumber matrix.

6. The method according to claim 3, characterized in that The preprocessed frequency domain audio data is processed in parallel through the CUDA architecture model, and the cross-spectral matrix is ​​determined based on a beamforming delay-sum algorithm.

7. The method according to claim 6, characterized in that The obtaining of the output power spectrum matrix according to the array steering vector and the cross-spectrum matrix comprises: The output power spectrum matrix is ​​generated by calculating the output power of each grid point in parallel through a CUDA architecture model according to the array steering vector and the cross-spectrum matrix.

8. The method according to claim 4, characterized in that The obtaining of the thermal map by coloring the sound intensity grid comprises: Each grid point of the sound intensity grid is colored according to the output power of the output power spectrum matrix to generate the thermal map.

9. The method according to claim 8, characterized in that The acquired video data is superimposed with the heat map to obtain a real-time video stream; wherein the video data is collected by the video collection device.

10. A sound source imaging computing system, characterized in that: The system comprises: A data acquisition module, based on a video collection device and a microphone array, the data acquisition module collects video data and audio data simultaneously and in real time; A data processing module, used to pre-process the acquired audio data and divide the sound intensity grid within the visible range of the video collection device; A matrix calculation module, used to determine the Euclidean distance matrix and array steering vector from the microphone array to the sound intensity grid, and obtain the output power spectrum matrix according to the array steering vector and the cross-spectrum matrix; An image processing module is used to obtain the thermal map by coloring the sound intensity grid.

11. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the sound source imaging calculation method as described in any one of claims 1-9.

12. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sound source imaging calculation method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The invention comprises a computer program stored in a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to execute the sound source imaging calculation method according to any one of claims 1 to 9.

Images