Methods, devices, and electronic equipment for locating sound-emitting objects using acoustic cameras

Abstract

This invention discloses a method, apparatus, and electronic device for locating sound-emitting objects using an acoustic camera, belonging to the field of audio and image signal processing technology. The method for locating sound-emitting objects using an acoustic camera includes: obtaining a sampling point compensation tensor for each microphone based on the microphone position, the pickup space range, and the pixel division within the pickup space range; summing the time-domain signals of each microphone after frame division, and upsampling the summation results according to the required sampling rate to obtain an upsampled time-domain signal for each microphone; substituting the sampling point compensation tensor of each microphone into the corresponding upsampled time-domain signal using the quantile method, performing time delay compensation on each microphone signal, and obtaining a display matrix based on each microphone signal; and locating the sound-emitting object. The processing is performed directly in the time domain, eliminating the need for time-frequency transformation operations for each microphone, and the frame division and summation before upsampling reduces the computational load.

Description

Technical Field

[0001] This invention relates to the field of audio and image signal processing technology, and in particular to a method, apparatus and electronic device for locating sound-emitting objects using an acoustic camera. Background Technology

[0002] An acoustic camera, also called a sound imager, is a specialized device that uses a microphone array to measure the sound field distribution within a certain range. Currently, acoustic cameras are widely used in areas such as capturing vehicle horn violations. The hardware of an acoustic camera includes a microphone array, which calculates the location of the sound source using the time difference between the sound source's arrival at the microphone. Generally, the more microphones there are, the higher the positioning accuracy. When there are a sufficient number of microphones, the location of the sound source can be pinpointed relatively accurately, and the spatial sound field distribution can be generated. Combined with optical image information, the sound source is imaged, providing a clear and intuitive understanding of the sound source's distribution within the sound field.

[0003] Acoustic cameras are typically implemented by calculating the weighted cross-power spectrum of each microphone signal in the frequency domain. This cross-power spectrum is then subjected to an inverse Fourier transform to obtain the cross-correlation function. The maximum value of the cross-correlation function is used to determine the microphone signal delay, thereby identifying the location of the sound source. This sound source imaging method requires first transforming each frame of signal from each microphone to the frequency domain, then calculating and weighting the cross-power spectrum, performing an inverse time-frequency transform to the time domain, and finally determining the delay and sound source location information. This process generally includes... For a time-frequency transformation operation, when the number of microphones M is large, the computational load for the time-frequency transformation step alone is enormous.

[0004] Another method for implementing acoustic cameras is the deconvolution method. In this method, the spatial beamform information is also required first. Then, by solving the convolution relationship between the sound source and the microphone array at each point in space, the sound source location can be located more accurately. The computational load is also very large. Even if the computational load of the algorithm is reduced, it is difficult to deploy it smoothly on a regular CPU. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, and electronic device for locating sound-emitting objects using an acoustic camera, so as to solve the problem of high computational load in current acoustic cameras.

[0006] To solve the above-mentioned technical problems, the present invention provides a method for locating sound-emitting objects using an acoustic camera, comprising the following steps:

[0007] S1, based on the microphone position, the pickup space range, and the pixel division within the pickup space range, obtain the sampling point compensation tensor for each microphone;

[0008] S2, after dividing the time-domain signal of each microphone into frames, sum them separately, and upsample the summation result according to the required sampling rate to obtain the upsampled time-domain signal of each microphone;

[0009] S3, use the quantile method to substitute the compensation tensor of the sampling point of each microphone into the corresponding upsampled time domain signal, perform time delay compensation on each microphone signal, and obtain the display matrix according to each microphone signal;

[0010] S4, match the displayed matrix with the actual image to locate the sound-emitting object.

[0011] Preferably, S1 specifically includes:

[0012] Obtain the microphone's location, the required pickup area, and the pixel divisions within it;

[0013] An incident plane is defined based on the required picking space range, and pixels are divided. A reference distance is also defined for each pixel.

[0014] Calculate the distance difference between each pixel on the incident plane and the distance to each microphone, and the reference distance;

[0015] The sampling point compensation tensor for each microphone is obtained based on the distance difference.

[0016] Preferably, in S1, the sampling point compensation tensor dotMatrix(i,j,k) is obtained by the following formula:

[0017]

[0018] Where fs is the required sampling rate, c is the speed of sound, round indicates rounding to the nearest integer, and dMatrix(i,j,k) represents the distance difference between the distance from the pixel in row i, column j to the k-th microphone and the reference distance.

[0019] Preferably, the step of upsampling the summation result according to the required sampling rate in S2 includes:

[0020] The summation result is then subjected to a high-pass filter;

[0021] The summation result is interpolated based on the relationship between the current sampling rate and the required sampling rate multiple;

[0022] The summation result is then low-pass filtered.

[0023] Preferably, an IIR filter is used for high-pass filtering, and / or an IIR filter is used for low-pass filtering.

[0024] Preferably, S3 specifically includes:

[0025] The upsampled time-domain signal of each microphone is divided into frames to obtain the framed signal. The framed signal is then combined with its corresponding sampling point compensation tensor to obtain the time-delay compensated microphone signal.

[0026] Select M microphone signals at preset intervals and sum them up. Take the average amplitude of the summed signals as the display value of the (i,j)th pixel. Then perform normalization processing to obtain the display matrix.

[0027] Preferably, S3 specifically includes:

[0028] The upsampled time-domain signal of each microphone is divided into frames to obtain the framed signal. The framed signal is then combined with its corresponding sampling point compensation tensor to obtain the time-delay compensated microphone signal.

[0029] The M microphone signals are selected, summed, and differentiated. The ratio of the sum to the difference is taken as the display value of the (i,j)th pixel, thus obtaining the display matrix.

[0030] Preferably, for each microphone, the upsampled time-domain signal du m (n) Perform frame division, and divide the frame signal du m (n,l) is combined with its corresponding sampling point compensation tensor, where l represents the frame index, and the time-delay compensated microphone signal is obtained based on the following formula:

[0031] duc m (i,j,n,l)=du m (n+dotMatrix(i,j,m),l)

[0032] Among them, duc m (i,j,n,l) represents the compensation signal duc m (n,l) represents the microphone signal du. m (n,l) represents the signal delayed by dotMatrix(i,j,m) sampling points, where the subscript m represents the microphone index and n is the sampling point index.

[0033] In a second aspect, the present invention provides an acoustic camera implementation apparatus, comprising:

[0034] The sampling point compensation tensor calculation unit is used to obtain the sampling point compensation tensor of each microphone based on the microphone position, the pickup space range, and the pixel division within the pickup space range.

[0035] The time-domain frame-sum unit is used to divide the time-domain signal of each microphone into frames and sum them separately to obtain the sum of the time-domain signals.

[0036] An upsampling unit is used to upsample the summation result according to the required sampling rate to obtain the upsampled time-domain signal of each microphone;

[0037] The time delay compensation unit is used to substitute the compensation tensor of the sampling point of each microphone into the corresponding upsampled time domain signal using the quantile method, and to perform time delay compensation for each microphone signal;

[0038] A pixel calculation unit is used to obtain a display matrix based on the time-delay compensated microphone signal, and to match the display matrix with the actual image to locate the sound-emitting object.

[0039] In a third aspect, the present invention provides an electronic device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it is used to implement the steps of the method for locating a sound-emitting object using an acoustic camera as described above.

[0040] The method, apparatus, and electronic device for locating sound-emitting objects using an acoustic camera provided by this invention first obtains the spatial range of the target sound source to determine the microphone pickup spatial range. Then, based on the microphone position and pixel division, the pickup space is divided into individual pixels. The microphone delay relationship of each pixel is calculated to obtain the sampling point compensation tensor. After summing the microphone time-domain signal frame by frame, the summation result is upsampled to a suitable sampling rate. The pixels are scanned using the quantile method to fill in the delay, and the spatial node with the largest output is found as the sound source position. The display matrix and the actual image are matched to accurately locate the sound-emitting object. Since the processing is performed directly in the time domain, the time-frequency transformation operation of each microphone is eliminated. Furthermore, the frame-by-frame summation is performed before upsampling, further reducing the computational load. This solves the problem of high computational load in existing methods and reduces the deployment difficulty on the CPU. Attached Figure Description

[0041] Figure 1 This is an execution flowchart of Embodiment 1 provided by the present invention;

[0042] Figure 2 This is a schematic diagram of orientation estimation according to Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of the acoustic camera position according to Embodiment 1 of the present invention;

[0044] Figure 4 This is a schematic diagram of pixel division on the sound source incident plane according to Embodiment 1 of the present invention;

[0045] Figure 5 This is a flowchart illustrating the implementation method of the acoustic camera provided in Embodiment 1 of the present invention.

[0046] Figure 6 This is a schematic diagram of the acoustic camera implementation device according to Embodiment 2 of the present invention;

[0047] Figure 7 This is a schematic diagram of the acoustic camera simulation effect provided in Embodiment 1 of the present invention;

[0048] Figure 8 This is a schematic diagram of the structure of the electronic device according to Embodiment 3 of the present invention.

[0049] In the picture,

[0050] 100. Sampling point compensation tensor calculation unit; 200. Temporal frame summation unit; 300. Upsampling unit; 400. Delay compensation unit; 500. Pixel calculation unit; 600. Electronic equipment; 700. Memory; 800. Processor. Detailed Implementation

[0051] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a more comprehensive overview of the method, apparatus, and electronic device for locating sound-emitting objects using an acoustic camera, as proposed in this invention. The advantages and features of this invention will become clearer from the following description. It should be noted that the accompanying drawings are in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of this invention.

[0052] The inventors discovered that conventional sound source imaging methods, such as time-frequency transformation and deconvolution, involve large computational loads, making it difficult to significantly reduce the computational load and placing high demands on hardware.

[0053] Based on this, the core idea of ​​this invention is to process the signal directly in the time domain, eliminating the need for time-frequency transformation operations for each microphone. First, the microphone signal is upsampled to a suitable sampling rate based on the positioning accuracy to determine the spatial range of the target sound source. The space is then divided into pixels, and the microphone delay relationship for each pixel is calculated. The pixels are then scanned using the quantile method to fill in the delays, and the spatial node with the largest output is found as the sound source location. Compared to existing algorithms, the method provided by this invention significantly reduces the computational load.

[0054] Example 1

[0055] For details, please refer to Figures 1-5 This is a schematic diagram of Embodiment 1 of the present invention. For example... Figure 1 As shown, it includes the following steps:

[0056] S1, based on the microphone position, the pickup space range, and the pixel division within the pickup space range, obtain the sampling point compensation tensor for each microphone;

[0057] S2, after dividing the time-domain signal of each microphone into frames, sum them separately, and upsample the summation result according to the required sampling rate to obtain the upsampled time-domain signal of each microphone;

[0058] S3, use the quantile method to substitute the compensation tensor of the sampling point of each microphone into the corresponding upsampled time domain signal, perform time delay compensation on each microphone signal, and obtain the display matrix according to each microphone signal;

[0059] S4, match the displayed matrix with the actual image to locate the sound-emitting object.

[0060] In this invention, the compensation relationship between each pixel and the sampling point of the microphone array within the required sampling spatial range is first obtained. Then, the time-domain signal is upsampled, and the upsampled signal is compensated according to the sampling point compensation tensor. Finally, the value of each pixel is calculated using the compensated time-domain signal. The linearity of the system is also utilized to optimize the computational load of the time-domain signal before upsampling. The quantile method and the sampling calculation method of the upsampled signal are adopted, thereby significantly reducing the computational load for locating the sound-emitting object. The computational load can be reduced to 1 / 100 or even lower than before.

[0061] Specifically, S1 includes:

[0062] S1.1, Obtain the microphone position, the required pickup space range, and the pixel divisions thereon.

[0063] S1.2, define an incident plane based on the required picking space range, divide the pixels, and define a reference distance for each pixel.

[0064] S1.3, calculate the distance difference between each pixel on the incident plane and the distance to each microphone and the reference distance.

[0065] S1.4, obtain the sampling point compensation tensor for each microphone based on the distance difference.

[0066] In one implementation, such as Figure 2 As shown, Figure 2 Given two microphones A and B, where `source` represents the sound source, the relationship between the incident angle θ and time delay τ of the far-field sound source is given. Figure 2 It can be seen that there is a one-to-one correspondence between the incident angle θ and the time delay τ:

[0067]

[0068] Where c is the speed of sound and L is the distance between the two microphones. For an array of multiple microphones, taking the array as an example where the microphone array is in the same plane, there is still a one-to-one correspondence between the incident angle and the time delay between each microphone.

[0069] Figure 3 This is a schematic diagram of an acoustic camera. "Source" represents the sound source. The microphone array (array in the diagram) is located within a plane (where I is located). A planar sound wave is incident on the front of the plane at a certain angle to the plane's normal. It's understandable that when the incident direction of the sound wave is constant, the phase relationship of the microphone array is also constant; that is, there is a one-to-one correspondence between the incident direction and the phase difference between the microphones. Just as the viewing angle presented by an optical lens has a certain range, the angle of the sound source received by the microphone array also needs to be defined, because as the angle of the sound source increases, the positioning accuracy will gradually decrease. Assume the maximum viewing angle of the microphone array (up, down, left, and right) is θ. m Then the range of sound source incident angles that the microphone array can display is [-θ m ,θ m Assuming the sound source is incident on a plane perpendicular to the normal in front of the microphone array, and the distance from the incident plane to the array is d, then the length and width range of the incident plane of the sound source are [-tan(θ). m )*d,tan(θ m )*d], such as Figure 4 As shown, the incident plane can be divided into 2*dpi+1 parts in both length and width, with the distance between two adjacent points being tan(θ). m )*d / dpi. dpi can be considered equivalent to resolution; the larger the value, the higher the resolution. Each point can be considered equivalent to a pixel, and for each pixel of the sound source, there is a unique time delay relationship corresponding to it in the microphone array.

[0070] Placing the microphone array at the origin in a Cartesian coordinate system, with the array's normal as the x-axis, the spatial coordinates of the incident plane are (d, y, z), where d is the distance from the incident plane to the array, and the values ​​of y and z range from [-tan(θ)]. m )*d,tan(θ m )*d]. Calculate the Euclidean distance from each pixel (d,y,z) to each microphone in the microphone array to obtain the tensor distMatrix. The dimensions of distMatrix are (2*dpi+1)*(2*dpi+1)*M, where M represents the number of microphones. Choose the distance from each pixel to the origin, or the center of the microphone array. Using the baseline distance, we obtain the (2*dpi+1)*(2*dpi+1) dimensional matrix d0Matrix. The difference between the tensors distMatrix and d0Matrix is ​​then calculated using the following formula:

[0071] dMatrix(i,j,k)=distMatrix(i,j,k)-d0Matrix(i,j)

[0072] (Lj∈[1,2*dpi+1],k∈[1,M]) (1)

[0073] Where dMatrix(i,j,k) represents the difference between the distance from the pixel in row i, column j to the k-th microphone and the distance to the center of the array. As mentioned above, when the distance d approaches infinity, this difference is equivalent to the distance difference when a plane wave is incident. Once the distance difference is reached, the number of sampling points corresponding to the distance difference can be calculated.

[0074] Then, in S1, the sampling point compensation tensor dotMatrix(i,j,k) is obtained by the following formula:

[0075]

[0076] Where fs is the required sampling rate, c is the speed of sound, round indicates rounding to the nearest integer, and dMatrix(i,j,k) represents the distance difference between the distance from the pixel in row i, column j to the k-th microphone and the reference distance.

[0077] As shown in the formula for obtaining the sampling point compensation tensor above, the k-th microphone signal starts taking values ​​from the dotMatrix(i,j,k), so the array's directivity is towards position (i,j). Therefore, based on the above formula, it can be seen that if the sampling rate fs is not large enough, the resolution of the compensated sampling points will be too low. The sampling rate of a typical microphone signal is 8k or 16k, which is far from sufficient compared to the required sampling rate. If the sampling rate is less than 192k, the time-domain signal needs to be upsampled to 192k. The step in S2 of upsampling the summation result according to the required sampling rate includes:

[0078] The summation result is subjected to high-pass filtering; the summation result is interpolated according to the relationship between the current sampling rate and the required sampling rate multiple; and the summation result is subjected to low-pass filtering.

[0079] In one implementation, an IIR filter is used for high-pass filtering, and / or an IIR filter is used for low-pass filtering.

[0080] Understandably, because low-frequency time-domain signals change slowly and are less sensitive to different time delay compensations, resulting in a wider beamwidth in beam directionality, a high-pass filter is first applied to the summation of the microphone's time-domain signal frame by frame. Another reason for performing high-pass filtering first is to select a specific frequency band for sound source imaging. In the actual algorithm, an IIR filter can be used for high-pass filtering to control computational load. Then, interpolation is performed based on the relationship between the current sampling rate and the required sampling rate. Here, to reduce the aliasing effect caused by interpolation, instead of directly inserting zero values, linear interpolation or other computationally inexpensive interpolation methods are used. Finally, a low-pass filter is applied to suppress aliasing frequencies caused by interpolation and to select the specified frequencies for sound source imaging in conjunction with the high-pass filter. An IIR filter can also be used for low-pass filtering here.

[0081] In one implementation of S3, S3 specifically includes:

[0082] S3.1, the upsampled time-domain signal of each microphone is divided into frames to obtain the framed signal. The framed signal is combined with its corresponding sampling point compensation tensor to obtain the time-delay compensated microphone signal.

[0083] Among them, the upsampled time-domain signal du from the microphone m (n) Perform frame division, and divide the frame signal du m (n,l) is combined with its corresponding sampling point compensation tensor, where l represents the frame index, and the time-delay compensated microphone signal is obtained based on the following formula:

[0084] duc m (i, j, n, l) = du m (n+dotMatrix(i,j,m),l) (3)

[0085] Among them, duc m (i,j,n,l) represents the compensation signal duc m (n,l) represents the microphone signal du. m (n,l) represents the signal delayed by dotMatrix(i,j,m) sampling points, where the subscript m represents the microphone index and n is the sampling point index.

[0086] S3.2, Select M microphone signals at preset intervals and sum them up. Take the average amplitude of the summed signals as the display value of the (i,j)th pixel. Then perform normalization processing to obtain the display matrix.

[0087] Here, the compensated signal from M microphones is duc m Summing (n, l) and then taking the average of the summed signal amplitudes as the display value for the (i, j)th pixel:

[0088]

[0089] In the above formula, `dispMatrix` is the final output display matrix. When there is a sound source at the corresponding pixel location, the value of the pixel corresponding to `dispMatrix(i,j)` will be relatively large, and the color at that location will be relatively dark when displayed after normalization. By matching the pixels of the `dispMatrix` matrix with the pixels of the image, the location of the sound source can be accurately displayed, realizing the function of an acoustic camera.

[0090] Since Equation (4) sums the upsampled and aligned signals, the number of summations is relatively large. Taking upsampling from 16k to 192k as an example, the upsampling factor is 12. Compared to summing the original signal directly, the computational cost of summing after upsampling is 12 times the number of additions. Therefore, when using Equation (4) for summation, summation can be performed at intervals of sampling points to ensure that the number of additions is controlled within an acceptable range. The number of additions can even be reduced to less than the number of additions corresponding to the original sampling rate, but this step needs to take into account the final imaging effect.

[0091] Furthermore, since the conventional technique of calculating each frame separately before upsampling is used, it is easy to cause the sound source position to be unstable between frames. In the upsampling stage, the large number of microphones and the more than tenfold increase in the number of sampling points after interpolation make the computational load relatively large even with the use of IIR filters. Therefore, in S2, the time domain signal of each microphone is first divided into frames and then summed separately before upsampling.

[0092] This utilizes the linearity of the system. in Indicates the convolution label. First, the same microphone signal d... m (n) is divided into frames d m (n,l), where each frame can be 10ms long, and l represents the frame index. Then, for signal d... m Summation of (n,l):

[0093]

[0094] d m (n,l') is the summation in the time domain of L frames. The larger L is, the more frames are considered simultaneously, the more stable the sound source location information can be, and the less computation is required. It should be noted that if the value of L is too large, it will also cause insufficient refresh rate.

[0095] In one implementation, to balance stability and refresh rate, L can be set to 5 with a frame shift of 10ms, allowing for 20 refreshes per second. Then, the summed signal d... m (n,l') is upsampled, i.e., high-pass filtering, interpolation, and low-pass filtering are performed sequentially to obtain the upsampled signal d. m (n,l'). The number of these operations becomes 1 / L of the previous number, and the overall computational cost of upsampling also becomes 1 / L of the previous number. After upsampling, the delay compensation in equation (3) and the calculation of the display matrix in equation (4) are performed in sequence.

[0096] Using equation (4) to obtain the final display matrix `dispMatrix` results in a large number of pixels, and the computational complexity is a power of two (the resolution squared). As the resolution increases, the computational complexity rises rapidly, making it unsuitable for deployment on commonly used CPUs. Therefore, in another implementation of S3, an alternative method for obtaining the display matrix is ​​presented, with the following specific steps:

[0097] S3.1, the upsampled time-domain signal of each microphone is divided into frames to obtain the framed signal. The framed signal is combined with its corresponding sampling point compensation tensor to obtain the time-delay compensated microphone signal.

[0098] In one implementation, the upsampled time-domain signal du from the microphone... m (n) Perform frame division, and divide the frame signal du m (n,l) is combined with its corresponding sampling point compensation tensor, where l represents the frame index, and the time-delay compensated microphone signal is obtained based on the following formula (3):

[0099] duc m (i, j, n, l) = du m (n+dotMatrix(i,j,m),l) (3)

[0100] Among them, duc m (i,j,n,l) represents the compensation signal duc m (n,l) represents the microphone signal du. m (n,l) represents the signal delayed by dotMatrix(i,j,m) sampling points, where the subscript m represents the microphone index and n is the sampling point index.

[0101] S3.2, Select the time delay compensated signals from M microphones, sum and differentiate them, and then take the ratio of the sum and the difference as the display value of the (i,j)th pixel to obtain the display matrix.

[0102] To optimize imaging, since the microphone signals at the corresponding pixels in the direction of the sound source should exhibit a synchronous trend after time delay compensation, their sum should have a maximum value, and similarly, their subtraction should have a minimum value. The sum and subtraction results can be divided as the pixel output, which not only improves the algorithm's robustness but also eliminates the need for additional result normalization.

[0103]

[0104] dispMatrix(i,j)=dispMatrix(i,j) / minusMatrix(i,j) (7)

[0105] Equation (6) is the difference result of the signal, and Equation (7) is an optimization based on Equation (4). The ratio of the summation to the difference is used as the display of the pixel, and dispMatrix(i,j) is the final output matrix.

[0106] Understandably, although every pixel in dispMatrix(i,j) needs to be displayed, most of these pixels are not what we need. What we need here is the pixel where the sound source is located and the pixels near the sound source. Therefore, we can use the quantile method to locate the sound source and calculate the pixels near the sound source step by step:

[0107] There are a total of (2*dpi+1)*(2*dpi+1) pixels. If each pixel is calculated using equation (7), the computational load would be enormous. To reduce the computational load, we use equation (7) once every dpis1 pixels, which will calculate (2*dpi / dpis1+1)*(2*dpi / dpis1+1) pixels. We then find the maximum value of the pixel, maxValue, and record its location (maxi, maxj). In this way, the sound source is located near the pixel (maxi, maxj).

[0108] Then, a second search is performed, with similar interval calculations within the pixel intervals (maxi-dpis1, maxi+dpis1) and (maxj-dpis1, maxj+dpis1), using an interval of dpis2. Simultaneously, the maximum pixel value (maxValue) and its position (maxi, maxj) are updated. The number of pixels to be calculated here is (2*dpis1 / dpis2+1)*(2*dpis1 / dpis2+1). Finally, all pixels within the prime pixel intervals (maxi-dpis2, maxi+dpis2) and (maxj-dpis2, maxj+dpis2) are calculated, again requiring (2*dpis2+1)*(2*dpis2+1). The default value for any uncalculated pixels is 0.

[0109] After optimization, the number of pixel calculations changed from (2*dpi+1)*(2*dpi+1) to (2*dpi / dpis1+1)*(2*dpi / dpis1+1)+(2*dpis1 / dpis2+1)*(2*dpis1 / dpis2+1)+

[0110] (2*dpis2+1)*(2*dpis2+1).

[0111] Based on the quantile method described above, if the resolution dpi is 128, dpis1 is 32, and dpis2 is 8, the number of pixels to be calculated decreases from 66049 to 451, reducing the computational load to less than 1 / 100. Furthermore, optimizations to the upsampling operation and temporal summation significantly reduce the computational complexity, even in scenarios with a large number of microphones, thus greatly reducing the deployment difficulty on the CPU side.

[0112] The above method for locating sound-emitting objects using an acoustic camera can be described as follows: Figure 5 The simulation process shown uses this method with a microphone array of M=8 in a ring, an array size of approximately 2.5dm, a sampling rate of 192k after upsampling, a maximum pickup angle θ_m=45 degrees, dpi=128, dpis1=32, and dpis2=8. Figure 7 This is a screenshot showing the obtained sound source localization.

[0113] Example 2

[0114] The present invention also provides an acoustic camera implementation device, such as... Figure 6 As shown, it includes:

[0115] The sampling point compensation tensor calculation unit 100 is used to obtain the sampling point compensation tensor for each microphone based on the microphone position, the pickup space range, and the pixel division within the pickup space range. First, the specific positions of each microphone in the acoustic camera microphone array, the required pickup angle range, and the required resolution and temporal signal sampling rate are determined. The area to be picked up is projected onto a plane, and the plane is divided into pixels according to the resolution. The distance from each pixel to each microphone is calculated, and a reference distance is defined for each pixel. The distance difference between each pixel's distance to each microphone and this reference distance is calculated, and the sampling point compensation is calculated based on the distance difference. The sampling point compensation is then placed into the corresponding pixel to obtain the sampling point compensation tensor dotMatrix. The step of obtaining the sampling point compensation tensor only needs to be performed once during algorithm initialization, and its computational cost is negligible.

[0116] The time-domain frame-splitting and summing unit 200 is used to divide the time-domain signal of each microphone into frames and sum them separately to obtain the summed time-domain signal result. Each microphone is framed, and then the time-domain signals are summed according to the number of frames to be processed simultaneously, resulting in signal d. m .

[0117] Upsampling unit 300 is used to upsample the summation result to obtain the upsampled time-domain signal of each microphone. The summation signal d of each microphone is then... m Upsampling is performed to obtain the upsampled signal du. m The upsampling process includes high-pass filtering, interpolation, and low-pass filtering. The order in which the time-domain frame summation unit and the upsampling unit are executed can be interchanged; that is, the upsampling unit performs the upsampling process first, followed by the summation process in the time-domain frame summation unit. However, this would increase the computational load. Therefore, based on the linearity of the system, the process executed by the time-domain frame summation unit is placed before that executed by the upsampling unit to reduce the computational load.

[0118] The time delay compensation unit 400 is used to substitute the sampling point compensation tensor of each pixel into the corresponding upsampled time-domain signal of the microphone using the quantile method, and to perform time delay compensation on each microphone signal. The upsampled signal of each microphone is then processed by du... m Delay compensation is performed at each pixel using the sampling point compensation tensor dotMatrix to obtain the compensation signal duc for each microphone at the corresponding pixel. m .

[0119] The pixel calculation unit 500 is used to obtain a display matrix based on the time-delay compensated microphone signal, and to match the display matrix with the actual image to locate the sound-emitting object. The compensation signal duc of the corresponding pixel is then used to... mThe summation is performed based on the number of microphones to obtain the display matrix `dispMatrix`. Optionally, the summation can be replaced with the ratio of the sum to the difference to increase the algorithm's robustness.

[0120] like Figure 6 This is a schematic diagram of one embodiment of the acoustic camera design method provided by the present invention. The corresponding processes are executed sequentially by the sampling point compensation tensor calculation unit 100, the temporal domain frame summation unit 200, the upsampling unit 300, the time delay compensation unit 400, and the pixel calculation unit 500. Figure 4 This is a flowchart of an acoustic camera, combined with... Figure 6 , Figure 5 and Figure 4 The specific process of each execution unit is described as follows: Based on the microphone position, pickup angle, and pixel division of the acoustic camera, the sampling point compensation tensor is calculated; the temporal signal of each microphone is divided into frames, and the framed signals of each microphone are summed as needed. The summation result of each microphone is upsampled to obtain the framed upsampled signal of each microphone; using the quantile method, the sampling point compensation tensor is used to compensate for the time delay of the framed upsampled signal of each microphone; the compensated signals are summed and differentiated; the summation result or the ratio of the summation result to the difference result is placed into the corresponding pixel position, the display array or display matrix is ​​updated, and the obtained complete display matrix is ​​matched with the actual graphic to accurately locate the sound-emitting object.

[0121] Example 3

[0122] The present invention also provides an electronic device 600, such as... Figure 8 As shown, the device includes a processor 800, a memory 700, and a computer program stored in the memory 700 and executable on the processor 800. When the computer program is executed by the processor 800, it is used to implement the steps of the method for locating a sound-emitting object using an acoustic camera as described in Embodiment 1. The memories communicate with each other via a communication bus, and the electronic device also includes a communication interface for communication between the electronic device and other devices.

[0123] In addition, other implementations of a method for locating a sound-emitting object using an acoustic camera, implemented by the processor 800 executing the program stored in the memory 700, are the same as those mentioned in the aforementioned method embodiment section, and will not be repeated here.

[0124] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0125] The memory 700 may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0126] The processor 800 mentioned above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0127] In summary, the method, apparatus, and electronic device for locating sound-emitting objects using an acoustic camera provided by this invention first calculates the time delay compensation relationship between each pixel on the display interface and the microphone array. Then, the time-domain signal is upsampled. Next, the upsampled signal is compensated according to the time delay compensation tensor. Finally, the value of each pixel is calculated using the compensated time-domain signal. The linearity of the system is utilized to optimize the computational complexity of upsampling. When calculating the pixels of the display matrix, a quantile method is used, and a sampling calculation method based on the upsampled signal is employed, reducing the computational complexity to 1 / 100 or even lower than previous algorithms. Furthermore, the pixel calculation can employ a combination of summation and difference, increasing the algorithm's robustness.

[0128] The above description is merely a description of preferred embodiments of the present invention and is not intended to limit the scope of the present invention in any way. Any changes or modifications made by those skilled in the art based on the above disclosure shall fall within the protection scope of the claims.

Claims

1. A method for locating a sound-emitting object using an acoustic camera, characterized in that, Includes the following steps: S1. Based on the microphone position, the pickup space range, and the pixel division within the pickup space range, obtain the sampling point compensation tensor for each microphone, wherein the sampling point compensation is placed into the corresponding pixel to obtain the sampling point compensation tensor. S2, after dividing the time-domain signal of each microphone into frames, sum them separately, and upsample the summation result according to the required sampling rate to obtain the upsampled time-domain signal of each microphone; S3, use the quantile method to substitute the compensation tensor of the sampling point of each microphone into the corresponding upsampled time domain signal, perform time delay compensation on each microphone signal, and obtain the display matrix according to each microphone signal; S4, match the displayed matrix with the actual image to locate the sound-emitting object; S3 specifically includes: The upsampled time-domain signal of each microphone is divided into frames to obtain the framed signal. The framed signal is then combined with its corresponding sampling point compensation tensor to obtain the time-delay compensated microphone signal. The M microphone signals are selected, summed, and differentiated. The ratio of the sum to the difference is taken as the display value of the (i,j)th pixel, thus obtaining the display matrix.

2. The method for locating a sound-emitting object using an acoustic camera as described in claim 1, characterized in that, S1 specifically includes: Obtain the microphone's location, the required pickup area, and the pixel divisions within it; An incident plane is defined based on the required picking space range, and pixels are divided. A reference distance is also defined for each pixel. Calculate the distance difference between each pixel on the incident plane and the distance to each microphone, and the reference distance; The sampling point compensation tensor for each microphone is obtained based on the distance difference.

3. The method for locating a sound-emitting object using an acoustic camera as described in claim 2, characterized in that, In S1, the sampling point compensation tensor It can be obtained through the following formula: in, This is the required sampling rate, c is the speed of sound, and round indicates rounding to the nearest integer. This represents the distance difference between the pixel in row i, column j and the distance to the k-th microphone, and the reference distance.

4. The method for locating a sound-emitting object using an acoustic camera as described in claim 1, characterized in that, The step of upsampling the summation result according to the required sampling rate in S2 includes: The summation result is then subjected to a high-pass filter; The summation result is interpolated based on the relationship between the current sampling rate and the required sampling rate multiple; The summation result is then low-pass filtered.

5. The method for locating a sound-emitting object using an acoustic camera as described in claim 4, characterized in that, High-pass filtering can be performed using an IIR filter, and / or, low-pass filtering can be performed using an IIR filter.

6. The method for locating a sound-emitting object using an acoustic camera as described in claim 1, characterized in that, S3 specifically includes: The upsampled time-domain signal of each microphone is divided into frames to obtain the framed signal. The framed signal is then combined with its corresponding sampling point compensation tensor to obtain the time-delay compensated microphone signal. Select M microphone signals at preset intervals and sum them up. Take the average amplitude of the summed signals as the display value of the (i,j)th pixel. Then perform normalization processing to obtain the display matrix.

7. The method for locating a sound-emitting object using an acoustic camera as described in claim 1, characterized in that, The upsampled time-domain signal for each microphone Perform framing, and divide the framing signal Combined with the corresponding sampling point compensation tensor, The frame index is used to obtain the delay-compensated microphone signal based on the following formula: = in, Indicates compensation signal microphone signal Delay The signal is sampled at n sampling points, where the subscript m represents the microphone index and n is the sampling point index.

8. An acoustic camera implementation device, characterized in that, A method for locating a sound-emitting object using an acoustic camera as described in any one of claims 1-7, comprising: The sampling point compensation tensor calculation unit is used to obtain the sampling point compensation tensor of each microphone based on the microphone position, the pickup space range, and the pixel division within the pickup space range. The time-domain frame-sum unit is used to divide the time-domain signal of each microphone into frames and sum them separately to obtain the sum of the time-domain signals. An upsampling unit is used to upsample the summation result according to the required sampling rate to obtain the upsampled time-domain signal of each microphone; The time delay compensation unit is used to substitute the compensation tensor of the sampling point of each microphone into the corresponding upsampled time domain signal using the quantile method, and to perform time delay compensation for each microphone signal; A pixel calculation unit is used to obtain a display matrix based on the time-delay compensated microphone signal, and to match the display matrix with the actual image to locate the sound-emitting object.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it is used to implement the steps of the method for locating a sound-emitting object using an acoustic camera as described in any one of claims 1-7.

Images