A feature preprocessing and extraction method for multi-channel audio positioning

Abstract

The application is a feature preprocessing and extraction method for multi-channel audio positioning. The application relates to the technical field of audio preprocessing, and extracts DSB features, LogMel features and IV features. The extracted features are globally normalized, and the normalized data is trained and tested on deep learning. The extracted features are used to train a CRNN model for sound source direction estimation through DSB edge end acceleration method. The application extracts LogMel features and IV features from the audio processed by DSB. In the feature extraction process, the application uses edge end GPU to accelerate the processing on Nano. Finally, a CRNN is trained using the extracted features. Compared with the features without DSB, the CRNN trained by the DSB method can effectively reduce the DoA error, and can still run in real time on the edge end.

Description

Technical Field

[0001] This invention relates to the field of audio preprocessing technology, and is a feature preprocessing and extraction method for multi-channel audio localization. Background Technology

[0002] With the continuous enhancement of computing power and the widespread adoption of network connectivity, artificial intelligence (AI) algorithms are increasingly being applied in industrial environments. In modern factories, product manufacturing largely depends on the correct and reliable operation of machines and tools. However, when machines and tools fail, the costs of repair and production interruptions can lead to significant revenue losses. Therefore, condition-based maintenance and early fault detection are crucial in today's factories. Sensors have been extensively researched and applied in equipment fault detection. While images are highly effective in fault detection, they are prone to information leakage, limiting their use in sensitive areas such as the defense industry. Therefore, non-image sensing data, such as current and voltage information, are often the preferred choice. However, this data typically relies either on sensors built into the machine or requires expensive instruments to acquire. In contrast, acoustic methods using microphones offer a low-cost data collection method for machine diagnostics. These microphones can be installed near the machine or placed in the factory floor. The sound source of a fault can come from various directions, making sound source location crucial. Sound source localization (SSL) typically uses multiple microphone audio channels to analyze the position of one or more sound sources relative to the microphones. In many cases, the SSL problem is simplified to estimating the Directions of Arrival (DOA). The primary approach to solving the DOA problem is to use deep neural networks to learn the correlations between multi-channel audio. However, existing machine learning-based DOA estimation solutions typically use temporal and spectral features as input. These features only implicitly capture spatial information, such as inter-temporal differences, making it difficult for Convolutional Neural Networks (CNNs) to effectively extract valid DOA information. Summary of the Invention

[0003] This invention addresses the application of multi-source sound localization in numerous fields, including auditory scene analysis, fault detection and diagnosis in manufacturing, and augmented reality. In the far field, 3D sound source localization is equivalent to determining the DOA (Direction of Arrival) of a sound source, i.e., its azimuth and elevation angles. Recent DOA estimation pipelines employ multi-channel audio input, extracting spectral features from each channel and feeding them into a deep neural network. Unfortunately, spectral features only contain time-frequency information of the audio signal, while spatial information is implicitly captured only in the signals of different channels, which is highly dependent on the geometry of the acoustic array. To embed the spatial information of the sound source into the spectral feature representation, a spatial mapping method based on Delay and Sum Beamforging (DSB) is proposed to encode the sound source location information. This method can be combined with different feature extraction methods and machine learning models for DOA estimation. Furthermore, a redundancy elimination method is proposed to accelerate DSB computation, enabling the pipeline to run in real-time on embedded GPUs such as the NVIDIA Jeston Nano. Extensive experiments were conducted on two datasets using two neural network models and the DSB method. Experiments show that the DSB method can effectively reduce DOA error. When combined with DSB for feature extraction, the DOA error can be reduced by 19.24%. Furthermore, after applying redundancy removal, the speed of the feature extraction process is improved by 30.42%. Based on this, this invention proposes a feature preprocessing and extraction method for multi-channel audio localization.

[0004] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0005] This invention provides a feature preprocessing and extraction method for multi-channel audio localization, and the technical solution provided by this invention is as follows: A feature preprocessing and extraction method for multi-channel audio localization, the method comprising the following steps: Step 1: Extract LogMel and IV features from DSB features. Perform global normalization on the extracted features. Use the normalized data for training and testing on deep learning. Step 2: Accelerate the process using the DSB edge acceleration method, and use the extracted features to train a CRNN model for sound source direction estimation.

[0006] Preferably, step 1 specifically comprises: set up On a unit sphere centered at the array center A set of locations. For polar coordinates. The delay of the assumed sound source on the array is given by the following formula, given the received signal. , No. The DSB of each channel is represented by the following formula:

[0007] Preferably, when With source If they overlap, then Consistent accumulation in the array output.

[0008] Preferably, the steps for DSB-based spatial mapping are as follows: Select in the monitoring area The location of the virtual sound source is denoted as These points are evenly distributed on a unit sphere; For each virtual source exist Calculate its array vector ; Given an array input calculate ,in Sampling frequency, Sample index The number of samples in each (windowed) segment and the score delay rounded to the nearest sample; Based on DSB, spatial mapping is Output of virtual sound source location set in -channel time series, called For each channel, frequency domain audio features can be further extracted from the data in that channel.

[0009] Preferably, the amplitude and phase spectra are determined, and the Discrete Short-Time Fourier Transform (STFT) is applied to the windowed data to calculate the amplitude and phase spectra, providing... The discrete STFT of data in the channel is expressed by the following formula:

[0010] in, It is a window function. In the implementation, this invention uses a Hamming window with a length of 2048.

[0011] Determining the Log-Mel spectrogram involves using a filter bank that simulates a human audio front end to analyze the incoming signal. The obtained data is used to calculate the logarithmic Mel spectrum by taking the logarithm of the Mel spectrum. Channel The -th logMel spectrum is as follows:

[0012] in, For Mel filter banks.

[0013] The intensity vector (IV) of a multi-channel audio signal is determined by calculating the correlation between the signal in each channel and the reference channel. For multiple signals after DSB, the reference channel audio is obtained by directly accumulating the received signals without delay. For any channel... The initial intensity vector of the audio after DSB is represented by the following formula:

[0014] in, It is a channel STFT characteristics, It is the STFT characteristic of the reference audio. It is the complex conjugate returned by the element. It is the real part of the complex number returned by the element; the extracted intensity vector features can be further normalized by dividing each element by the square root of the sum of squares of the audio data after dividing by all DSBs; the square root of the sum of squares of the audio data after DSBs is:

[0015] By multiplying with the Mel filter bank, the luminance vector features are transformed to the Mel domain. After DSB, the audio intensity vector of any channel h is:

[0016] Among them, For Mel filter banks.

[0017] Preferably, step 2 specifically comprises: Compute delayed summation, SFTF, and LogMel features on a GPU using the Cupy library; in `delay` and `sum`, when the summation is delayed by the same amount of time, the resulting summation is also delayed by the same amount of time, considering locations... and The signals received by the two microphones and For any position where the unit sphere intersects with the hyperbola that satisfies the condition... The delay sum is in the form of: , , For different Delayed and moved These parts are stored and reused in different locations to further accelerate the computation of DSB.

[0018] Preferably, when there are four microphones and two points in the space and After delay calculation, the delays corresponding to the two points are respectively and Redundancy exists in the following cases: Case 1: When it is ,calculate The delays of the x and y microphones at a point are summed and stored in memory, then the delay results of the other microphones are added together; at the point... When, call directly The sum of the delay results from the x and y microphones at each location is then added together with the delay results from the other microphones. Scenario 2: When The calculation process is similar to that of Case 1. In the overall calculation process, Case 1 is given priority, while Case 2 can directly call the result from Case 1.

[0019] A feature preprocessing and extraction system for multi-channel audio localization, the system comprising: The feature extraction module performs DSB feature extraction to extract LogMel and IV features. The extracted features are then globally normalized, and the normalized data is used for training and testing on deep learning. The sound source estimation module accelerates the process using the DSB edge acceleration method and uses the extracted features to train a CRNN model for sound source direction estimation.

[0020] A computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a feature preprocessing and extraction method for multi-channel audio localization.

[0021] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement a feature preprocessing and extraction method for multi-channel audio localization.

[0022] The present invention has the following beneficial effects: Compared with the prior art, the present invention: This invention proposes a feature preprocessing and extraction method for multi-channel audio localization. This method uses the DSB algorithm to perform beamforming on multi-channel audio at multiple points in space. Simultaneously, to accelerate the beamforming process, this invention employs a redundancy removal algorithm within the DSB edge acceleration algorithm, thereby reducing unnecessary computations in beamforming. Subsequently, this invention extracts LogMel and IV features from the DSB-processed audio. During feature extraction, this invention utilizes edge GPUs on Nano processors for accelerated processing. Finally, a CRNN is trained using the extracted features. Compared to using features without DSB, the CRNN trained using the DSB method can effectively reduce DoA error and still run in real-time at the edge. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 An example of locating a sound source in space. Detailed Implementation

[0025] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0027] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0028] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0029] The present invention will be described in detail below with reference to specific embodiments. Specific Implementation Example 1: according to Figure 1 As shown, the specific optimization technical solution adopted by the present invention to solve the above-mentioned technical problems is: The present invention relates to a feature preprocessing and extraction method for multi-channel audio localization.

[0031] Step 1: Extract LogMel and IV features from DSB features. Perform global normalization on the extracted features. Use the normalized data for training and testing on deep learning. Step 2: Accelerate the process using the DSB edge acceleration method, and use the extracted features to train a CRNN model for sound source direction estimation.

[0032] Step 1 specifically involves: set up On a unit sphere centered at the array center A set of locations. For polar coordinates. The delay of the assumed sound source on the array is given by the following formula, given the received signal. , No. The DSB of each channel is represented by the following formula:

[0033] when With source If they overlap, then Consistent accumulation in the array output.

[0034] The steps for DSB-based spatial mapping are as follows: Select in the monitoring area The location of the virtual sound source is denoted as These points are evenly distributed on a unit sphere; For each virtual source exist Calculate its array vector ; Given an array input calculate ,in Sampling frequency, Sample index The number of samples in each (windowed) segment and the score delay rounded to the nearest sample; Based on DSB, spatial mapping is Output of virtual sound source location set in -channel time series, called For each channel, frequency domain audio features can be further extracted from the data in that channel.

[0035] To determine the amplitude and phase spectra, the Discrete Short-Time Fourier Transform (STFT) is applied to windowed data to calculate the amplitude and phase spectra, and the results are presented. The discrete STFT of data in the channel is expressed by the following formula:

[0036] in, It is a window function. In the implementation, this invention uses a Hamming window with a length of 2048.

[0037] Determining the Log-Mel spectrogram involves using a filter bank that simulates a human audio front end to analyze the incoming signal. The obtained data is used to calculate the logarithmic Mel spectrum by taking the logarithm of the Mel spectrum. Channel The -th logMel spectrum is as follows:

[0038] in, For Mel filter banks.

[0039] The intensity vector (IV) of a multi-channel audio signal is determined by calculating the correlation between the signal in each channel and the reference channel. For multiple signals after DSB, the reference channel audio is obtained by directly accumulating the received signals without delay. For any channel... The initial intensity vector of the audio after DSB is represented by the following formula:

[0040] in, It is a channel STFT characteristics, It is the STFT characteristic of the reference audio. It is the complex conjugate returned by the element. It is the real part of the complex number returned by the element; the extracted intensity vector features can be further normalized by dividing each element by the square root of the sum of squares of the audio data after dividing by all DSBs; the square root of the sum of squares of the audio data after DSBs is:

[0041] By multiplying with the Mel filter bank, the luminance vector features are transformed to the Mel domain. After DSB, the audio intensity vector of any channel h is:

[0042] Among them, For Mel filter banks.

[0043] Step 2 specifically involves: Compute delayed summation, SFTF, and LogMel features on a GPU using the Cupy library; in `delay` and `sum`, when the summation is delayed by the same amount of time, the resulting summation is also delayed by the same amount of time, considering locations... and The signals received by the two microphones and For any position where the unit sphere intersects with the hyperbola that satisfies the condition... The delay sum is in the form of: , , For different Delayed and moved These parts are stored and reused in different locations to further accelerate the computation of DSB.

[0044] When there are four microphones and two points in the space and After delay calculation, the delays corresponding to the two points are respectively and Redundancy exists in the following cases: Case 1: When it is ,calculate The delays of the x and y microphones at a point are summed and stored in memory, then the delay results of the other microphones are added together; at the point... When, call directly The sum of the delay results from the x and y microphones at each location is then added together with the delay results from the other microphones. Scenario 2: When The calculation process is similar to that of Case 1. In the overall calculation process, Case 1 is given priority, while Case 2 can directly call the result from Case 1. Specific Implementation Example 2: The only difference between Embodiment 2 and Embodiment 1 of this application is that: Description of the DoA problem Assume M microphones are located at To avoid loss of generality, this invention assumes that the center of mass of the microphone is located at the origin of the 3D coordinate space, i.e. ..set up Let be the (unknown) number of active sound sources within the time interval [0, T]. The set of sound sources is represented as... Because of the aperture size of commercially available loudspeaker microphone arrays, which is typically on the order of several meters from the center, sound sources a few meters away from the array center can be considered to be in the far field. Therefore, the sound waves arriving at the array are approximately plane waves. In other words, the DOA of a sound wave from a single source is the same at each microphone. In a far-field environment, only the DOA of the sound source can be estimated, and its range remains uncertain.

[0046] Therefore, using It is sufficient to represent the azimuth and elevation angles of the sound source in polar coordinates. Let... As the sound source Propagation wave, For microphone The received wave, that is,

[0047] Among them The delay in transmission originates from The time, that is , This is the speed of sound traveling through the air. Note that under the plane wave assumption and time-invariant channel, the speed of sound from... The sound waves were transmitted through all the microphones. Attenuation. Let... To start from the source Delay to the center of the array. When the source In the far field

[0048] The last term is obtained by transforming polar coordinates to Cartesian coordinates. An important implication of the above equation is that the delay difference between different microphones depends only on the DOA of the sound source and the geometry of the array (i.e., 's), rather than the range of the sound source, which is in It is absorbed and cannot be determined. Therefore, this invention can define a vector. , of which Elements are composed of And the following vector form,

[0049] like Figure 1 As shown, in Within a time interval, four microphones captured acoustic signals from three different spatial directions. An embedded device then performed real-time feature extraction on the acquired acoustic data, followed by prediction using a neural network. Finally, the embedded device provides an output indicating the number of detected sounds and their corresponding directional locations. Given... It sounds like the multi-source localization problem is deterministic. The number of sound sources and their DOAs .

[0050] Audio feature extraction Traditional audio preprocessing primarily removes noise and reverberation from audio. Noise refers to unwanted additional sound components that interfere with the original audio signal. Noise can be generated by the environment, equipment, or other electromagnetic interference; common noises include background noise, power supply noise, and electromagnetic interference. Noise removal can be achieved through static noise removal, adaptive filtering, and spectral subtraction filtering. Reverberation is caused by the delay and frequency response changes resulting from the reflection and diffraction of sound in an enclosed space, often adding a tail or making the sound hollow. Reverberation removal can be achieved by performing a Fourier transform on the audio signal, converting the time-domain signal to a frequency-domain signal, which facilitates the analysis and processing of the reverberation component. Digital filters can also be used to compensate for the impact of reverberation on the sound spectrum, restoring the characteristics of the original signal as much as possible. With the development of deep learning, current audio preprocessing is often used before feature extraction, rather than processing the signal separately. Taking LogMel spectrum features as an example, the original audio signal is first preprocessed to extract more meaningful features in subsequent steps. Preprocessing typically includes removing silence segments, noise reduction, eliminating DC components, and normalizing audio amplitudes. The preprocessed audio signal is divided into short frames, typically 20-30 milliseconds long. This is done to assume that the characteristics of the sound are stable within each time period and to capture short-term changes in speech. A window function is applied to each frame; common window functions include the Hamming window and the Hanning window. This step helps reduce the effects of spectral leakage. A Fast Fourier Transform (FFT) is performed on the windowed audio frames to convert the time-domain signal to the frequency-domain signal. This produces the spectrogram of each frame. A set of Mel filters is applied to the frequency-domain signal. The Mel filter bank is a set of triangular filters, dense in the low-frequency region and sparse in the high-frequency region. This is because the human ear is more sensitive to low-frequency sounds, while high-frequency sounds change rapidly and require higher resolution. The result of filtering the filter bank is then logarithmically calculated. This is because human auditory perception is closer to the logarithmic response to frequency, and using logarithms better simulates human perception.

[0051] DSB Feature Extraction Distributed Signal Burning (DSB) is a well-known array processing technique that aligns the delay of the received signal to a target direction and then sums the corresponding delayed signals on the array. Signals from the target direction are coherently superimposed, while signals from other directions exhibit distortion and incoherent aggregation. The DSB algorithm can be directly used for sound source localization, searching all possible candidate sound source directions to find the direction of the sound source with the highest power. Although the DSB algorithm is simple, its computational cost increases linearly with the size of the search area. This invention uses DSB to generate a spatial representation of the signal in a fixed set of directions. These directions are typically not aligned with the actual source location, but by introducing different delayed versions of the original signal, spatial information is directly embedded into the generated representation.

[0052] Specifically, let's set On a unit sphere centered at the array center A set of locations. For polar coordinates. The delay of the assumed sound source on the array is given by the following formula. Given the received signal... , No. The DSB calculation method for each channel is as follows:

[0053] The effect of DSB is obvious: if With source If they overlap, then Consistent accumulation in the array output.

[0054] A slight complexity arises in the DSB implementation using discrete samples. To handle fractional delays that are not multiples of the sampling interval, one approach is to interpolate the received waveform. However, at a sampling frequency of 48 kHz, rounding the resulting delay to the nearest sampling interval is sufficient without significantly degrading the final performance. As an envelope back-side check, for a sound source located on the same side of the linear array, an arrival time difference of 1 / 48000 second roughly corresponds to a DOA difference of 0.0071 rad or 0.41°. The steps for DSB-based spatial mapping are as follows: Select in the monitoring area The location of the virtual sound source is denoted as These points are evenly distributed on a unit sphere.

[0055] For each virtual source exist Calculate its array vector .

[0056] Given an array input calculate ,in Sampling frequency, Sample index The number of samples in each (windowed) segment and the score delay rounded to the nearest sample.

[0057] Based on DSB, spatial mapping is Output of virtual sound source location set in -channel time series (referred to as (Channels). Frequency domain audio features can be further extracted from the data in each channel.

[0058] (1) Amplitude and Phase Spectra: The Discrete Short-Time Fourier Transform (STFT) can be applied to windowed data to calculate the amplitude and phase spectra. The following is given... Discrete STFT of data in the channel

[0059] in It is a window function. In the implementation, this invention uses a Hamming window with a length of 2048.

[0060] (2) Log-Mel spectrogram: The Mel spectrogram is a filter bank that simulates the human audio front end to analyze the incoming signal. The result is obtained through transmission. Then, the log-Mel spectrum is calculated by taking the logarithm of the Mel spectrum. Specifically, Channel -th log Mel spectrum is

[0061] Among them For Mel filter banks.

[0062] (3) Intensity Vector (IV): The intensity vector of a multi-channel audio signal is obtained by calculating the correlation between the signal in each channel and the reference channel. For multiple signals after DSB, the reference channel audio is obtained by directly accumulating the received signals without delay. For any channel... The initial intensity vector of the audio after DSB is:

[0063] in It is a channel STFT characteristics, It is the STFT characteristic of the reference audio. It is the complex conjugate returned by the element. This is the real part of the complex number returned by the element. The extracted intensity vector features can be further normalized by dividing each element by the square root of the sum of squares of the audio data after dividing by all DSBs. The square root of the sum of squares of the audio data after DSB is:

[0064] Finally, the luminance vector features are transformed to the Mel domain by multiplying with the Mel filter bank. After DSB, the audio intensity vector of any channel h is:

[0065] Among them For Mel filter banks.

[0066] After performing DSB on multi-channel audio, LogMel and IV features can be extracted. The extracted features need to be globally normalized, and the normalized data is used for training and testing on a deep learning platform. Experimental results show that, with the same data scale and model, both DSB and IV features can effectively reduce DOA error.

[0067] DSB Edge Acceleration Algorithm To enable DoA to run in real-time or near real-time on embedded devices, this invention utilizes the NVIDIA Jetson Nano GPU to accelerate spatial mapping and feature extraction, and proposes a redundancy removal algorithm.

[0068] First, to accelerate computation on Nano devices, this invention computes delayed summation, SFTF, and LogMel features on their GPUs using the Cupy library. Cupy is an open-source array library for GPU-accelerated computation. Second, in `delay` and `sum`, when the summation delay is the same amount of time, the resulting summation is also delayed by the same amount of time (assuming the attenuation difference is negligible). Considering the location... and The signals received by the two microphones and For a unit sphere and satisfying , Any position where the hyperbolas intersect The delay sum is in the form of , For different Delayed and moved Therefore, this invention allows these parts to be stored and reused in different locations to further accelerate the calculation of DSB.

[0069] For example, suppose there are four microphones and two points in the space. and After delay calculation, the delays corresponding to the two points are respectively... and Redundancy exists in the following cases: (1) If it is ,calculate The delays of the x and y microphones at a point are summed and stored in memory. Then, the delay results of the other microphones are added together. This is done at the point where the delays are calculated. When, call directly The result of the sum of the delays from the x and y microphones is calculated, and then the delay results from the other microphones are added together.

[0070] (2) If The calculation process is similar to (1). In the overall calculation process, (1) is given priority, while (2) can directly call the result in (1). Specific Implementation Example 3: The only difference between Embodiment 3 and Embodiment 2 of this application is that: This invention uses a microphone to acquire multi-channel audio data. Subsequently, it applies the DSB algorithm to beamform this multi-channel audio data, thereby improving the accuracy of sound source localization. To accelerate beamforming, this invention introduces a DSB edge acceleration algorithm, which effectively removes redundant calculations and optimizes computational efficiency. Next, this invention extracts features from the DSB-processed audio data, including the calculation of LogMel and IV features. This feature extraction process utilizes edge GPUs to ensure real-time performance, making it particularly suitable for embedded devices such as Nano. Finally, this invention uses these extracted features to train a CRNN model for sound source direction estimation. This innovative method, compared to methods without DSB, significantly reduces DoA error, thereby improving the accuracy of sound source localization. More importantly, this invention's method has the capability to run in real-time at the edge, making it suitable for audio processing scenarios requiring rapid response, such as fault detection in industrial environments. This comprehensive method achieves efficient and high-quality sound source localization by optimizing preprocessing, feature extraction, and model training for multi-source localization. Specific Implementation Example 4: The only difference between Embodiment 4 and Embodiment 3 of this application is that: The present invention provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement a feature preprocessing and extraction method for multi-channel audio localization. Specific Implementation Example 5: The only difference between Embodiment 5 and Embodiment 4 of this application is that: The present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a feature preprocessing and extraction method for multi-channel audio localization.

[0074] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or N embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified. Any process or method described in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logical functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order according to the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain. The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include the following: electrical connections having one or N wires (electronic devices), portable computer disks (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM).Furthermore, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory. It should be understood that various parts of the invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0075] The above description is merely a preferred embodiment of a feature preprocessing and extraction method for multi-channel audio localization. The scope of protection for this method is not limited to the above embodiments; all technical solutions falling within this conceptual framework are within the scope of protection of this invention. It should be noted that for those skilled in the art, any improvements and variations made without departing from the principles of this invention should also be considered within the scope of protection of this invention.

Claims

1. A feature preprocessing and extraction method for multi-channel audio localization, characterized by: The method includes the following steps: Step 1: After performing DSB spatial mapping, extract LogMel features and IV features. Perform global normalization on the extracted features. Use the normalized data for training and testing on deep learning. Step 2: Accelerate the process using the DSB edge acceleration method, and use the extracted features to train a CRNN model for sound source direction estimation.

2. The method according to claim 1, characterized in that: Step 1 specifically involves: set up On a unit sphere centered at the array center The set of positions, for polar coordinates The delay of the assumed sound source on the array is given by the following formula, given the received signal. , No. The DSB of each channel is represented by the following formula:

3. The method according to claim 2, characterized in that: when With source If they overlap, then Consistent accumulation in the array output.

4. The method according to claim 3, characterized in that: The steps for DSB-based spatial mapping are as follows: Select in the monitoring area The location of the virtual sound source is denoted as These points are evenly distributed on a unit sphere; For each virtual source exist Calculate its array vector ; Given an array input calculate ,in Sampling frequency, Sample index The number of samples in each window segment and the score delay rounded to the nearest sample; Based on DSB, spatial mapping is Output of virtual sound source location set in -channel time series, called For each channel, frequency domain audio features are further extracted from the data in that channel.

5. The method according to claim 4, characterized in that: To determine the amplitude and phase spectra, the Discrete Short-Time Fourier Transform (STFT) is applied to windowed data to calculate the amplitude and phase spectra, and the results are presented. The discrete STFT of data in the channel is expressed by the following formula: in, It is a window function, and in the implementation, we use a Hamming window with a length of 2048; Determining the Log-Mel spectrogram involves using a filter bank that simulates a human audio front end to analyze the incoming signal. The obtained data is used to calculate the logarithmic Mel spectrum by taking the logarithm of the Mel spectrum. Channel The -th log Mel spectrum is as follows: in, For Mel filter banks; The intensity vector (IV) of a multi-channel audio signal is determined by calculating the correlation between the signal in each channel and the reference channel. For multiple signals after DSB, the reference channel audio is obtained by directly accumulating the received signals without delay. For any channel... The initial intensity vector of the audio after DSB is represented by the following formula: in, It is a channel STFT characteristics, It is the STFT characteristic of the reference audio. It is the complex conjugate returned by the element. It is the real part of the complex number returned by the element; the extracted intensity vector features are further normalized by dividing each element by the square root of the sum of squares of the audio data after dividing by all DSBs; the square root of the sum of squares of the audio data after DSBs is: By multiplying with the Mel filter bank, the luminance vector features are transformed to the Mel domain. After DSB, any channel... The audio intensity vector is: Among them, For Mel filter banks.

6. The method according to claim 5, characterized in that: Step 2 specifically involves: Compute delayed summation, SFTF, and LogMel features on a GPU using the Cupy library; in `delay` and `sum`, when the summation is delayed by the same amount of time, the resulting summation is also delayed by the same amount of time, considering locations... and The signals received by the two microphones and For any position where the unit sphere intersects with the hyperbola that satisfies the condition... The delay sum is in the form of: ， ， For different Delayed and moved These parts are stored and reused in different locations to further accelerate the computation of DSB.

7. The method according to claim 6, characterized in that: When there are four microphones and two points in the space and After delay calculation, the delays corresponding to the two points are respectively and Redundancy exists in the following cases: Case 1: When it is ,calculate The delays of the x and y microphones at a point are summed and stored in memory, then the delay results of the other microphones are added together; at the point... When, call directly The sum of the delay results from the x and y microphones at each location is then added together with the delay results from the other microphones. Scenario 2: When The calculation process is similar to that of case 1. In the overall calculation process, case 1 is given priority, while case 2 directly calls the result from case 1.

8. A feature preprocessing and extraction system for multi-channel audio localization, characterized by: The system includes: The feature extraction module performs DSB spatial mapping and extracts LogMel and IV features. The extracted features are then globally normalized, and the normalized data is used for training and testing in deep learning. The sound source estimation module accelerates the process using the DSB edge acceleration method and uses the extracted features to train a CRNN model for sound source direction estimation.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the method according to any one of claims 1-7.

Images