A method for locating multiple sound sources based on a sound spectrum

Abstract

A multi-sound source positioning method based on a sound spectrum, characterized in that the method comprises: a data acquisition module acquires original audio data in a field environment in real time; a preprocessing module performs a preprocessing operation on the original audio data to generate a normalized spectrum graph; a feature extraction unit in a sound source positioning module extracts spectral features from the normalized spectrum graph to obtain spectral feature data; a frequency dimension compression unit compresses the frequency dimension of the spectral feature data through adaptive average pooling to obtain compressed features; an encoder encodes the compressed features to obtain encoded data; a decoder decodes the encoded data to obtain decoded data; a sound source number prediction unit predicts the number of sound sources from the decoded data to obtain a sound source number result; and a three-dimensional coordinate prediction unit predicts three-dimensional coordinates from the decoded data to obtain a three-dimensional coordinate result. Effect: The method can accurately capture the spatial characteristics of sound sources, has good real-time performance, and has high positioning accuracy.

Description

Technical Field

[0001] This invention relates to the fields of acoustic signal processing and deep learning technology, and in particular to a multi-source localization method based on the sound spectrum. Background Technology

[0002] Traditional sound source localization techniques, such as those based on Time Difference of Arrival (TDOA) or beamforming, often rely on single-channel audio or complex signal processing, facing limitations in accuracy and computational latency in multi-source scenarios. Key limitations include: limited single-channel audio information, making it difficult to capture spatial features; complex preprocessing increasing system overhead, making it unsuitable for real-time applications; and the inability of single-task models to jointly optimize the prediction of the number and location of sound sources. In recent years, the DETR end-to-end object detection framework in deep learning has provided a new approach to multi-task processing, but its application in the audio field remains immature.

[0003] Disadvantages of existing technologies: Existing sound source localization technologies suffer from problems such as a single sound source channel, difficulty in capturing spatial characteristics of the sound source, poor real-time performance, and insufficient positioning accuracy. Summary of the Invention

[0004] The present invention provides a multi-sound source localization method based on the sound spectrum, which can accurately capture the spatial features of sound sources, with good real-time performance and high localization accuracy.

[0005] To achieve the above objectives, the present invention provides a multi-source localization method based on sound spectrum, the key of which includes the following steps:

[0006] Step 1: Construct a multi-source localization system based on sound spectrum. The multi-source localization system is equipped with a data acquisition module, a preprocessing module and a sound source localization module connected in sequence. The sound source localization module is equipped with a feature extraction unit, a frequency dimension compression unit, a Conformer encoder and a Transformer decoder connected in sequence. The output of the Transformer decoder is connected to a sound source number prediction unit and a three-dimensional coordinate prediction unit.

[0007] Step 2: The data acquisition module collects raw audio data 'a' from the field environment in real time and sends it to the preprocessing module;

[0008] Step 3: The preprocessing module performs preprocessing operations on the original audio data a to generate a normalized spectrogram b, and sends it to the sound source localization module;

[0009] Step 4: The feature extraction unit in the sound source localization module performs spectral feature extraction on the normalized spectrum map b to obtain spectral feature data c, and sends it to the frequency dimension compression unit.

[0010] Step 5: The frequency dimension compression unit compresses the frequency dimension of the spectral feature data c through adaptive average pooling to obtain compressed feature d, and sends it to the Conformer encoder;

[0011] Step 6: The Conformer encoder encodes the compressed feature d to obtain encoded data e, and sends it to the Transformer decoder;

[0012] Step 7: The Transformer decoder decodes the encoded data e to obtain decoded data f, and sends it to the sound source quantity prediction unit and the three-dimensional coordinate prediction unit;

[0013] Step 8a: The sound source number prediction unit performs sound source number prediction on the decoded data f to obtain the sound source number result;

[0014] Step 8b: The three-dimensional coordinate prediction unit performs three-dimensional coordinate prediction on the decoded data f to obtain the three-dimensional coordinate result.

[0015] Through the above design, this invention acquires multi-channel raw audio data with spatial feature information via a data acquisition module. Then, a spectrogram is obtained through preprocessing. A lightweight MobileNetV3-Small network, constructed using a feature extraction unit and a frequency dimension compression unit, is then combined with a Conformer encoder to build an end-to-end target detection DETR framework. This enables the prediction of the number of multiple sound sources and their 3D coordinate localization in a field environment. This method significantly improves localization accuracy and computational efficiency through efficient feature extraction and multi-task optimization, making it suitable for real-time sound source monitoring in the field of ecological protection.

[0016] Preferably, the data acquisition module is an N-channel microphone array used to acquire raw audio data from N channels. The audio data from the N channels together constitute the input data for the sound source localization module, enabling accurate identification and localization of sound source information from all channels and improving the effective recognition range of the system.

[0017] Preferably, in step 3, the preprocessing module performs preprocessing operations on the raw audio data, including the following steps:

[0018] Step 3.1: The preprocessing module first performs a Short Time Fourier Transform (STFT) on the original audio data a to obtain the initial spectrogram. The expression is:

[0019] ;

[0020] in, This is the initial spectrum. For the current time, For the original audio data a, For time window functions, such as Hamming window, Gaussian window, etc.; This is the time offset. For frequency, The base is the natural number;

[0021] Step 3.2: Then, the initial spectrum is denoised using spectral subtraction and Gaussian filtering to obtain a denoised spectrum, expressed as:

[0022] Spectral subtraction: ;

[0023] Gaussian filtering: ;

[0024] in, It is a constant. , For noise, It is a function with maximum value. The audio data after noise reduction. For Gaussian kernel, Standard deviation;

[0025] Step 3.3: Finally, analyze the denoised spectrum. After performing logarithmic power spectrum transformation and normalization, the normalized spectrum b is obtained, expressed as:

[0026] Logarithmic power spectrum conversion:

[0027] Normalization: ;

[0028] in, It is a very small constant used to prevent the logarithm from being zero. The mean of the spectrum, This represents the standard deviation of the spectrum.

[0029] After noise reduction processing, the obtained The amplitudes are often still in linear amplitude space. To enable the model to better capture the dynamic range of audio and mimic human hearing, the amplitudes are usually logarithmically taken. To generate the final normalized spectrogram b, the data needs to be scaled to a numerical range that is easily handled by the neural network, i.e., normalization is performed.

[0030] Through the above design, the preprocessing module optimizes the model input by performing spectrum conversion and noise reduction on the multi-channel audio signal.

[0031] Preferably, the feature extraction unit is provided with an input convolutional layer, the input end of which is used to obtain the normalized spectrum b, and the output end of the input convolutional layer is connected to an inverted residual block;

[0032] The inverted residual block contains n inverted residual layers connected end to end. The n inverted residual layers have the same structure. Each inverted residual layer is provided with an input layer, an extended convolutional layer, a depthwise separable convolutional layer, a projective convolutional layer, and a residual connection layer connected in sequence. The output of the input layer is also connected to the input of the residual connection layer.

[0033] Preferably, the frequency dimension compression unit is provided with an adaptive average pooling layer and an upscaling output layer connected in sequence. The adaptive average pooling layer is used to compress the frequency dimension of the spectral feature data c through adaptive average pooling, and the upscaling output layer is used to perform upscaling processing on the compressed data to finally obtain the compressed feature d.

[0034] Preferably, the Conformer encoder has m identical coding layers connected end to end. Each coding layer has a first feedforward module, a multi-head self-attention module, a convolution module, a second feedforward module, and a final normalization layer connected in sequence. A residual connection layer is provided between the input and output of the first feedforward module, the multi-head self-attention module, the convolution module, and the second feedforward module.

[0035] Preferably, the first feedforward module is used to obtain the compressed feature d or the output feature of the previous coding layer, and then perform feedforward processing and residual connection, as expressed by:

[0036] ;

[0037] ;

[0038] Where i is the coding layer index, i∈[1,m]; The output feature tensor is the result of the first feedforward module and the residual connection. Let be the input feature tensor of the i-th coding layer, and be the compressed feature d or the output feature of the previous coding layer; This is a half-step scaling factor used to smooth network transformations. This is the first feedforward neural network function; This is the weight matrix of the first layer linear mapping in the first feedforward module. This is the bias vector of the first layer linear mapping in the first feedforward module. This is the weight matrix of the second-layer linear mapping in the first feedforward module. This is the bias vector of the second-layer linear mapping in the first feedforward module; This is a linear rectification activation function used to introduce nonlinearity;

[0039] The output expression of the multi-head self-attention module is:

[0040]

[0041]

[0042] in, Let be the output feature tensor of the multi-head self-attention module in the i-th coding layer. For multi-head self-attention mechanism functions, For the splicing operation, the features calculated by multiple independent attention heads are spliced ​​together along the channel dimension. For standard scaled dot product attention functions, For querying the matrix, The key matrix, For value matrices, Let h be the query linear projection weight matrix for the h-th attention head. Let h be the key linear projection weight matrix of the h-th attention head. The linear projection weight matrix is ​​the value of the h-th attention head. The output projection matrix is ​​used to perform the final linear fusion and integration of the concat-stitched multi-head features;

[0043] The output expression of the convolution module is:

[0044]

[0045] in, The output feature tensor of the convolution module This is a convolution operation;

[0046] The output expression of the first feedforward module is:

[0047]

[0048] in, The output feature tensor of the first feedforward module, This is the second feedforward neural network function;

[0049] The output expression of the final normalization layer is:

[0050]

[0051] in, Let be the output feature of the i-th coding layer. When it is the last coding layer, the output feature is the coded data e. This is the layer normalization function.

[0052] Preferably, the Transformer decoder has M identical decoding layers connected end-to-end, with each decoding layer connected to the output of the Transformer encoder; the Transformer decoder decodes the encoded data e as follows:

[0053] The j-th decoding layer obtains the encoded data e, and then calculates the output query vector of the (j-1)-th decoding layer. Similarity matrix between the encoded data e and the data e The expression is:

[0054] ;

[0055] Then, the normalized attention weight matrix is ​​calculated using static EMA decay weights and dimensional scaling. The expression is:

[0056] ;

[0057] ;

[0058] Finally, by fusing attention-weighted features and global average features, the output query vector of the j-th decoding layer is calculated. The expression is:

[0059] ;

[0060] in, This is the output query vector of the j-th decoding layer; the initial query vector is constructed by the first decoding layer through initialization. Then, a similarity matrix is ​​calculated; the output of the final decoding layer is the query vector. The decoded data f is passed to the sound source number prediction unit and the three-dimensional coordinate prediction unit; j is the decoding layer index, j∈[1,M]; the superscript T is the matrix transpose. For matrix multiplication, To calculate the global average of the encoded feature e over the time dimension; This indicates the generation of numbers from 0 to... Time step index sequence; It is a natural exponential function. Sum the elements of the sequence.

[0061] Preferably, the sound source number prediction unit uses global average features to predict the total number of sound sources, and the output expression is:

[0062] ;

[0063] The output expression of the three-dimensional coordinate prediction unit is:

[0064] ;

[0065] in, For the predicted three-dimensional coordinates of the sound source, The predicted number of sound sources, To encode features Calculate the global average over the time dimension. It is a linear function. To calculate the global average of the decoded data f over time; the number 2 indicates that the input global features are mapped to a 2-dimensional output vector. In quantity prediction tasks, this usually means that the network outputs two values ​​to represent the number of sound sources. The number 3 indicates that the query features are mapped to a 3-dimensional output vector. These three dimensions directly and precisely correspond to the three-dimensional coordinates (X, Y, Z) of the sound source in space.

[0066] Preferably, the sound source localization module uses a hybrid loss function for iterative training, and the expression of the hybrid loss function is:

[0067] ;

[0068] ;

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] in, For the total loss, For position regression loss, For object-oriented loss, For quantity prediction loss, For regularization loss, The actual number of sound sources. Indexing for real sound sources For the predicted index assigned to the k-th true target, For the predicted 3D coordinates of the pair, For the corresponding true three-dimensional coordinates, As background weight, The binary cross-entropy loss after equal weighting, for Activation function This is used to predict the confidence level of the presence of a real sound source at a given location, and to distinguish the real sound source from background noise. For the target category label, This represents the total number of sound sources predicted by the network. For pairing probability, For a single weight, It is the set of all weight parameters of the network. For weight The square of the L2 norm;

[0074] The matching loss expression for the sound source localization module is:

[0075] ;

[0076] in, Let k be the k-th true target; the matching loss is used to find the optimal binary match between a fixed set of predictions and the set of true targets.

[0077] The cost matrix C of the sound source localization module is:

[0078] ;

[0079] in, For predicting coordinates with actual coordinates The L1 norm distance between them is represented by a unit pixel. This cost matrix is ​​used to minimize the one-to-one correspondence between the predicted result and the true label during forward propagation, and does not participate in backpropagation.

[0080] The beneficial effects of this invention are:

[0081] (1) By building an end-to-end audio DETR framework, the joint prediction of the number of sound sources and three-dimensional coordinates was realized, which simplified the prediction process.

[0082] (2) The lightweight MobileNetV3-Small network built by the feature extraction unit and the frequency dimension compression unit effectively reduces the number of parameters by about 50%, adapts to resource-constrained devices, and can be applied to more prediction scenarios.

[0083] (3) By using one-dimensional depthwise convolution and multi-head self-attention in Conformer encoding, efficient modeling of global relationships of multi-channel features is achieved, which effectively improves the accuracy of prediction results.

[0084] (4) By combining Hungarian matching, MSE regression and quantity prediction loss, the loss function was optimized, which improved the localization performance of the sound source localization module. Attached Figure Description

[0085] Figure 1 This is a block diagram of the multi-sound source localization system in the embodiment;

[0086] Figure 2 This is a block diagram of the feature extraction unit and the frequency dimension compression unit in the embodiment;

[0087] Figure 3 This is a block diagram of the Conformer encoder structure in the embodiment. Detailed Implementation

[0088] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples. The following embodiments or drawings are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0089] A spectral-based multi-source localization method includes the following steps:

[0090] Step 1: Construct a spectrum-based multi-source localization system. This system includes a data acquisition module, a preprocessing module, and a sound source localization module connected in sequence. The sound source localization module includes a feature extraction unit, a frequency dimension compression unit, a Conformer encoder, and a Transformer decoder connected in sequence. The output of the Transformer decoder is connected to a sound source quantity prediction unit and a three-dimensional coordinate prediction unit, such as... Figure 1 As shown;

[0091] Step 2: The data acquisition module collects raw audio data 'a' from the field environment in real time and sends it to the preprocessing module;

[0092] Step 3: The preprocessing module performs preprocessing operations on the original audio data a to generate a normalized spectrogram b, and sends it to the sound source localization module;

[0093] Step 4: The feature extraction unit in the sound source localization module performs spectral feature extraction on the normalized spectrum map b to obtain spectral feature data c, and sends it to the frequency dimension compression unit.

[0094] Step 5: The frequency dimension compression unit compresses the frequency dimension of the spectral feature data c through adaptive average pooling to obtain compressed feature d, and sends it to the Conformer encoder;

[0095] Step 6: The Conformer encoder encodes the compressed feature d to obtain encoded data e, and sends it to the Transformer decoder;

[0096] Step 7: The Transformer decoder decodes the encoded data e to obtain decoded data f, and sends it to the sound source quantity prediction unit and the three-dimensional coordinate prediction unit;

[0097] Step 8a: The sound source number prediction unit performs sound source number prediction on the decoded data f to obtain the sound source number result;

[0098] Step 8b: The three-dimensional coordinate prediction unit performs three-dimensional coordinate prediction on the decoded data f to obtain the three-dimensional coordinate result.

[0099] The data acquisition module is an N-channel microphone array used to acquire N channels of raw audio data.

[0100] In step 3, the preprocessing module performs preprocessing operations on the raw audio data, including the following steps:

[0101] Step 3.1: The preprocessing module first performs a Short Time Fourier Transform (STFT) on the original audio data a to obtain the initial spectrogram. The expression is:

[0102] ;

[0103] in, This is the initial spectrum. For the current time, For the original audio data a, For time window functions, such as Hamming window, Gaussian window, etc.; This is the time offset. For frequency, The base is the natural number;

[0104] Step 3.2: Then, the initial spectrum is denoised using spectral subtraction and Gaussian filtering to obtain a denoised spectrum, expressed as:

[0105] Spectral subtraction: ;

[0106] Gaussian filtering: ;

[0107] in, It is a constant. , For noise, It is a function with maximum value. The audio data after noise reduction. For Gaussian kernel, Standard deviation;

[0108] Step 3.3: Finally, analyze the denoised spectrum. After performing logarithmic power spectrum transformation and normalization, the normalized spectrum b is obtained, expressed as:

[0109] Logarithmic power spectrum conversion:

[0110] Normalization: ;

[0111] in, It is a very small constant used to prevent the logarithm from being zero. The mean of the spectrum, This represents the standard deviation of the spectrum.

[0112] like Figure 2 As shown, the feature extraction unit is provided with an input convolutional layer, the input of which is used to obtain the normalized spectrum b, and the output of which is connected to an inverted residual block.

[0113] The inverted residual block contains n inverted residual layers connected end to end. The n inverted residual layers have the same structure. Each inverted residual layer is provided with an input layer, an extended convolutional layer, a depthwise separable convolutional layer, a projective convolutional layer, and a residual connection layer connected in sequence. The output of the input layer is also connected to the input of the residual connection layer.

[0114] The frequency dimension compression unit is provided with an adaptive average pooling layer and a dimension-upgrading output layer connected in sequence. The adaptive average pooling layer is used to compress the frequency dimension of the spectral feature data c through adaptive average pooling, and the dimension-upgrading output layer is used to perform dimension-upgrading processing on the compressed data to finally obtain the compressed feature d.

[0115] The Conformer encoder has m identical coding layers connected sequentially. Each coding layer includes a first feedforward module, a multi-head self-attention module, a convolutional module, a second feedforward module, and a final normalization layer, all connected sequentially. Residual connection layers are provided between the input and output of the first feedforward module, the multi-head self-attention module, the convolutional module, and the second feedforward module. Figure 3 As shown.

[0116] The first feedforward module is used to obtain the compressed feature d or the output feature of the previous coding layer, and then perform feedforward processing and residual connection, as expressed in the following expression:

[0117] ;

[0118] ;

[0119] Where i is the coding layer index, i∈[1,m]; The output feature tensor is the result of the first feedforward module and the residual connection. Let be the input feature tensor of the i-th coding layer, and be the compressed feature d or the output feature of the previous coding layer; This is a half-step scaling factor used to smooth network transformations. This is the first feedforward neural network function; This is the weight matrix of the first layer linear mapping in the first feedforward module. This is the bias vector of the first layer linear mapping in the first feedforward module. This is the weight matrix of the second-layer linear mapping in the first feedforward module. This is the bias vector of the second-layer linear mapping in the first feedforward module; This is a linear rectification activation function used to introduce nonlinearity;

[0120] The output expression of the multi-head self-attention module is:

[0121]

[0122]

[0123] in, Let be the output feature tensor of the multi-head self-attention module in the i-th coding layer. For multi-head self-attention mechanism functions, For the splicing operation, the features calculated by multiple independent attention heads are spliced ​​together along the channel dimension. For standard scaled dot product attention functions, For querying the matrix, The key matrix, For value matrices, Let h be the query linear projection weight matrix for the h-th attention head. Let h be the key linear projection weight matrix of the h-th attention head. The linear projection weight matrix is ​​the value of the h-th attention head. The output projection matrix is ​​used to perform the final linear fusion and integration of the concat-stitched multi-head features;

[0124] The output expression of the convolution module is:

[0125]

[0126] in, The output feature tensor of the convolution module This is a convolution operation;

[0127] The output expression of the first feedforward module is:

[0128]

[0129] in, The output feature tensor of the first feedforward module, This is the second feedforward neural network function;

[0130] The output expression of the final normalization layer is:

[0131]

[0132] in, Let be the output feature of the i-th coding layer. When it is the last coding layer, the output feature is the coded data e. This is the layer normalization function.

[0133] The Transformer decoder has M identical decoding layers connected end-to-end, each layer connected to the output of the Transformer encoder. The Transformer decoder decodes the encoded data e as follows:

[0134] The j-th decoding layer obtains the encoded data e, and then calculates the output query vector of the (j-1)-th decoding layer. Similarity matrix between the encoded data e and the data e The expression is:

[0135] ;

[0136] Then, the normalized attention weight matrix is ​​calculated using static EMA decay weights and dimensional scaling. The expression is:

[0137] ;

[0138] ;

[0139] Finally, by fusing attention-weighted features and global average features, the output query vector of the j-th decoding layer is calculated. The expression is:

[0140] ;

[0141] in, This is the output query vector of the j-th decoding layer; the initial query vector is constructed by the first decoding layer through initialization. Then, a similarity matrix is ​​calculated; the output of the final decoding layer is the query vector. The decoded data f is passed to the sound source number prediction unit and the three-dimensional coordinate prediction unit; j is the decoding layer index, j∈[1,M]; the superscript T is the matrix transpose. For matrix multiplication, To calculate the global average of the encoded feature e over the time dimension; This indicates the generation of numbers from 0 to... Time step index sequence; It is a natural exponential function. Sum the elements of the sequence.

[0142] The sound source quantity prediction unit uses global average features to predict the total number of sound sources, and the output expression is:

[0143] ;

[0144] The output expression of the three-dimensional coordinate prediction unit is:

[0145] ;

[0146] in, For the predicted three-dimensional coordinates of the sound source, The predicted number of sound sources, To encode features Calculate the global average over the time dimension. It is a linear function. To calculate the global average of the decoded data f over time; the number 2 indicates that the input global features are mapped to a 2-dimensional output vector. In quantity prediction tasks, this usually means that the network outputs two values ​​to represent the number of sound sources. The number 3 indicates that the query features are mapped to a 3-dimensional output vector. These three dimensions directly and precisely correspond to the three-dimensional coordinates (X, Y, Z) of the sound source in space.

[0147] This invention acquires multi-channel raw audio data with spatiotemporal characteristics using a multi-channel microphone array, then generates spectral features through short-time Fourier transform, and extracts these features using a lightweight neural network, MobileNetV3-Small. Combined with a convolutionally enhanced Conformer encoder for speech recognition and the DETR framework, an end-to-end target detection framework based on Transformer, this method achieves end-to-end number prediction and 3D localization of multiple sound sources. This approach fills the gaps in real-time performance and accuracy compared to traditional methods, significantly reduces computational costs, is suitable for field monitoring scenarios, and provides an efficient solution for ecological protection.

[0148] This method uses a multi-channel microphone array to collect outdoor audio data to train the sound localization module. The sampling rate is 44.1 kHz, and the data includes various sound sources such as birdsong, animal roars, and human voices. The dataset contains 6000 audio segments, each 5 seconds long, labeled with 0-3 sound sources and their 3D coordinates. Labeling was completed using sound source localization software and manual verification to ensure accuracy. To enhance the model's robustness, data augmentation was employed.

[0149] Gaussian noise: Add noise with a signal-to-noise ratio of 10-20 dB to enhance robustness.

[0150] Time masking: Randomly mask segments of 0.1-0.5 seconds.

[0151] Volume adjustment: Randomly adjust the volume by ±15%.

[0152] The dataset is divided into training, validation, and test sets in an 8:1:1 ratio.

[0153] For end-to-end distance prediction and 3D localization of multiple sound sources, this invention designs a multi-task loss function combining a binary matching mechanism. Its calculation process involves first estimating the matching segmentation and then calculating the loss:

[0154] 1. Matching Loss: The Hungarian algorithm is used to find the optimal binary match between a fixed number of prediction sets and the true target set. The matching loss expression is:

[0155] ;

[0156] in, Let k be the k-th true target; the matching loss is used to find the optimal binary match between a fixed set of predictions and the set of true targets.

[0157] To find the optimal matching, construct the pairing cost matrix C:

[0158] ;

[0159] in, For predicting coordinates with actual coordinates The L1 norm distance between them is represented by a unit pixel. This cost matrix is ​​used to minimize the one-to-one correspondence between the predicted result and the true label during forward propagation, and does not participate in backpropagation.

[0160] The sound source localization module uses a hybrid loss function for iterative training. The expression for the hybrid loss function is as follows:

[0161] ;

[0162] ;

[0163] ;

[0164] ;

[0165] ;

[0166] ;

[0167] in, For the total loss, For position regression loss, For object-oriented loss, For quantity prediction loss, For regularization loss, The actual number of sound sources. Indexing for real sound sources For the predicted index assigned to the k-th true target, For the predicted 3D coordinates of the pair, For the corresponding true three-dimensional coordinates, The background weight is used when the prediction result matches the background, i.e., there is no sound source. When the prediction result matches the foreground sound source, Background weight Used to alleviate the imbalance between positive and negative samples. The binary cross-entropy loss after equal weighting, for Activation function This is used to predict the confidence level of the presence of a real sound source at a given location, and to distinguish the real sound source from background noise. The target category label is set to 1 for the sound source and 0 for the background. This represents the total number of sound sources predicted by the network. For pairing probability, For a single weight, It is the set of all weight parameters of the network. For weight The square of the L2 norm.

[0168] The sound source localization model was trained using the AdamW optimizer with a learning rate of 0.0001, a momentum of 0.9, for 150 epochs, a batch size of 32, a cosine annealing scheduling period of 40 epochs, and a minimum learning rate of 0.00001. The training was performed on a GPU, and the test set results are as follows:

[0169] The quantity prediction accuracy is 99.9%, and the positioning error is 0.6 meters.

[0170] The model is deployed on an embedded device in the field, processes multi-channel spectral signals, and outputs in JSON format.

[0171] {

[0172] "count": 2,

[0173] "coords": [[10.0, 5.0, 2.0], [8.0, 3.0, 1.5]]

[0174] }

[0175] Output results in real time, adapted for real-time monitoring.

[0176] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-source localization method based on sound spectrum, characterized in that, Includes the following steps: Step 1: Construct a multi-source localization system based on sound spectrum. The multi-source localization system is equipped with a data acquisition module, a preprocessing module and a sound source localization module connected in sequence. The sound source localization module is equipped with a feature extraction unit, a frequency dimension compression unit, a Conformer encoder and a Transformer decoder connected in sequence. The output of the Transformer decoder is connected to a sound source number prediction unit and a three-dimensional coordinate prediction unit. Step 2: The data acquisition module collects raw audio data 'a' from the field environment in real time and sends it to the preprocessing module; Step 3: The preprocessing module performs preprocessing operations on the original audio data a to generate a normalized spectrogram b, and sends it to the sound source localization module; Step 4: The feature extraction unit in the sound source localization module performs spectral feature extraction on the normalized spectrum map b to obtain spectral feature data c, and sends it to the frequency dimension compression unit. Step 5: The frequency dimension compression unit compresses the frequency dimension of the spectral feature data c through adaptive average pooling to obtain compressed feature d, and sends it to the Conformer encoder; Step 6: The Conformer encoder encodes the compressed feature d to obtain encoded data e, and sends it to the Transformer decoder; Step 7: The Transformer decoder decodes the encoded data e to obtain decoded data f, and sends it to the sound source quantity prediction unit and the three-dimensional coordinate prediction unit; Step 8a: The sound source number prediction unit performs sound source number prediction on the decoded data f to obtain the sound source number result; Step 8b: The three-dimensional coordinate prediction unit performs three-dimensional coordinate prediction on the decoded data f to obtain the three-dimensional coordinate result.

2. The multi-source localization method based on the sound spectrum according to claim 1, characterized in that: The data acquisition module is an N-channel microphone array used to acquire N channels of raw audio data.

3. The multi-source localization method based on the sound spectrum according to claim 1, characterized in that: In step 3, the preprocessing module performs preprocessing operations on the raw audio data, including the following steps: Step 3.1: The preprocessing module first performs a Short Time Fourier Transform (STFT) on the original audio data a to obtain the initial spectrogram. The expression is: ； in, This is the initial spectrum. For the current time, For the original audio data a, For time window functions; This is the time offset. For frequency, The base is the natural number; Step 3.2: Then, the initial spectrum is denoised using spectral subtraction and Gaussian filtering to obtain a denoised spectrum, expressed as: Spectral subtraction: ; Gaussian filtering: ; in, It is a constant. For noise, It is a function with maximum value. The audio data after noise reduction. For Gaussian kernel, Standard deviation; Step 3.3: Finally, analyze the denoised spectrum. After performing logarithmic power spectrum transformation and normalization, the normalized spectrum b is obtained, expressed as: Logarithmic power spectrum conversion: Normalization: ; in, It is a constant. The mean of the spectrum, This represents the standard deviation of the spectrum.

4. The multi-source localization method based on spectral density according to claim 1, characterized in that: The feature extraction unit is provided with an input convolutional layer, the input of which is used to obtain the normalized spectrum b, and the output of which is connected to an inverted residual block. The inverted residual block contains n inverted residual layers connected end to end. The n inverted residual layers have the same structure. Each inverted residual layer is provided with an input layer, an extended convolutional layer, a depthwise separable convolutional layer, a projective convolutional layer, and a residual connection layer connected in sequence. The output of the input layer is also connected to the input of the residual connection layer.

5. The multi-source localization method based on the sound spectrum according to claim 1, characterized in that: The frequency dimension compression unit is provided with an adaptive average pooling layer and a dimension-upgrading output layer connected in sequence. The adaptive average pooling layer is used to compress the frequency dimension of the spectral feature data c through adaptive average pooling, and the dimension-upgrading output layer is used to perform dimension-upgrading processing on the compressed data to finally obtain the compressed feature d.

6. The multi-source localization method based on spectral density according to claim 1, characterized in that: The Conformer encoder has m identical coding layers connected end to end. Each coding layer has a first feedforward module, a multi-head self-attention module, a convolution module, a second feedforward module, and a final normalization layer connected in sequence. A residual connection layer is provided between the input and output of the first feedforward module, the multi-head self-attention module, the convolution module, and the second feedforward module.

7. The multi-source localization method based on the sound spectrum according to claim 6, characterized in that: The first feedforward module is used to obtain the compressed feature d or the output feature of the previous coding layer, and then perform feedforward processing and residual connection, as expressed in the following expression: ； ； Where i is the coding layer index, i∈[1,m]; The output feature tensor is the result of the first feedforward module and the residual connection. Let be the input feature tensor of the i-th coding layer. It is a half-step scaling factor. This is the first feedforward neural network function; This is the weight matrix of the first layer linear mapping in the first feedforward module. This is the bias vector of the first layer linear mapping in the first feedforward module. This is the weight matrix of the second-layer linear mapping in the first feedforward module. This is the bias vector of the second-layer linear mapping in the first feedforward module; It is a linear rectification activation function; The output expression of the multi-head self-attention module is: in, Let be the output feature tensor of the multi-head self-attention module in the i-th coding layer. For multi-head self-attention mechanism functions, For splicing operations, For scaling dot product attention functions, For querying the matrix, The key matrix, For value matrices, Let h be the query linear projection weight matrix for the h-th attention head. Let h be the key linear projection weight matrix of the h-th attention head. The linear projection weight matrix is ​​the value of the h-th attention head. Output projection matrix; The output expression of the convolution module is: in, The output feature tensor of the convolution module This is a convolution operation; The output expression of the first feedforward module is: in, The output feature tensor of the first feedforward module, This is the second feedforward neural network function; The output expression of the final normalization layer is: in, Let be the output feature of the i-th coding layer. When it is the last coding layer, the output feature is the coded data e. This is the layer normalization function.

8. The multi-source localization method based on spectral density according to claim 1, characterized in that: The Transformer decoder has M identical decoding layers connected end-to-end, each layer connected to the output of the Transformer encoder. The Transformer decoder decodes the encoded data e as follows: The j-th decoding layer obtains the encoded data e, and then calculates the output query vector of the (j-1)-th decoding layer. Similarity matrix between the encoded data e and the data e The expression is: ； Then, the normalized attention weight matrix is ​​calculated using static EMA decay weights and dimensional scaling. The expression is: ； ； Finally, by fusing attention-weighted features and global average features, the output query vector of the j-th decoding layer is calculated. The expression is: ； in, This is the output query vector of the j-th decoding layer; the initial query vector is constructed by the first decoding layer through initialization. Then, a similarity matrix is ​​calculated; the output of the final decoding layer is the query vector. The decoded data f is passed to the sound source number prediction unit and the three-dimensional coordinate prediction unit; j is the decoding layer index, j∈[1,M]; the superscript T is the matrix transpose. For matrix multiplication, To calculate the global average of the encoded feature e over the time dimension; This indicates the generation of numbers from 0 to... Time step index sequence; It is a natural exponential function. Sum the elements of the sequence.

9. The multi-source localization method based on the sound spectrum according to claim 1, characterized in that: The sound source quantity prediction unit uses global average features to predict the total number of sound sources, and the output expression is: ； The output expression of the three-dimensional coordinate prediction unit is: ； in, For the predicted three-dimensional coordinates of the sound source, The predicted number of sound sources, To encode features Calculate the global average over the time dimension. It is a linear function. To calculate the global average of the decoded data f over the time dimension.

10. The multi-source localization method based on the sound spectrum according to claim 1, characterized in that: The sound source localization module uses a hybrid loss function for iterative training. The expression for the hybrid loss function is as follows: ； ； ； ； ； ； in, For the total loss, For position regression loss, For object-oriented loss, For quantity prediction loss, For regularization loss, The actual number of sound sources. Indexing for real sound sources For the predicted index assigned to the k-th true target, For the predicted 3D coordinates of the pair, For the corresponding true three-dimensional coordinates, As background weight, For binary cross-entropy loss, for Activation function To predict the confidence level that a real sound source exists at a location, For the target category label, This represents the total number of sound sources predicted by the network. For pairing probability, For a single weight, It is the set of all weight parameters of the network. For weight The square of the L2 norm; The matching loss expression for the sound source localization module is: ； in, For the k-th real target; The cost matrix C of the sound source localization module is: ； in, For predicting coordinates with actual coordinates The L1 norm distance between them.

Images