Multi-source positioning and detection method based on global-local feature recalibration

Abstract

The application discloses a multi-sound source positioning and detection method based on global-local feature recalibration, which comprises the following steps: calculating the short-time Fourier transform of a multi-channel spatial audio signal in a first-order stereo format, obtaining a log-linear spectrum and a normalized sound intensity vector as input features, and then performing data augmentation on the features of a training set; splicing the augmented spectrum and sound intensity vector as the input of a neural network model, training the neural network model, obtaining optimal network model parameters and saving them; preprocessing a test sample and inputting it into the trained model, outputting the predicted sound event category and position information, drawing a sound event detection graph, a direction angle and an azimuth angle trajectory curve graph according to the prediction result, and comparing them with the visualized image of the test sample real label to analyze the performance of the model. The application can achieve high sound source positioning and detection performance, and the model shows good generalization on real and synthetic data sets.

Description

Technical Field

[0001] This invention belongs to the field of multi-source localization and detection, and particularly relates to a multi-source localization and detection method based on global-local feature recalibration. Background Technology

[0002] Sound source localization and detection can be viewed as a joint task of sound source localization and sound event detection. Specifically, a sound source localization and detection system needs to predict the boundaries of active sound events, identify their categories, and provide the spatial trajectory of the sound source. In recent years, this task has become increasingly popular, aiding in many aspects of everyday applications. For example, robots can better perform human-computer interaction with the assistance of sound source localization and detection systems; sound source localization and detection tasks can collaborate with speech enhancement tasks to denoise a speaker's voice by capturing the speaker's location in a smart conference room; and it can also be applied to real-time environmental sound monitoring in smart cities.

[0003] Research methods for sound source localization and detection tasks can be mainly divided into traditional parameterization methods and deep neural network-based methods. Some popular traditional parameterization methods include: Time-of-Arrival (TDOA) method, Multiple Signal Classification (MUSIC) method, Steering Response Power (SPR) method, and Rotation Invariance (ESPRIT) method.

[0004] In recent years, with the in-depth research of deep learning methods such as neural networks, structures such as convolutional neural networks and recurrent neural networks have shown great promise in the field of sound source localization and detection. The classic SELDnet model based on CRNN (convolutional recurrent neural network) has been widely recognized. As the ratio of overlapping sound events in datasets increases, improving the accuracy of localization and detection in complex sound environments is an urgent need for multi-source localization and detection tasks. Summary of the Invention

[0005] To achieve multi-source detection and localization under conditions where multiple overlapping sound events exist in a sound segment, this invention provides the following solution: a multi-source localization and detection method based on global-local feature recalibration, comprising:

[0006] Calculate the short-time Fourier transform of the multi-channel spatial audio signal in first-order stereo format, obtain the log linear spectrum and normalized sound intensity vector as input features, and then perform data augmentation on the features of the training set.

[0007] The augmented spectrum and sound intensity vector are concatenated and used as input to the neural network model. The neural network model is then trained to obtain and save the optimal network model parameters.

[0008] After preprocessing the test samples, they are fed into the trained model to obtain prediction results. Based on the prediction results, sound event detection maps, direction angle and azimuth trajectory curves are plotted and compared with the visualization images of the real labels of the test samples to analyze the performance of the model.

[0009] Preferably, each first-order stereo signal of the multi-channel spatial audio signal in the first-order stereo format includes four channels (W, X, Y, Z); wherein: W is a 0th-order spherical harmonic used to obtain omnidirectional information, and X, Y, Z are 1st-order spherical harmonics that transmit spatial information along the Cartesian coordinate system of the sound field;

[0010] The log linear spectrum of the spatial audio signal is calculated using the complex spectrum X(t, f), and the formula is as follows:

[0011]

[0012] The expression for the normalized sound intensity vector is:

[0013]

[0014] Wherein: the sound intensity vector is used to transmit valuable information along the direction of sound propagation, and the reverse direction of the sound intensity vector is interpreted as the direction of arrival. The formula expression for the sound intensity vector is:

[0015]

[0016] ρ0 and c represent density and sound speed, respectively. Represents the real part of a complex number. Indicates conjugate.

[0017] Preferably, data augmentation of the features of the training set includes continuously performing data augmentation on each training sample using audio channel swapping, random cropping, and frequency shifting without increasing the amount of data.

[0018] The audio channel switching method is a spatial augmentation method designed for datasets collected by spherical microphones. It represents the directional response of a first-order stereo audio signal using a cosine function, corresponding to the direction of arrival (DOA), which represents the spatial location information of a sound event. Based on the angular deformation of the DOA expressed by the cosine function, for each DOA, a rotation matrix is ​​used to obtain 16 DOA combinations, including the original DOA and 15 new combinations. A combination is randomly selected as the new DOA for each sample.

[0019] The random cropping includes a random cropping method and a SpecAugment-based time-frequency masking method. One of these methods is randomly selected and applied to all channels of the input features for each sample. The random cropping method involves arbitrarily selecting a rectangular region on the spectrogram and setting the values ​​within the region to random values ​​within the range of each channel's values ​​in the linear spectrum. For each channel of the sound intensity vector, all values ​​in the rectangular region are set to 0, similar to a masking operation, with all channels sharing a single mask. The SpecAugment-based time-frequency masking method involves selecting a cross-shaped region and applying the same operation as the random cropping method.

[0020] The frequency shift is a random upward or downward shift of a specific frequency band along the frequency dimension of all channels of the input feature.

[0021] Preferably, the augmented spectrum and sound intensity vector are concatenated as input to the neural network model, and the process of training the neural network model includes:

[0022] The augmented features of the data are used as input to the neural network model; wherein the input features are obtained by concatenating the log linear spectrum and the normalized sound intensity vector, and have seven channels; the dataset contains a label file for each sample, which records the sound event category and sound source location information for supervised learning;

[0023] The input features are initially processed by the Encoder module, and then high-level features containing global and local information are obtained in parallel by the global-local feature extractor.

[0024] Based on the high-level features, the key components in the feature map are emphasized along multiple dimensions by the feature recalibration module to obtain the recalibrated fine features.

[0025] The neural network model ultimately outputs the predicted sound event category and sound source location information along the sound event detection and sound source localization branches, respectively. A joint optimization strategy is adopted to calculate the loss functions of the sound event detection branch and the sound source localization branch by weighting, and obtain the corresponding binary cross-entropy loss and mean square error. The binary cross-entropy loss and mean square error are linearly combined as the final loss function, and the network parameters are updated according to the final loss function.

[0026] Preferably, the process of obtaining high-level features through a global-local feature extractor after the input features have undergone preliminary processing by the Encoder module includes:

[0027] The augmented input features are fed into the network. First, the encoder structure performs preliminary processing on the features for shallow feature extraction. The encoder structure consists of two convolutional layers and an average pooling operation with a kernel size of 2x2. Each convolutional layer contains a convolutional neural network with a kernel size of 3x3, a batch normalization layer, and a Gaussian error linear unit activation function. There are no residual connections after each convolutional layer in the encoder.

[0028] Subsequently, the features are fed into a global-local feature extractor with residuals for in-depth processing. The global-local feature extractor includes a backbone branch consisting of omnidirectional dynamic convolution and multi-scale feature extraction modules, and a local feature extraction unit. The backbone branch extracts global features, and the local feature extraction unit extracts local features. Finally, the attention feature unit selectively fuses the global and local features.

[0029] Preferably, based on the high-level features, the process of obtaining recalibrated fine features by emphasizing key components in the feature map along multiple dimensions using a feature recalibration module includes:

[0030] The feature recalibration module calculates attention along the channel, time, and frequency dimensions to emphasize key channels, time frames, and frequency bands related to the sound source in the features. The resulting fine features are then fed into the sound event detection branch and the direction of arrival estimation branch, respectively.

[0031] Preferably, the global-local feature extractor includes omnidirectional dynamic convolution, a multi-scale feature extraction module, a local feature extraction unit, and an attention feature fusion unit;

[0032] The cross-scale shuffling unit in the multi-scale feature extraction module is used to increase information exchange between multi-scale features.

[0033] The asymmetric convolution in the local feature extraction module is used to extract fine-grained features;

[0034] The neural network model further includes replacing ordinary convolution with group convolution in the multi-scale feature extraction module and adding residual structures to alleviate the overfitting problem.

[0035] Preferably, the omnidirectional dynamic convolution introduces a multi-dimensional attention mechanism, adopts a parallel strategy, learns different attention of the convolution kernel along the spatial dimension of the kernel space, the input channel dimension, the output channel dimension and the convolution kernel dimension, and multiplies it by the convolution kernel ωi in the order of position, channel, filter and kernel.

[0036] The formula for the omnidirectional dynamic convolution is:

[0037]

[0038] in, Represents the attention scalar, , and These represent the three attention weights calculated along the spatial, input channel, and output channel dimensions of the convolution kernel Wi in the kernel space, respectively, and are obtained by the compression-dilation module.

[0039] Preferably, the process by which the cross-scale shuffling unit increases information exchange between multi-scale features includes,

[0040] Channel shuffling facilitates information flow between multi-scale feature maps. This shuffling operation is modeled as a "deform-transpose-deform" process: given the dimension of the input feature (… First, transform it into (n). g s n / g s ), further transposed to ( n / g s g s Finally, it transforms back to its original shape, where n and g s These represent the number of channels and the group size, respectively. Next, the shuffled features are aggregated with the original features through an aggregation block. The aggregation block includes two CNNs with a kernel size of 1x1. The first CNN is used to reduce the number of channels, and the second CNN is used to further fuse the feature maps of different channels, while retaining the information at the original channel positions.

[0041] Compared with the prior art, the present invention has the following advantages and technical effects:

[0042] 1. This invention employs a global-local feature extractor to obtain feature representations containing rich information. Specifically, the multi-scale feature extraction module acquires global information from receptive fields of different sizes, which helps identify sound events with different time-frequency characteristics. Simultaneously, combined with the local feature extraction unit, it focuses on detailed information in the time and frequency dimensions, capturing fine-grained differences within and at the boundaries of the time and frequency features of different sound events, which is more conducive to determining the start and end times of different sound events. This invention also employs an attention feature fusion module in the global-local feature extractor to selectively aggregate global and local features, further reducing feature redundancy.

[0043] 2. This invention employs a feature recalibration module to further refine the features. The module consists of three independent attention branches, which emphasize key features along the time, frequency, and channel dimensions, respectively, effectively combining the advantages of attention in different dimensions to achieve a synergistic effect.

[0044] 3. Experimental results show that the method used in this invention exhibits excellent performance on multiple datasets and has generalization ability for both synthetic and real datasets. The data augmentation method can greatly improve the performance of the model. Attached Figure Description

[0045] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a diagram of the network model based on global-local feature extraction and recalibration in an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the extraction process of the global-local feature extractor in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the structure of the multi-scale feature extraction module in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the cross-scale shuffling unit according to an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the feature recalibration module according to an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the method flow according to an embodiment of the present invention. Detailed Implementation

[0052] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0053] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0054] like Figure 1 and Figure 6As shown, the present invention provides a multi-source localization and detection method based on global-local feature recalibration, which includes sound source localization and sound event detection. The goal of the multi-source localization and detection task is to detect the category and start and end times of sound events in a multi-channel audio signal containing multiple overlapping sound events, and to locate the sound sources. Specifically, it involves a global-local feature extraction and feature recalibration method for detecting the category of active sound events in a multi-channel spatial audio signal and locating them. Specifically, it includes:

[0055] I. Feature Preprocessing Stage: The multi-channel spatial audio signal undergoes a short-time Fourier transform to calculate its log linear spectrum and normalized sound intensity vector, which serve as the original input features. To enhance the model's generalization ability, three data augmentation methods are applied consecutively to the training samples as a data transformation mechanism to increase the complexity of the original input features.

[0056] II. Training Phase: The augmented spectrum and sound intensity vector are concatenated as input to the neural network model. First, the Encoder module performs preliminary feature processing, followed by a global-local feature extractor to obtain more comprehensive high-level features. Next, a feature recalibration module emphasizes key components in the feature map along multiple dimensions, resulting in refined, recalibrated features. Then, a two-layer BiGRU model is used to model temporal context information and obtain the relationships between frames. Finally, the model outputs the predicted sound event category and sound source location information along the sound event detection and sound source localization branches, respectively. A joint optimization strategy is employed, weighted to calculate the loss function for both branches.

[0057] III. Visualization Stage: Save the optimal network model parameters, preprocess the test samples, feed them into the trained model, output the prediction results, and draw the sound event detection map, direction angle and azimuth trajectory curves based on the prediction results. Compare these with the visualization map of the real labels of the test samples to analyze the model's performance.

[0058] Furthermore, the present invention provides a multi-source localization and detection method based on global-local feature recalibration, which includes three steps in implementation: applying various data augmentation methods, training a multi-source localization and detection network model, and visualizing the model's prediction results. For example... Figure 1 The diagram shows a network model based on global-local feature extraction and recalibration. This network is used for both training and testing. During training, the network parameters are updated according to the loss function. During testing, the optimal trained model is used, and the network parameters are fixed. In addition, during network training, a joint optimization strategy is adopted to calculate the loss functions of the sound event detection and sound source localization branches, namely the binary cross-entropy loss and the mean squared error. The loss functions of the two branches are linearly weighted and combined as the final loss function, where the weight of the sound event detection branch is 0.3 and the weight of the sound source localization branch is 0.7.

[0059] Among them: three data augmentation methods were applied.

[0060] The multi-channel spatial audio signal in FOA (first-order stereo) format is first subjected to a short-time Fourier transform, and then its log linear spectrum and normalized sound intensity vector are calculated as the original input features. To increase the generalization ability of the model, various data augmentation methods are used to increase the complexity of the original input features. The data augmentation methods include: 1. audio channel swapping; 2. random pruning; 3. frequency shifting.

[0061] Further optimization of the scheme: FOA (First-Order Stereo) is the primary dataset format used in this invention. Each FOA signal has four channels (W, X, Y, Z), where W is a 0th-order spherical harmonic used to acquire omnidirectional information, and X, Y, and Z are 1st-order spherical harmonics that transmit spatial information along the Cartesian coordinate system of the sound field. The log-linear spectrum of the spatial audio signal can be calculated using the complex spectrum X(t, f):

[0062] (1)

[0063] The sound intensity vector conveys valuable information along the direction of sound propagation, and its inverse direction can usually be interpreted as the direction of arrival (DOA). Sound intensity vectors in FOA format can be directly used for accurate DOA estimation. In the short-time Fourier domain, the sound intensity vector is calculated using the following formula:

[0064] (2)

[0065] Where ρ0 and c represent density and sound speed, respectively. Represents the real part of a complex number. Indicates conjugate.

[0066] The normalized sound intensity vector is expressed as:

[0067] (3)

[0068] Audio Channel Swapping: This is a spatial augmentation method designed for datasets collected by spherical microphones. The directional response of FOA format audio signals can be represented by a cosine function, corresponding to the Direction of Arrival (DOA), which represents the spatial location information of a sound event. Therefore, the cosine function can be used to describe the angular deformation of the DOA. For each DOA, a rotation matrix is ​​used to obtain 16 DOA combinations, including the original DOA and 15 new combinations, and one is randomly selected as the new DOA for each sample. This method preserves the reverberation characteristics of the original data.

[0069] Random cropping: This method includes two approaches: random cropping and SpecAugment-based time-frequency masking. One approach is randomly selected and applied to all channels of the input features for each sample. Specifically, the random cropping method arbitrarily selects a rectangular region on the spectrogram and sets the values ​​within the region to random values ​​within the range of each channel's value in the linear spectrum. For each channel of the sound intensity vector, the values ​​in the rectangular region are all set to 0. This process is similar to a masking operation, with all channels sharing a single mask. The SpecAugment method selects a cross-shaped region and applies the same operation.

[0070] Frequency shifting: This method randomly shifts a certain number of frequency bins up or down along the frequency dimension of all channels of the input features. This method can effectively diversify the frequency patterns of sound events.

[0071] Specifically, the three data augmentation methods described above are continuously applied to each training sample. This approach serves as a data transformation mechanism, increasing data complexity without altering the total number of samples. This introduces complex data into the model, further enhancing its learning ability. Training the model with augmented, challenging samples increases its generalization ability and significantly improves model performance.

[0072] Training a multi-source localization and detection network model

[0073] Step 1: Obtain multiple datasets for sound source localization and detection tasks;

[0074] Specifically, the dataset includes both synthetic and real datasets. Training the network using multiple datasets can improve its robustness in various sound environments. Each dataset can be divided into training, validation, and test sets; the dataset also contains a label file for each sample, recording the sound event category and location information for supervised learning.

[0075] Step train2: Perform short-time Fourier transform on the multi-channel spatial audio signals in FOA format in the dataset, calculate the log linear spectrum and normalized sound intensity vector, and perform data augmentation on them;

[0076] Specifically, only the FOA format dataset is used as the main sample. Short-time Fourier transform is performed on the four-channel spatial audio signal in FOA format to calculate the log linear spectrum of the four channels and the normalized sound intensity vector of the three channels. The two features are concatenated along the channel dimension as the original features of the network. Three data augmentation methods are continuously used to transform the training samples to obtain more complex features.

[0077] Step 3: Feed the augmented input features into the network for training. The training speed depends on the configuration of the machine hardware and the size of the samples.

[0078] Specifically, according to Figure 1 A neural network is built based on a multi-source localization and detection network model using global-local feature extraction and recalibration. Augmented features are used as input to the network, and the network's parameters are updated by calculating a loss function using the network's output and label files. The training samples include both synthetic and real-world data, containing multiple overlapping sound events. This data transformation increases the training difficulty, which is beneficial for improving the model's generalization ability.

[0079] Step train4: Save the optimal network model parameters.

[0080] To further optimize the scheme, the augmented input features are fed into the network. First, the encoder structure performs preliminary processing on the features for shallow feature extraction. The encoder structure consists of two convolutional layers and a 2x2 average pooling operation. Each convolutional layer contains a 3x3 convolutional neural network (CNN), a batch normalization (BN) layer, and a Gaussian error linear unit (GELU) activation function. Notably, there are no residual connections after each convolutional layer in the encoder to avoid noise interference from the original input features. Subsequently, the encoder output is... Figure 1 The portion enclosed by the dashed box is processed for deep feature extraction. This part includes a global-local feature extractor with residuals, a feature recalibration module, and a 2x2 average pooling operation. After processing by the encoder and the portion enclosed by three consecutive dashed boxes, the temporal and frequency dimensions of the intermediate features are downsampled by a factor of 16. Next, a two-layer BiGRU (Bidirectional Gated Recurrent Unit) is used to acquire temporal context information and model the relationships between time frames. Finally, the SED and DOA branches use two fully connected layers to generate sound event categories and DOA estimates, respectively. The SED branch uses the sigmoid activation function, and the DOA branch uses the tanh activation function.

[0081] To further optimize the approach, the augmented features are used as input to the neural network model. The input features are obtained by concatenating the log linear spectrum and the normalized sound intensity vector, and the features have seven channels. The labels of sound event categories and sound source locations in the dataset are used for supervised learning.

[0082] The input features are first processed by a CNN (Convolutional Neural Network) to extract high-level features. To obtain comprehensive task-related information from these features, a global-local feature extractor is used to extract global and local features in parallel. Considering that sound events have different time-frequency characteristics, a multi-scale feature extraction module replaces the single-scale CNN, using convolutional kernels of different sizes to obtain global information. The feature recalibration module emphasizes key information related to the sound source in the features along multiple dimensions.

[0083] The global-local feature extraction module mainly includes omnidirectional dynamic convolution, a local feature extraction unit in the multi-scale feature extraction module, and an attention feature fusion unit. In the multi-scale feature extraction module, a cross-scale shuffling unit increases information exchange between multi-scale features. In the local feature extraction unit, asymmetric convolution is used to extract fine-grained features. The network also includes: group convolution replacing ordinary convolution in the multi-scale feature extraction module to reduce computational complexity and the number of model parameters; and using residual structures in multiple parts of the model to alleviate overfitting.

[0084] To further optimize the scheme, this embodiment implements global-local feature extraction through a parallel structure to obtain comprehensive high-level features. The structure of the global-local feature extractor is as follows: Figure 2 As shown, the main branch consists of an omnidirectional dynamic convolution (ODConv) and a multi-scale feature extraction module (MSFE) to obtain global information. Furthermore, this invention introduces a local feature extraction unit (LFE) to supplement the main branch with fine-grained information. This unit is stacked twice to ensure that a sufficient number of local features are extracted. The attention feature fusion unit (AFF) selectively fuses global and local features.

[0085] Further optimization of the scheme: Omnidirectional Dynamic Convolution (ODConv)

[0086] Dynamic convolutions dynamically assign adaptive weights to the convolution kernel, enhancing model generalization. However, dynamic convolutions have a significant limitation: only the kernel dimension exhibits dynamic characteristics, while other dimensions are ignored. Omnidirectional Dynamic Convolution (ODConv) introduces a multi-dimensional attention mechanism, employing a parallel strategy to learn different attention levels along the four dimensions of the kernel space: spatial dimension, input channel dimension, output channel dimension, and kernel dimension. These four attention levels complement each other and are multiplied by the kernel ωi in the order of position, channel, filter, and kernel. Omnidirectional Dynamic Convolution can be expressed by the formula:

[0087] (4)

[0088] in, Represents the attention scalar, , and These represent three attention weights computed along the spatial, input channel, and output channel dimensions for the convolution kernel Wi in the kernel space, respectively. They are all calculated by the SE (compression-dilation) module. In this invention example, n=1, indicating that only one convolution kernel is used. =1. Introducing such multi-dimensional dynamic weights can enhance the modeling capability of convolution through additional spatial interactions. In this invention example, omnidirectional dynamic convolution is used to calibrate the input features of the Global-Local Feature Extractor (GLFE), thereby effectively reducing redundant information and highlighting important information. The calibrated input features are fed into the Multi-Scale Feature Extraction (MSFE) module to extract multi-scale features with a larger receptive field, and further global features are obtained through the Cross-Scale Shuffle Unit (CSS).

[0089] Further optimization of the scheme includes the Local Feature Extraction Unit (LFE).

[0090] Asymmetric convolution consists of three parallel convolutional layers with kernel sizes of [sizes to be filled in]. , and It can replace standard CNNs with squared kernels, enriching the feature space. Asymmetric convolution can significantly enhance the representational power of CNNs while extracting more accurate spatial features. This invention uses a Local Feature Extraction (LFE) unit based on asymmetric convolution to obtain finer-grained features. For example... Figure 2 As shown in the lower half, the kernel size is and Asymmetric convolutions focus on frequency and temporal information, respectively. This invention example achieves local feature extraction through a combination of these two types of convolutions, with the convolution kernel for one path being... and The other one is and The outputs of these two paths are added together, then passed through a batch normalization layer and the GELU activation function.

[0091] Finally, the Attention Feature Fusion Unit (AFF) replaces the direct addition operation to achieve soft selection between the two branches, aggregating global features from the main branch and local features from the Local Feature Extraction Unit (LFE). This approach highlights the main components among different features while suppressing unnecessary components. The entire process of the Global-Local Feature Extractor (GLFE) can be described by the following formula:

[0092] (5)

[0093] (6)

[0094] (7)

[0095] Further optimization of the scheme includes a multi-scale feature extraction module (MSFE).

[0096] This invention provides an example of obtaining information from feature maps across multiple scales through a multi-scale feature extraction module, which helps in identifying sound events with different time-frequency characteristics. The detailed structure of the multi-scale feature extraction module (MSFE) is as follows: Figure 3 As shown. Specifically, the output of the omnidirectional dynamic convolution (ODConv) layer first passes through a multi-branch convolutional unit. Meanwhile, to reduce computational complexity, we divide the feature map into four equal parts along the channel dimension and use group convolutions with different kernel sizes in parallel to extract spatial information at multiple scales. The grouping strategy of the group convolutions can be represented by the relationship between the multi-scale kernel K and the group size gi:

[0097] (8)

[0098] The SE mechanism computes attention weights for multi-scale feature maps along multiple branches. Using the softmax function, guided by the multi-scale channel weights (att), it adaptively selects feature maps of different scales. For the i-th branch, the channel weights are calculated as follows:

[0099] (9)

[0100] This is the output of the SE module in the i-th branch. Long-range channel dependencies can be obtained through the above operations. The feature map Fi from the i-th group convolutional branch, after attention calibration, can be represented as:

[0101] (10)

[0102] ⊙ indicates channel-by-channel multiplication. This operation allows the module to assign different weights to feature maps at different scales, giving different levels of attention to different channels within the same scale, thus emphasizing multi-scale spatial information across channels. The four calibrated feature maps Yi are then concatenated as the output of the Multi-Scale Feature Extraction (MSFE) module.

[0103] In the Multi-Scale Feature Extraction (MSFE) module, multi-scale features are directly concatenated along the channel dimension without considering the interdependencies between them. On the other hand, using group convolutions can have side effects because the output of each group only relates to a small portion of the input feature map, ignoring effective communication between groups. To mitigate this side effect, this invention introduces a Cross-Scale Shuffle Unit (CSS) that fuses multi-scale features and enhances their connections, thereby capturing global features for the network. Detailed structure is as follows... Figure 4As shown. The channel shuffle operation facilitates the flow of information between multi-scale feature maps. This operation is modeled as a "deform-transpose-deform" process: given the dimension of the input features ( First, transform it into (n). , gs, n / gs), further transpose to ( The shuffled features are then transformed back to their original shape (n / gs, gs), where n and gs represent the number of channels and the group size, respectively. Specifically, in this embodiment, gs = 8. Next, the shuffled features are aggregated with the original features through an aggregation block. The aggregation block mainly consists of two CNNs with a kernel size of 1x1. The first CNN reduces the number of channels, and the second CNN further fuses feature maps from different channels, preserving information at the original channel positions. Furthermore, this invention applies residual connections after both the cross-scale shuffling unit (CSS) and the multi-scale feature extraction module (MSFE).

[0104] Further optimization of the scheme, feature recalibration module

[0105] Attention mechanisms improve the feature representation of CNNs by establishing dependencies between channels or spatial locations. This invention uses a feature recalibration module based on three attention branches, independently computing attention weights along the time, channel, and frequency dimensions, effectively combining the advantages of three attention mechanisms. The complete structure of the feature recalibration module is as follows: Figure 5 As shown.

[0106] In the FR module, the input feature M of shape (C, T, F) is mapped or its dimensionality is changed to form feature maps M1 (T, C, F), M2 (C, T, F), and M3 (F, T, C), which serve as inputs to the three attention branches, respectively. C, T, and F represent channels, time, and frequency, respectively. This invention applies global average pooling to obtain spatial statistics g for each branch along time t, channel c, and frequency f.

[0107] (11)

[0108] (12)

[0109] (13)

[0110] Where m1t is the feature map of time frame t in M1, m2c is the feature map of channel c in M2, and m3f is the feature map of frequency bin f in M3. Then, three 1D CNNs with kernel size 3 are used in parallel to dynamically select task-relevant time frames, channels, and frequency bins. These three attention maps can be obtained using the sigmoid function:

[0111] (14)

[0112] The three attention maps are multiplied by M1, M2, and M3 respectively, recalibrating the feature maps along three directions. This process increases attention to important temporal frames, channels, and frequency bins in the feature maps. Finally, the dimensions of these three recalibrated feature maps are reordered to have the same shape, denoted as M'1, M'2, and M'3 respectively, and summed to obtain the final output.

[0113] Visualize model prediction results

[0114] Step test1: Obtain the input features of the test set samples

[0115] Specifically, the FOA format multi-channel spatial audio signals in the test set are subjected to short-time Fourier transform to extract their log linear spectrum and normalized sound intensity vector.

[0116] Step test2: Test the network model

[0117] The log linear spectrum and normalized sound intensity vector of the test sample are concatenated and fed into the saved optimal model to test the performance of the multi-source localization and detection model, and the prediction results of the network model are output at the same time.

[0118] Step test3: Draw sound event categories and spatial trajectory diagrams.

[0119] Specifically, the network outputs the predicted sound event categories and sound source location information, generating a prediction document for each sample. The document includes an index of active sound event categories, start and end times, direction angle, and azimuth angle. Finally, using one sample as an example, based on the output of the prediction document visualization model, it plots the detection results of different sound events, along with direction angle and azimuth angle trajectories. By comparing this with the visualization image of the test sample's true label, the quality of the model's prediction results can be intuitively observed.

[0120] Experiments were conducted using the technical solution of this embodiment. The experimental results show that the present invention, which adopts a network model based on global-local feature extraction and recalibration, combined with data augmentation methods, can achieve high sound source localization and detection performance. The model exhibits good generalization on both real and synthetic datasets.

[0121] In summary, the advantages of this embodiment are: it achieves comparable multi-source localization and detection performance using a smaller number of parameters compared to current mainstream models, improves the model's generalization ability across various datasets, and reduces the training burden. Furthermore, the structure used in this invention provides the model with rich multi-scale feature information, which helps improve the detection accuracy of different sound events.

[0122] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-source localization and detection method based on global-local feature recalibration, characterized in that, include: Calculate the short-time Fourier transform of the multi-channel spatial audio signal in first-order stereo format, obtain the log linear spectrum and normalized sound intensity vector as input features, and then perform data augmentation on the features of the training set. The augmented spectrum and sound intensity vector are concatenated and used as input to the neural network model. The neural network model is then trained to obtain and save the optimal network model parameters. After preprocessing the test samples, they are fed into the trained model to obtain prediction results. Based on the prediction results, sound event detection maps, direction angle and azimuth trajectory curves are plotted and compared with the visualization images of the real labels of the test samples to analyze the performance of the model. The process of concatenating the augmented spectrum and sound intensity vector as input to the neural network model and training the neural network model includes: The augmented features of the data are used as input to the neural network model; wherein the input features are obtained by concatenating the log linear spectrum and the normalized sound intensity vector, and have seven channels; the dataset contains a label file for each sample, which records the sound event category and sound source location information for supervised learning; The input features are initially processed by the Encoder module, and then high-level features containing global and local information are obtained in parallel by the global-local feature extractor. Based on the high-level features, the key components in the feature map are emphasized along multiple dimensions by the feature recalibration module to obtain the recalibrated fine features. The neural network model ultimately outputs the predicted sound event category and sound source location information along the sound event detection and sound source localization branches, respectively. A joint optimization strategy is adopted to calculate the loss functions of the sound event detection branch and the sound source localization branch by weighting, and obtain the binary cross-entropy loss and mean square error accordingly. The binary cross-entropy loss and mean square error are linearly combined as the final loss function, and the network parameters are updated according to the final loss function. The process of obtaining high-level features through a global-local feature extractor after the input features are initially processed by the Encoder module includes: The augmented input features are fed into the network. First, the encoder structure performs preliminary processing on the features for shallow feature extraction. The encoder structure consists of two convolutional layers and an average pooling operation with a kernel size of 2x2. Each convolutional layer contains a convolutional neural network with a kernel size of 3x3, a batch normalization layer, and a Gaussian error linear unit activation function. There are no residual connections after each convolutional layer in the encoder. Subsequently, the features are fed into a global-local feature extractor with residuals for in-depth processing. The global-local feature extractor includes a backbone branch consisting of omnidirectional dynamic convolution and multi-scale feature extraction modules, and a local feature extraction unit. The backbone branch extracts global features, and the local feature extraction unit extracts local features. Finally, the attention feature unit selectively fuses the global and local features.

2. The multi-source localization and detection method based on global-local feature recalibration according to claim 1, characterized in that, Each first-order stereo signal in the first-order stereo format multi-channel spatial audio signal includes four channels (W, X, Y, Z); where: W is a 0th-order spherical harmonic used to obtain omnidirectional information, and X, Y, Z are 1st-order spherical harmonics that transmit spatial information along the Cartesian coordinate system of the sound field. The log linear spectrum of the spatial audio signal is calculated using the complex spectrum X(t, f), and the formula is as follows: The expression for the normalized sound intensity vector is: Wherein: the sound intensity vector is used to transmit valuable information along the direction of sound propagation, and the reverse direction of the sound intensity vector is interpreted as the direction of arrival. The formula expression for the sound intensity vector is: ρ0 and c represent density and sound speed, respectively. Represents the real part of a complex number. Indicates conjugate.

3. The multi-source localization and detection method based on global-local feature recalibration according to claim 1, characterized in that, Data augmentation of the training set features includes continuously performing data augmentation on each training sample using audio channel swapping, random cropping, and frequency shifting without increasing the amount of data. The audio channel switching method is a spatial augmentation method designed for datasets collected by spherical microphones. It represents the directional response of a first-order stereo audio signal using a cosine function, corresponding to the direction of arrival (DOA), which represents the spatial location information of a sound event. Based on the angular deformation of the DOA expressed by the cosine function, for each DOA, a rotation matrix is ​​used to obtain 16 DOA combinations, including the original DOA and 15 new combinations. A combination is randomly selected as the new DOA for each sample. The random cropping includes a random cropping method and a SpecAugment-based time-frequency masking method. One of these methods is randomly selected and applied to all channels of the input features for each sample. The random cropping method involves arbitrarily selecting a rectangular region on the spectrogram and setting the values ​​within the region to random values ​​within the range of each channel's values ​​in the linear spectrum. For each channel of the sound intensity vector, all values ​​in the rectangular region are set to 0, similar to a masking operation, with all channels sharing a single mask. The SpecAugment-based time-frequency masking method involves selecting a cross-shaped region and applying the same operation as the random cropping method. The frequency shift is a random upward or downward shift of a specific frequency band along the frequency dimension of all channels of the input feature.

4. The multi-source localization and detection method based on global-local feature recalibration according to claim 1, characterized in that, Based on the high-level features, the process of obtaining recalibrated fine features by emphasizing key components in the feature map along multiple dimensions using a feature recalibration module includes the following steps. The feature recalibration module calculates attention along the channel, time, and frequency dimensions to emphasize key channels, time frames, and frequency bands related to the sound source in the features. The resulting fine features are then fed into the sound event detection branch and the direction of arrival estimation branch, respectively.

5. The multi-source localization and detection method based on global-local feature recalibration according to claim 1, characterized in that, The global-local feature extractor includes an omnidirectional dynamic convolution, a multi-scale feature extraction module, a local feature extraction unit, and an attention feature fusion unit; The cross-scale shuffling unit in the multi-scale feature extraction module is used to increase information exchange between multi-scale features. The asymmetric convolution in the local feature extraction unit is used to extract fine-grained features; The neural network model also includes replacing ordinary convolution with group convolution in the multi-scale feature extraction module and adding residual structures to alleviate the overfitting problem.

6. The multi-source localization and detection method based on global-local feature recalibration according to claim 5, characterized in that, The omnidirectional dynamic convolution introduces a multi-dimensional attention mechanism, which adopts a parallel strategy to learn different attentions of the convolution kernel along the spatial dimension of the kernel space, the input channel dimension, the output channel dimension and the convolution kernel dimension, and multiplies them by the convolution kernel ωi in the order of position, channel, filter and kernel. The formula for the omnidirectional dynamic convolution is: in, Represents the attention scalar, , and These represent the three attention weights calculated along the spatial, input channel, and output channel dimensions of the convolution kernel Wi in the kernel space, respectively, and are obtained by the compression-dilation module.

7. The multi-source localization and detection method based on global-local feature recalibration according to claim 5, characterized in that, The process by which the cross-scale shuffling unit enhances information exchange between multi-scale features includes... Channel shuffling facilitates information flow between multi-scale feature maps. This shuffling operation is modeled as a "deform-transpose-deform" process: given the dimension of the input feature (… First, transform it into (n). g s n / g s ), further transposed to ( n / g s g s Finally, it transforms back to its original shape, where n and g s These represent the number of channels and the group size, respectively. Next, the shuffled features are aggregated with the original features through an aggregation block. The aggregation block includes two CNNs with a kernel size of 1x1. The first CNN is used to reduce the number of channels, and the second CNN is used to further fuse the feature maps of different channels, while retaining the information at the original channel positions.

Images