A multi-task acoustic analysis method for scene awareness and event classification

Abstract

The application discloses a kind of scene perception and multi-task acoustic analysis method of event classification, comprising: establishing multi-task network model, by SED and ASC two sub-networks are constituted, each sub-network includes frequency dynamic convolution, pooling layer, attention mechanism and fully connected layer.Cross-stitch module is used, and the soft parameter sharing between the two sub-networks is realized.In the feature extraction stage, the application uses log-mel spectrogram as input feature.Processing is carried out by sending to the convolution module corresponding to each task.Attention module is used to model global and local context information in audio sequence.Each sub-task generates final output through its corresponding fully connected layer.The final result of each task is obtained by weighted average processing of the output of each task.The advantages of the application are: improve the overall performance of network.Improve the accuracy of scene perception and event classification.Improve the generalization ability of network.Good versatility and applicability.

Description

Technical Field

[0001] This invention relates to the field of environmental sound analysis technology, and in particular to a multi-task acoustic analysis method for scene perception and event classification. Background Technology

[0002] With the rapid development of modern intelligent technology, environmental sound analysis has become a highly regarded research field, demonstrating significant importance and potential in numerous application scenarios. Examples include automated life recording systems, monitoring systems, abnormal sound detection systems, and biological monitoring systems. The field of environmental sound analysis primarily focuses on two key tasks: Sound Event Detection (SED) and Acoustic Scene Classification (ASC). Sound Event Detection involves analyzing collected audio data to identify and classify specific sound events within the audio, such as "children playing," "vehicles passing by," "gunshots," or "alarms." This process requires not only accurately identifying the type of sound, such as "keyboard tapping," "car driving," "clattering cutlery," or "people talking," but also estimating the start and end times of each sound event. Acoustic Scene Classification aims to identify and distinguish specific scene categories from different audio segments, helping machines to more effectively understand and judge their acoustic environment. Common scenarios such as "offices in indoor environments," "coffee shops," and "grocery stores" are typically composed of a variety of complex acoustic events, which together form a complex yet realistic acoustic environment.

[0003] In traditional research, sound event detection and acoustic scene classification are often considered two separate tasks. However, many sound events are intrinsically correlated with acoustic scenes and can provide crucial auxiliary information for each other's recognition. For example, sound events such as "keyboard tapping," "mouse clicking," and "people talking" are more common in an acoustic scene like an "office," while sound events such as "car driving" and "birdsong" are relatively less likely to occur in an "office" environment. Therefore, when performing sound event detection, if it is known that a sound event occurs in a specific acoustic scene such as an "office," this information can increase the accuracy of recognizing sound events such as "keyboard tapping" and "mouse clicking," and vice versa.

[0004] Existing technology 1

[0005] In traditional single-task learning methods, acoustic scene classification and sound event detection are typically treated as independent tasks. This approach employs separate models designed specifically for each task, optimizing and processing each task individually, but it suffers from the following drawbacks:

[0006] 1) Unable to utilize the potential connections between tasks

[0007] Single-task learning methods fail to fully utilize relevant information from other tasks when processing each task. This lack of cross-task communication can lead to performance limitations.

[0008] 2) Insufficient resources and computing efficiency

[0009] This approach can lead to low resource utilization and reduced computational efficiency because it requires separate model training and optimization for each task. This is particularly evident when dealing with multiple related tasks, as each task requires independent data processing and model training.

[0010] 3) Limited generalization ability

[0011] Although single-task learning methods may perform well when dealing with specific tasks, they generally have poor generalization ability when dealing with diverse data and scenarios, which limits their potential for application in a wider range of environments.

[0012] Existing technology 2

[0013] Given the close relationship between acoustic scenes and sound events, prior art two proposes a neural network based on multi-task learning (MTL) for joint analysis of acoustic scenes and events. However, this scheme mainly employs a hard parameter sharing mechanism, which to some extent limits the information flow and collaborative learning between different tasks during training. It typically involves sharing parameters between different layers of the network to improve efficiency and performance when handling multiple tasks. However, prior art two has the following drawbacks:

[0014] 1) Restricting information flow and collaborative learning between tasks

[0015] In hard parameter sharing mechanisms, different tasks share the same network layers. This can limit information flow and collaborative learning processes because the shared layers need to meet the requirements of all tasks and may not be able to be deeply optimized for any single task.

[0016] 2) Limited generalization ability

[0017] Hard parameter sharing mechanisms can lead to poor network performance when processing novel or unseen data. This is because the tuning of shared parameters is primarily based on the task in a specific training dataset and may not adapt well to new or changing scenarios.

[0018] 3) Poor flexibility

[0019] An architecture with shared hard parameters may not be flexible enough. Because all tasks depend on the same set of parameters, this limits optimization for specific tasks.

[0020] References

[0021] [1]Tsubaki S,Imoto K,Ono N.Joint Analysis Of Acoustic Scenes AndSound Events With Weakly Labeled Data[C] / / 2022International Workshop onAcoustic Signal Enhancement(IWAENC).IEEE,2022:1-5;

[0022] [2]Nada K,Imoto K,Iwamae R,et al.Multitask learning of acousticscenes and events using dynamic weight adaptation based on multi-focal loss[C] / / 2021 Asia-Pacific Signal and Information Processing Association AnnualSummit and Conference(APSIPA ASC).IEEE,2021:1156-1160;

[0023] [3]Igarashi A,Imoto K,Komatsu Y,et al.How Information on AcousticScenes and Sound Events Mutually Benefits Event Detection and SceneClassification Tasks[C] / / 2022Asia-Pacific Signal and Information ProcessingAssociation Annual Summit and Conference(APSIPA ASC).IEEE,2022:7-11;

[0024] [4]Nada K,Imoto K,Tsuchiya T.Joint analysis of acoustic scenes andsound events based on multitask learning with dynamic weight adaptation[J].Acoustical Science and Technology,2023,44(3):167-175;

[0025] [5]Tonami N,Imoto K,Niitsuma M,et al.Joint analysis of acousticevents and scenes based on multitask learning[C] / / 2019IEEE Workshop onApplications of Signal Processing to Audio and Acoustics(WASPAA).IEEE,2019:338-342;

[0026] [6]N.Tonami,K.Imoto,R.Yamanishi and Y.Yamashita,“Joint analysis ofsound events and acoustic scenes using multitask learning,’IEICETrans.Inf.Syst.,E104-D,294–301(2021);

[0027] [7]H.L.Bear,I.Nolasco and E.Benetos,“Towards joint sound scene andpolyphonic sound event recognition,’Proc.Inter-speech 2019,pp.4594–4598(2019);

[0028] [8]Liang H,Ji W,Wang R,et al.A scene-dependent sound event detectionapproach using multi-task learning[J].IEEE Sensors Journal,2021,22(18):17483-17489;

[0029] [9] Komatsu T, Imoto K, Togami M. Scene-dependent acoustic event detection with scene conditioning and fake-scene-conditioned loss [C] / / ICASSP2020-2020IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP). IEEE, 2020: 646-650. Summary of the Invention

[0030] This invention addresses the shortcomings of existing technologies by providing a multi-task acoustic analysis method for scene perception and event classification. It employs a soft parameter sharing mechanism to promote collaborative learning between acoustic scene classification and sound event detection tasks, thereby improving the accuracy of sound event detection.

[0031] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0032] A multi-task acoustic analysis method for scene perception and event classification includes the following steps:

[0033] 1) Data preparation: Collect and prepare the acoustic datasets required for Sound Event Detection (SED) and Sound Scene Classification (ASC), including audio segments and their corresponding labels (event labels and scene labels).

[0034] 2) Multi-branch attention-based convolutional network design: A multi-task joint learning network is adopted, consisting of two sub-networks: one for sound event detection and the other for sound scene classification. Each sub-network includes modules such as frequency-dynamic convolution, pooling layers, attention mechanisms, and fully connected layers. A cross-stitch module is connected to the back end of each convolutional and attention module. The cross-stitch module connects the SED and ASC tasks, allowing them to supervise each other and determine the optimal combination of shared information and task-specific information.

[0035] 3) Soft parameter sharing: The Cross-stitch module enables the sharing of features between tasks, thereby enhancing the network's ability to process key features and effectively avoiding potential performance loss.

[0036] 4) Loss Function: The cross-entropy (CE) loss function is used to train the ASC model parameters to calculate the loss for acoustic scene classification. The binary cross-entropy (BCE) loss function is used to train the SED model parameters to calculate the loss for sound event detection. The final loss function is a weighted sum of the two task loss functions, and weighted joint optimization is used to achieve more accurate classification performance.

[0037] 5) Model training: Using the prepared dataset and the designed network structure, train the model and optimize the model parameters through the backpropagation algorithm so that the loss function of both tasks can be effectively reduced.

[0038] 6) Apply the trained model to real-world scene perception and event classification tasks to classify and analyze the input sound data, thereby achieving multi-task acoustic analysis of sound event detection and acoustic scene classification.

[0039] Further, in step 1), logarithmic Mel features are selected as the time-frequency domain features of the audio. The Mel spectrogram is calculated from the original audio signal using the librosa library, and then the Mel spectrogram is converted to a logarithmic scale.

[0040] Furthermore, the time-frequency domain features include 64 Mel bands, a frame jump length of 512, and a short-time Fourier transform size of 2048 points.

[0041] The specific steps for calculating the Mel spectrogram include dividing the audio signal into a 40-millisecond Hamming window, performing a short-time Fourier transform to extract the Mel spectrogram, and selecting 64 bandpass filters.

[0042] Furthermore, both the SED and ASC tasks employ 3×3 frequency-adaptive dynamic convolution. Average pooling is performed on the input signal along the time axis, followed by one-dimensional convolution along the channel axis to extract frequency-adaptive attention weights. These attention weights are then adjusted using a softmax function to generate convolution kernels adapted to specific frequencies. These kernels are then applied to standard two-dimensional convolution.

[0043] Furthermore, the multi-branch attention convolutional network consists of two parallel branch modules: an attention branch module and a convolutional branch module. The attention branch module extracts global features, while the convolutional branch module extracts local detail features. These two branches receive the same input features and process them independently, then merge the processed features and feed them into a fusion unit. The fusion unit employs a coordinate attention mechanism for more refined processing of the feature maps. In addition, the network also employs a residual connection strategy. The original input features are passed to subsequent layers of the network through residual connections, combining with the features processed at deeper layers.

[0044] Furthermore, the attention branch module specifically includes: modeling the global context of the input sequence; performing layer normalization on the input features to make the distribution of the input features more stable; using a multi-head self-attention mechanism to capture global information in the sequence; and performing a dropout operation to randomly set the output of a portion of neurons to zero to reduce the overfitting of the model.

[0045] Furthermore, the convolutional branch module specifically includes:

[0046] The convolutional branch module performs layer normalization on the input data to ensure consistency and stability of the data across different network layers.

[0047] In the convolutional branch module, two linear layers are used before and after the convolutional block, combined with the GELU (Gaussian Error Linear Unit) activation function. Through linear transformation and nonlinear activation, the ability to represent input features is enhanced.

[0048] The convolutional block uses depthwise separable convolution.

[0049] Convolutional blocks introduce residual connections, which connect the input to the final output.

[0050] Furthermore, the execution steps of the fusion unit are as follows:

[0051] 1) Perform global pooling on the feature map Y, in both the width and height directions, to obtain Y. h and Y w .

[0052] Y h =GlobalAvgPool h (Y),Y w =GlobalAvgPool w (Y)

[0053] 2)Y h and Y wAn attention vector A is generated using two fully connected layers and an activation function. h and A w .

[0054] A h =σ(FC(ReLU(FC(Y) h )))),A w =σ(FC(ReLU(FC(Y) w ))))

[0055] 3) Use the generated attention vector to weight the feature map Y to obtain the adjusted feature map Y'.

[0056]

[0057] 4) Adjust the feature map Y′ h and Y′ w The features are merged along the channel dimension and added to the original feature map Y to obtain the final fused feature map Y. CA .

[0058] Y CA =Y+Concat(Y′) h ,Y′ w )

[0059] Fusion Unit: The fusion unit employs a coordinate attention mechanism for more refined processing of the feature map. After depthwise separable convolution, an attention vector is generated through coordinate attention. This attention vector is then used to weight and adjust the feature map before being merged with the original feature map to form the final fused feature map.

[0060] Furthermore, the formula for the Cross-stitch module is as follows:

[0061]

[0062] In the formula, α is a 2×2 matrix, α 00 α 01 α 10 and α 11 For the fusion weights between the SED and ASC tasks, the parameters of the Cross-stitch unit are set between [0, 1], x ASC x SED , and These are the inputs to the two tasks and the weighted fusion output, respectively.

[0063] Furthermore, the loss function process is as follows:

[0064] Using the cross-entropy (CE) loss function The model parameters for ASC are trained using this method. The final prediction layer uses SoftMax units, matching the number of sound scene labels in the dataset, and the loss function is... It is represented as follows:

[0065]

[0066] Among them, y n The output of the network is defined by N and Sn, representing the number of acoustic scene categories and the target scene label, respectively. For the SED task, a binary cross-entropy (BCE) loss function for multi-label classification is used. To train the model parameters. The final prediction layer consists of as many Sigmoid units as there are sound event labels in the dataset, and the loss function is... It is represented as follows:

[0067]

[0068] Among them, y t,m For network output, T, M, and z t,m These represent the number of time frames, the number of sound event categories, and the target event label, respectively.

[0069] The task loss functions of ASC and SED tasks are weighted and summed. Through weighted joint optimization, the joint loss function is as follows:

[0070]

[0071] Compared with the prior art, the advantages of the present invention are as follows:

[0072] (1) Improve the accuracy of sound event detection

[0073] By employing a soft parameter sharing mechanism, the collaborative learning of acoustic scene classification and sound event detection tasks is facilitated, significantly improving the accuracy of sound event detection. In experiments, compared to traditional single-task learning models, the proposed method achieves average F-scores of 52.54% and 47.92% on the TUT Acoustic Scenes 2016 / 2017 and TUT Sound Events 2016 / 2017 datasets, as well as the Synthetic Sound Scenes dataset, respectively, fully demonstrating the effectiveness of the method.

[0074] (2) Innovation in Multi-task Learning Network Design

[0075] The multi-task learning network model designed in this invention can efficiently process acoustic scene classification and sound event detection tasks in parallel. This design cleverly utilizes the correlation between the two tasks, improving both processing efficiency and accuracy. Experimental results show that, compared to state-of-the-art methods, our model improves the average F-score by 2.27% and 3.22% on the TUT AcousticScenes 2016 / 2017 and TUT Sound Events 2016 / 2017 datasets, respectively, and on the Synthetic Sound Scenes dataset.

[0076] (3) Optimization of the dual-branch attention module

[0077] This invention designs a dual-branch attention module to effectively capture global and local contextual information in audio sequences. Furthermore, the introduced coordinate attention mechanism serves as a key fusion unit of the model, further enhancing its overall performance and perceptual capabilities. Attached Figure Description

[0078] Figure 1 This is a diagram of the multi-task learning structure according to an embodiment of the present invention.

[0079] Figure 2 This is a structural diagram of the dual-branch attention convolution module according to an embodiment of the present invention.

[0080] Figure 3 This is a flowchart of the fusion unit in an embodiment of the present invention. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0082] Multi-task collaborative learning networks such as Figure 1 As shown, the network consists of two sub-networks, SED and ASC. Each sub-network has its own dedicated modules, including frequency-dynamic convolution, pooling layers, attention mechanisms, and fully connected layers. To improve the overall performance of the network, this invention employs a cross-stitch module to achieve soft parameter sharing between the two sub-networks.

[0083] In the feature extraction stage, this invention uses a 64-dimensional log-Melogram as the input feature. These features are then processed by the convolutional modules corresponding to their respective tasks. Furthermore, an attention module is employed to model the global and local contextual information in the audio sequence. After completing the above steps, each subtask generates its final output through its corresponding fully connected layer. In the network proposed in this invention, each convolutional and attention module's backend is connected to a cross-stitch module, facilitating information sharing between the two tasks. Finally, the final result for each task is obtained by weighted averaging of the outputs of each task.

[0084] I. Feature Extraction Section

[0085] Because log-Mel band energy performs exceptionally well in acoustic scene classification, sound event detection, and many other acoustically related tasks, we chose log-Mel features as the time-frequency domain features for audio. These features are configured with 64 Mel bands, a frame jump length of 512, and a short-time Fourier transform size of 2048 points. This invention uses the librosa library to compute the Mel spectrogram from the raw audio signal. Specifically, we extract the Mel spectrogram using a short-time Fourier transform within a 40-millisecond Hamming window, selecting 64 bandpass filters in this process. Finally, the Mel spectrogram is converted to a logarithmic scale to better represent the characteristics of the audio signal. Therefore, for a sequence length of T frames, the feature dimension of the resulting log-Mel spectrogram is 64×T.

[0086] II. Model Structure

[0087] 2.1 Frequency-based dynamic convolution

[0088] To effectively capture key frequency features in audio signals, this invention employs 3×3 frequency-dynamic convolution in both tasks. Specifically, the input signal is first average-pooled along the time axis, and then one-dimensional convolution is performed along the channel axis to extract frequency-adaptive attention weights. Next, these attention weights are adjusted using a softmax function to generate convolution kernels adapted to specific frequencies. Subsequently, these convolution kernels are applied to standard two-dimensional convolution, a step that enables the network to more accurately capture and process key frequency information in the audio signal.

[0089] Frequency-adaptive convolution enhances the frequency dependence of 2D convolution by employing frequency-adaptive convolution kernels, improving the physical consistency of the model when processing time-frequency patterns of sound events. The key to this method is its ability to dynamically adjust the convolution kernels based on the characteristics of the audio data, thereby more accurately capturing key frequency information in the sound signal and improving performance in sound event detection and acoustic scene classification tasks. In the actual algorithm, the output of each base kernel is first calculated according to equation (1), and then the weighted sum is calculated according to equation (2), as shown below:

[0090] y i (t,f)=W i *x(t,f)+b i (1)

[0091]

[0092] Where t is time, f is frequency, x and y are the input and output of a frequency-dynamic convolutional layer, and W is... i and b i y is the weight and bias of the i-th basic kernel. i It is the output of the i-th basic kernel, π i (f,x) is the frequency-adaptive attention weight of the i-th basic kernel, and K is the number of basic kernels.

[0093] 2.2 Multi-branch attention convolutional network

[0094] This invention proposes an attention network structure based on two parallel branches, the detailed structure of which is as follows: Figure 2 As shown in (b), in this structure, the left branch employs an attention mechanism. The main function of this branch is to extract global features; the right branch, based on convolution, focuses on extracting local detail features from the audio data. Both branches of the model receive the same input features and process them independently. After processing, these two types of features are merged and fed into a fusion unit. Furthermore, considering the potential gradient vanishing and exploding problems during deep network training, this model employs a residual connection strategy. The original input features are also passed to subsequent layers of the network through residual connections, combining with the deeply processed features. In addition, to further enhance the model's ability to represent and extract key features from the input data, this invention also introduces a coordinate attention mechanism. This mechanism is used to fuse the feature information from the two branches, allowing the model to more specifically focus on specific regions of the input data, thereby improving the model's performance and perceptual capabilities.

[0095] (1) Attention Branch

[0096] In this model, the attention module is located on the left side of the graph, and its main purpose is to model the global context of the input sequence. For example... Figure 2As shown in (b), the input features are first normalized layer by layer, and then the model uses a multi-head attention mechanism to capture global information in the sequence. Finally, dropout is performed to improve the robustness of the model.

[0097] In this structure, multi-head attention plays a crucial role in the model. MHSA is responsible for analyzing global contextual information, while the analysis of local information is handled by the convolutional branch. This design achieves an effective division of labor between global and local information, enabling the model to simultaneously capture multiple sound features in a scene. These features may include background noise, typical sounds of a specific scene, and other environmental sounds. More importantly, it can also focus on different characteristics of sound, including frequency, intensity, and temporal relationships, thereby more accurately identifying and locating sound events. The multi-head attention mechanism is implemented by dividing the attention module into several independent "heads." Each head focuses on learning a specific aspect of the input data, and the information acquired from these different aspects is finally integrated to form a more comprehensive and integrated data representation. The effect of this process is as follows: Figure 2 As shown in (a).

[0098] (2) Convolutional branch

[0099] The convolution branch of this model is located at Figure 2 The right side of (b) focuses on modeling the local context in the input sequence. First, the input data is normalized using Layer Normalization (Layernorm) to ensure consistency and stability across different network layers. Then, the model employs two linear layers before and after each convolutional block, combined with the GELU (Gaussian Error Linear Unit) activation function. Before convolution, linear transformations and non-linear activations effectively enhance the representation of input features.

[0100] Next is the convolutional block, such as Figure 2 As shown in (c), this invention employs Depthwise Separable Convolution, a convolutional block composed of a combination of depthwise convolution and pointwise convolution. Compared to traditional convolutional layers, Depthwise Separable Convolution has fewer parameters and lower computational cost, improving model efficiency and performance. Furthermore, the model introduces residual connections to link the input to the final output. This design effectively preserves the original input information and accelerates gradient propagation, thereby enhancing the model's learning efficiency and stability. In this way, the convolutional module can not only effectively capture local features of the input sequence but also process and understand audio data more accurately.

[0101] (3) Fusion Unit

[0102] This invention employs a coordinate attention (CA) mechanism as the fusion unit, and its process is as follows: Figure 3 As shown, after the outputs of the two branches undergo depthwise convolution, coordinate attention (CA) is used to further refine the feature map. Specifically, CA performs global pooling along the width and height dimensions of the feature map. This process not only captures global information of the feature map in each dimension but also allows the model to learn and utilize the spatial distribution of features, thereby improving its ability to adjust and capture feature details. In the final stage, the feature map processed by CA is merged with the original feature map to form the final fused feature map. This feature map not only retains the basic information of the input data but also integrates features captured by the global context, thereby enhancing the model's ability to capture key features.

[0103] Assuming Y represents the feature map after depthwise convolution, the calculation process of CA can be described as follows:

[0104] 1) Perform global pooling operations on Y in both the width and height directions to obtain Y. h and Y w

[0105] Y h =GlobalAvgPool h (Y),Y w =GlobalAvgPool h (Y)

[0106] 2)Y h and Y w Attention vectors are generated using two fully connected layers (FC) and an activation function.

[0107] A h =σ(FC(ReLU(FC(Y) h )))),A w =σ(FC(ReLU(FC(Y) w ))))

[0108] 3) Use the generated attention vector to weight Y to obtain the adjusted feature map Y'

[0109]

[0110] 4) Adjust the feature map Y′ h and Y′ w The features are merged along the channel dimension and added to the original feature map Y to obtain the final fused feature map Y. CA ,

[0111] Y CA =Y+Concat(Y′) h ,Y′ w )

[0112] In this process, This indicates element-wise multiplication, σ represents the Sigmoid function, and Concat represents the concatenation operation performed along the channel dimension.

[0113] 2.3 Soft parameter sharing

[0114] Sound event detection and sound scene classification, as tasks that influence each other, constitute a typical application scenario of multi-task learning. Within the multi-task learning framework, hard parameter sharing and soft parameter sharing are two mainstream implementation methods. Hard parameter sharing mainly refers to sharing most of the model parameters, especially the hidden layers, across multiple tasks, while maintaining independence in the task-specific output layers. This method effectively reduces the risk of overfitting and improves the generalization ability of the parameters. On the other hand, soft parameter sharing is a method favored by modern research. It allows each task to maintain an independent model and parameter set, while establishing connections between different tasks through specific mechanisms, promoting information sharing and knowledge transfer. Modern research in the field of multi-task learning has proposed several effective mechanisms to achieve soft parameter sharing.

[0115] For SED and ASC, this invention designs two dedicated sub-networks respectively. To facilitate collaborative learning between the two tasks, we introduce a Cross-stitch module. As a key component of the multi-task model, this module's main function is to connect the SED and ASC tasks, allowing them to supervise each other and determine the optimal combination of shared and task-specific information. The application of the Cross-stitch module enables feature sharing between tasks, which not only enhances the network's hidden layer's ability to process key features but also effectively avoids potential performance loss between the two sub-tasks. The mathematical model of the Cross-stitch unit is described below:

[0116]

[0117] Where α is a 2×2 matrix, α 00 α 01 α 10 and α 11 For the fusion weights between the two tasks, the parameters of the Cross-stitch unit are set between [0, 1], x ASC x SED , and These are the inputs to the two tasks and the weighted fused output, respectively. The parameters of the cross-stitch unit are set between [0, 1] to ensure the stability of the learning process, and the input and output values ​​should remain on the same order of magnitude.

[0118] 2.4 Loss Function

[0119] For the ASC task, this invention employs the cross-entropy (CE) loss function. To train the model parameters. The final prediction layer uses SoftMax units with the same number of sound scene labels as in the dataset, and the loss function is... It is represented as follows:

[0120]

[0121] Among them, y n For the network output, N and Sn represent the number of acoustic scene categories and the target scene label, respectively.

[0122] For the SED task, this invention employs the binary cross-entropy (BCE) loss function for multi-label classification. To train the model parameters. The final prediction layer consists of as many Sigmoid units as there are sound event labels in the dataset, and the loss function is... It is represented as follows:

[0123]

[0124] Among them, y t,m For network output, T, M, and z t,m These represent the number of time frames, the number of sound event categories, and the target event label, respectively.

[0125] The model's final loss function is a weighted sum of the two task loss functions. During training, the weights of these two loss functions are used for joint optimization to achieve more accurate classification performance. The joint loss function is represented as follows:

[0126]

[0127] III. Experimental Section

[0128] (1) Dataset

[0129] To comprehensively evaluate the performance of the proposed multi-task learning method on SED and ASC, all experiments in this invention were conducted on two datasets: the TUT Acoustic Scenes 2016 / 2017 and TUTSound Events 2016 / 2017 datasets (referred to as the TUT 2016 / 2017 dataset) and the Synthetic SoundScenes dataset.

[0130] On one hand, the TUT 2016 / 2017 dataset includes four different acoustic scenarios: “city center,” “home,” “office,” and “residential area.” These scenarios contain a total of 266 minutes of audio recordings (192 minutes for the development set and 74 minutes for the evaluation set). The selected audio clips cover 25 different types of sound events.

[0131] On the other hand, the Synthetic Sound Scenes dataset contains 3000 recordings, each 30 seconds long, created in Scaper. The recordings in this dataset were synthesized using 10 background scenes from the DCASE 2013 ASC challenge and 32 possible sound events from FreeSound.org. All synthesized scenes are mono and sampled at 44100Hz. For the Synthetic Sound Scenes dataset, each recording is detailedly annotated, including the start and end times and labels of the sound events. This invention divides the entire dataset into three parts: 2400 recordings for training, 300 for validation, and another 300 for testing.

[0132] (2) Model training settings

[0133] All neural networks in this invention employ the Adam optimizer with a learning rate of 0.0001 and are trained for 110 epochs. To enhance the model's generalization ability, a Dropout regularization method with a probability of 0.15 is introduced after each neural network layer. Furthermore, this invention uses an adaptive thresholding method to estimate the occurrence of active sound events, effectively improving the accuracy of sound event detection. Our proposed multi-task learning method is implemented using the PyTorch library.

[0134] (3) Evaluation indicators

[0135] Because acoustic events may overlap, this invention uses frame-based metrics to evaluate the performance of Sound Event Detection (SED), since the SED system outputs frame-by-frame predictions, and this metric allows us to understand the basic trend of the system's output. This invention uses the metrics Macro-Fscore, Micro-Fscore, and Error Rate for evaluation. The Macro-Fscore is calculated cumulatively across all categories, while the Micro-Fscore is calculated separately for each category, and then the average of these F-scores is calculated.

[0136] The F-score is calculated as shown in the following formula:

[0137]

[0138] First, calculate precision and recall, as shown in the following formula, where TP, FP, and FN refer to the total number of true positive, false positive, and false negative sound segments across all time ranges and acoustic events, respectively.

[0139]

[0140]

[0141] Error Rate is calculated as shown in the following formula:

[0142]

[0143] First, calculate the replacement (S), deletion (D), and insertion (I), as shown in the following formula:

[0144] S(k) = min(FN(k), FP(k))

[0145] D(k) = max(0, FN(k) - FP(k))

[0146] I(k) = max(0, FP(k) - FN(k))

[0147] Where k represents the index of the time frame, and N(k) represents the number of sound events in time frame k.

[0148] (4) Explanation of Experimental Results

[0149] In this invention, we used two key datasets for experimental validation: first, the training hyperparameter settings (Table 1), the TUT Acoustic Scenes 2016 / 2017 and TUT Sound Events 2016 / 2017 datasets (Table 2), and the Synthetic Sound Scenes dataset (Table 3). The experimental settings on these two datasets were kept consistent. Our experiments consisted of three main parts: first, the overall performance of the single-task model and the multi-task model was compared; second, a comparative analysis was performed on each sound event in both the single-task and multi-task models; and finally, the multi-task model was compared with other related research methods using the same datasets.

[0150] Experimental results show that, compared with single-task methods, traditional methods, and state-of-the-art methods, the proposed model achieves significant improvements in average F-score on the TUT Acoustic Scenes 2016 / 2017 and TUT Sound Events 2016 / 2017 datasets. Compared to state-of-the-art methods, the average F-score increases by 2.27% and 3.22%, respectively. Experiments confirm the effectiveness of the proposed multi-task learning method in acoustic scene classification and sound event detection, and demonstrate that information from the acoustic scene assists in sound event detection.

[0151] Table 1: Training Hyperparameter Settings

[0152]

[0153]

[0154] Table 2: Comparison results of the method of this invention with mainstream models on the TUT 2016 / 2017 dataset.

[0155] 1 S. Tsubaki et al. 2022[1] 42.78％ —— 2 K. Nada et al. 2021[2] 44.29％ —— 3 A.Igarashi et al.2022[3] 44.63％ —— 4 K. Nada et al. 2023[4] 46.06％ —— 5 N.Tonami et al. 2019[5] 49.56％ 0.695 6 K.Imoto et al. 2021[6] 49.82％ 0.691 7 Proposed 52.54％ 0.561

[0156] Table 3: Comparison results of the method of this invention on the Synthetic Sound Scenes dataset and mainstream models.

[0157] 1 HLBear et al. 2019[7] 22.02％ 2 H.Liang et al. 2021[8] 40.68％ 3 T. Komatsu et al. 2020[9] 43.00％ 4 Proposed 47.92％

[0158] The methods described above according to the invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code. When said software or computer code is accessed and executed by the computer, processor, or hardware, it implements the multi-task acoustic analysis method for scene perception and event classification described herein. Furthermore, when a general-purpose computer accesses the code used to implement the processing shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the processing shown herein.

[0159] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A multi-task acoustic analysis method for scene perception and event classification, characterized in that, Includes the following steps: 1) Data preparation: Collect and prepare the acoustic datasets required for Sound Event Detection (SED) and Sound Scene Classification (ASC), including audio segments and their corresponding labels; 2) Multi-branch attention convolutional network design: A multi-task joint learning network is adopted, which is a multi-branch attention convolutional network, including two parallel sub-networks. The two sub-networks receive the same input features. One sub-network is used for sound event detection (SED), and the other sub-network is used for sound scene classification (ASC). Each sub-network includes multiple convolutional units connected in sequence and a fully connected layer. Each convolutional unit includes a frequency-dynamic convolutional layer and a pooling layer connected after the frequency-dynamic convolutional layer. A cross-stitch module is set between every two adjacent convolutional units. The cross-stitch module is used to weight and fuse the features output by the previous convolutional unit in the two sub-networks according to a 2×2 fusion weight matrix. The fused features are then input into the last convolutional unit of each of the two sub-networks. The elements of the fusion weight matrix are the fusion weights between the two sub-networks. The features output by the last convolutional unit in each sub-network are input in parallel into two dual-branch attention convolutional modules. Each dual-branch attention convolutional module includes a parallel attention branch module, a convolutional branch module, and a fusion unit for fusing the outputs of the two branch modules. The features output by the two dual-branch attention convolutional modules in each sub-network are weighted and fused with the features output by the other sub-network through a cross-stitch module according to the fusion weight matrix. After being added to the features output by the last convolutional unit of the sub-network through a residual connection, the features are input into the fully connected layer of the sub-network to obtain the output of the sub-network. 3) Soft parameter sharing: Feature sharing between two sub-networks is achieved through the Cross-stitch module; 4) Loss function: The cross-entropy loss function is used to train the model parameters of ASC, which is used to calculate the loss of the model for acoustic scene classification; the binary cross-entropy loss function is used to train the model parameters of SED, which is used to calculate the loss of the model for sound event detection; the final loss function is the weighted sum of the loss functions of the two tasks. 5) Model training: Using the prepared dataset and the designed network structure, train the model and optimize the model parameters through the backpropagation algorithm; 6) Apply the trained model to real-world scene perception and event classification tasks to classify and analyze the input sound data, thereby achieving multi-task acoustic analysis of sound event detection and acoustic scene classification.

2. The multi-task acoustic analysis method for scene perception and event classification according to claim 1, characterized in that: Step 1) Select log-Mel features as the time-frequency domain features of the audio; The Mel spectrogram is calculated from the original audio signal using the librosa library, and then the Mel spectrogram is converted to a logarithmic scale.

3. The multi-task acoustic analysis method for scene perception and event classification according to claim 2, characterized in that: The time-frequency domain features include 64 Mel bands, a frame jump length of 512, and a short-time Fourier transform size of 2048 points. The specific steps for calculating the Mel spectrogram include dividing the audio signal into a 40-millisecond Hamming window, performing a short-time Fourier transform to extract the Mel spectrogram, and selecting 64 bandpass filters.

4. The multi-task acoustic analysis method for scene perception and event classification according to claim 1, characterized in that: Both the SED and ASC tasks employ 3×3 frequency dynamic convolution; average pooling is performed on the input signal along the time axis, and one-dimensional convolution is performed along the channel axis to extract frequency-adaptive attention weights; the attention weights are adjusted using the softmax function to generate convolution kernels adapted to specific frequencies; these convolution kernels are then applied to standard two-dimensional convolution.

5. The multi-task acoustic analysis method for scene perception and event classification according to claim 1, characterized in that: The dual-branch attention convolution module includes two parallel branch modules: an attention branch module and a convolution branch module. The attention branch module is used to extract global features, and the convolution branch module is used to extract local detail features. These two branch modules receive the same input features and process them independently. Then, the processed features are merged and sent to the fusion unit. The fusion unit adopts a coordinate attention mechanism.

6. The multi-task acoustic analysis method for scene perception and event classification according to claim 5, characterized in that: The attention branch module specifically includes: modeling the global context of the input sequence; performing layer normalization on the input features; using a multi-head self-attention mechanism to capture global information in the sequence; and performing a dropout operation to randomly set the output of a portion of the neurons to zero.

7. The multi-task acoustic analysis method for scene perception and event classification according to claim 5, characterized in that: The convolutional branch module specifically includes: The convolution branch module performs layer normalization on the input data; In the convolutional branch module, two linear layers are used before and after the convolutional block, and the GELU activation function is combined. The convolutional block uses depthwise separable convolution; Convolutional blocks introduce residual connections, which connect the input to the final output.

8. The multi-task acoustic analysis method for scene perception and event classification according to claim 5, characterized in that: The execution steps of the fusion unit are as follows: 1) Perform global pooling on feature map Y, in both the width and height directions, to obtain... and ; , 2) and An attention vector is generated using two fully connected layers and an activation function. and ; ， 3) Use the generated attention vector to weight the feature map Y to obtain the adjusted feature map Y'. ， 4) Adjust the feature map and The features are merged along the channel dimension and added to the original feature map Y to obtain the final fused feature map Y. CA ; ， Fusion Unit: The fusion unit adopts a coordinate attention mechanism; after depthwise separable convolution, an attention vector is generated through coordinate attention, and then the attention vector is used to adjust the weight of the feature map, and finally merged with the original feature map to form the final fused feature map.

9. The multi-task acoustic analysis method for scene perception and event classification according to claim 1, characterized in that: The formula for the Cross-stitch module is as follows: ， In the formula, It is a 2×2 matrix. , , and The fusion weights between the SED and ASC tasks are... The parameters of the cross-stitch unit are set between [0, 1]. , , and These are the inputs to the two tasks and the weighted fusion output, respectively.

10. The multi-task acoustic analysis method for scene perception and event classification according to claim 1, characterized in that: The loss function process is as follows: Using the cross-entropy loss function Scene The model parameters for ASC were trained; the final prediction layer was set with SoftMax units, the same number of sound scene labels as in the dataset, and the loss function was... It is represented as follows: ， in, For the network output, N and Sn represent the number of acoustic scene categories and the target scene label, respectively; For the SED task, the binary cross-entropy (BCE) loss function for multi-label classification is adopted. event To train the model parameters; the final prediction layer includes as many sigmoid units as the sound event labels in the dataset, and the loss function is... It is represented as follows: ， in, For network output, T, M and These represent the number of time frames, the number of sound event categories, and the target event label, respectively. The task loss functions of ASC and SED tasks are weighted and summed; through weighted joint optimization, the joint loss function is as follows: 。

Images