An acoustic scene and sound event joint task analysis method

Abstract

The application discloses a kind of acoustic scene and sound event joint analysis method based on multi-task learning, target is by joining additional sound event information, focus on improving the performance of acoustic scene perception, including: Multi-gate Mixture-of-Experts model is introduced into acoustic field, replace traditional multi-task learning model, weaken the inherent conflict caused by difference between tasks.On the basis of Multi-gate Mixture-of-Experts model, increase two information interaction branches, propose a new model-Cross_MMoE.Class-Balanced Loss replaces traditional multi-task weighted loss, in traditional multi-task model, MMoE model, Cross_MMoE model is evaluated, and evaluation result is obtained.The application has the advantages that: using multi-task learning strategy, two tasks associated with acoustic field are jointly analyzed, the calculation and storage cost is reduced, the inference speed is greatly increased, and the accuracy of acoustic scene classification is improved.

Description

Technical Field

[0001] This invention relates to the field of acoustic scene classification and sound event detection technology, and in particular to a joint analysis method for acoustic scenes and sound events based on multi-task learning using the Cross_MMoE model and Class-Balanced Loss. Background Technology

[0002] Multitask Learning (MTL) involves simultaneously learning multiple related tasks, leveraging the rich connections between them to mutually enhance task performance. MTL has wide applications, ranging from computer vision and natural language processing to even recommendation systems, where it can optimize objectives. For example, in e-commerce, optimization goals might consider metrics like the number of items added to favorites, shopping cart entries, and total purchases—all of which are interconnected. Using a single model for multitask learning not only saves on data processing, training, and deployment costs but also improves learning efficiency and prediction accuracy.

[0003] With the rapid development of modern intelligent devices, automatic environmental sound analysis has found wider applications, such as in autonomous driving, abnormal sound detection systems, and hearing impairment support systems. Most studies analyze acoustic scene classification (ASC) and sound event detection (SED) separately, but these two are closely related. Sound event information can assist in sound scene classification; for example, using sound event information as prior information can narrow down the range of possible scene types. Combining sound event information can improve the performance of acoustic scene classification. Compared to standard single-task learning, multi-task learning methods can be designed from the perspectives of network structure and loss function to improve task performance. Continuous innovation in model network structure addresses how to most efficiently achieve parameter sharing and separation among multiple tasks, allowing the model to both integrate the commonalities between different tasks and provide each task with independent space to prevent interference. Another perspective is how to optimize the training process of multi-task learning, such as optimizing the loss function.

[0004] In terms of models, previous researchers have mostly adopted traditional multi-task learning models when dealing with joint ASC and SED tasks. These models improve generalization ability by sharing domain information through shared representations within the multi-task learning framework. However, such multi-task learning models are very sensitive to factors such as differences in data distribution and relationships between tasks, which often leads to the performance of multi-task learning being worse than that of single-task learning.

[0005] Regarding the loss function, its design is a crucial factor affecting the model's expressive power during multi-task learning and training. Generally, the loss function is a weighted average of the losses from different tasks. These weights are either constant during training or require manual adjustment. Model performance is highly dependent on the choice of these weights, but searching for the optimal weights is prohibitively expensive and difficult to resolve manually. While dynamic weight adaptation can improve these issues to some extent, it has limitations in terms of model and dataset generality.

[0006] References

[0007] [1]Y.Zhang, and Q.Yang, "A survey on multi-task learning," IEEETransactions on Knowledge and Data Engineering, vol.34, no.12, pp.5586-5609, 2021;

[0008] [2] Y.Hou, B.Kang, W.Van Hauwermeiren et al., "Relation-guided acousticscene classification aided with event embeddings." pp.1-8, 2022. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a joint task analysis method for acoustic scenes and sound events, which significantly improves the performance of acoustic scene classification by combining sound event information.

[0010] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0011] A method for joint task analysis of acoustic scenes and sound events, characterized by the following steps:

[0012] 1) Audio feature extraction: Collect raw audio data and use Mel-spectrum cepstral coefficients to convert the raw audio data into audio features;

[0013] 2) Baseline model construction: Construct a shared underlying network, including three sets of 3×3 convolutional and 2×2 max pooling layers. The baseline model uses a CRNN model for the sound event classification task branch and a CNN model for the acoustic scene classification task branch.

[0014] 3) Baseline model training: Training is performed using the structure and parameters of the baseline model; the model is trained by inputting audio features and learning scene and event perception capabilities;

[0015] 4) MMoE Model Construction: The initial version of the MMoE model is constructed using multiple expert networks and gating units on a shared underlying network; CRNN is used for the sound event classification task branch in the MMoE model, and CNN is used for the acoustic scene classification task branch; the optimal number of expert networks is found through experimental adjustments.

[0016] 5) MMoE model training: Input audio features and train and optimize the MMoE model to learn scene and event perception capabilities;

[0017] 6) Cross_MMoE model construction and training: Based on the MMoE model, two information interaction branches are added to obtain the Cross_MMoE model; audio features are input and the Cross_MMoE model is trained and optimized to learn scene and event perception capabilities.

[0018] 7) Class-Balanced Loss Usage: Replace the cross-entropy loss function in the baseline model, MMoE model, and Cross_MMoE model with Class-Balanced Loss for evaluation of acoustic scene classification tasks;

[0019] 8) Evaluation Model: Evaluate the baseline model, MMoE model and Cross_MMoE model, check their F1 scores on the acoustic scene perception task, and obtain the evaluation results.

[0020] Furthermore, in step 4), the underlying model structure of the MMoE model contains three expert networks. Each expert network is a feedforward network. A gating unit is introduced for each task. The gating unit outputs different weights through the softmax function. The weights are weighted and summed with the output of each expert network as the input to the two task branches of acoustic scene and sound event.

[0021] The MMoE model can explicitly model task relationships and learn task-specific features to leverage shared representations, allowing for automatic parameter assignment to capture shared task information or task-specific information.

[0022] Furthermore, in step 6), the output of the sound event classification task branch is added to the output of the shared underlying network and used as the input of the sound scene classification task branch. The same method is used to enrich the input information of the sound event classification task branch and increase the interaction information between the two tasks.

[0023] Furthermore, in step 7), Class-Balanced Loss is used instead of the traditional cross-entropy loss function, and a weighting factor that is inversely proportional to the number of effective samples is introduced.

[0024] Furthermore, the Class-Balanced Loss is expressed by the following formula:

[0025]

[0026] In the formula, P represents the class probability vector estimated by the model, indicating the probability that a sample belongs to each class. The vector length is C, where C is the total number of classes.

[0027] y represents the true label of the sample, which belongs to one of the categories {1,2,…,C}.

[0028] Represents the number of valid samples in the y-th class, indicating the number of valid samples in the target class. It is calculated using a weighting factor, which is determined by the parameter β.

[0029] L(P,y) represents the loss function.

[0030] Compared with the prior art, the advantages of the present invention are as follows:

[0031] 1. Overcoming the limitations of traditional multi-task learning models: For the first time, the MMoE model is introduced into the fields of acoustic scene and event perception, replacing traditional multi-task learning models and enhancing performance in handling different tasks, thereby improving model performance and accuracy. By introducing multiple expert networks and gating mechanisms, MMoE can automatically adjust the degree of parameterization between shared modeling information and task-specific modeling information, improving the model's expressive power and, to some extent, resolving the performance conflicts caused by task differences in traditional multi-task learning models. Even with imbalanced datasets, it still achieves high accuracy, with an F1 score of 98.74%, a 1.43% improvement compared to traditional multi-task learning models.

[0032] 2. Improve model performance: Based on the MMoE model, a new model structure—Cross_MMoE model—is proposed. This model adds information interaction between the two task branches of acoustic scene classification and sound event classification, further improving the classification accuracy of the two tasks on the basis of MMoE, and enhancing the robustness and generalization ability of the model.

[0033] 3. Improved Loss Function: Considering the imbalanced nature of the data in this invention, Class-Balanced Loss was ultimately selected as the improved method. In CB Loss, the loss weight of each class is inversely proportional to its sample size, which can enhance the model's attention to the minority classes and dynamically balance the model's performance on the entire dataset. Experiments show that this method has advantages on different models. Attached Figure Description

[0034] Figure 1 This is a baseline model structure diagram of an embodiment of the present invention.

[0035] Figure 2 This is a flowchart of the MMoE model according to an embodiment of the present invention.

[0036] Figure 3 This is a structural diagram of the Cross_MMoE model according to an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0038] I. Baseline Model

[0039] like Figure 1 As shown, it includes a shared underlying network and two branches for two classification tasks. A detailed explanation follows:

[0040] The baseline model of this invention selects a multi-task neural network model, which includes a shared layer and two separate task branches. The shared layer extracts common feature information about sound events and sound scenes, which is then fed into the two separate task branches to obtain the classification results of scenes and events. The shared layer consists of three 3×3 convolutional layers and 2×2 max pooling layers. A CRNN model is used for sound event classification, and a CNN is used as the sound scene classification network.

[0041] Traditional loss functions often employ a weighted summation approach, and this invention continues this method in the baseline. Since the SED task is not the primary focus but only serves as an auxiliary means to improve ASC task performance, frame-level audio events are not considered; only segment-level events are taken into account. Therefore, the cross-entropy (CE) loss function is used to optimize both the ASC and SED tasks, where N and y... n z n These represent the number of acoustic scene (sound event) categories, the model's output, and the ground truth scene labels, respectively. Expressed as a formula:

[0042]

[0043] The objective function of this method is ultimately expressed as:

[0044] E(θ)=(1-α)E1(θ)+αE2(θ) (2)

[0045] α represents the weight of the acoustic scene classification loss, E1(θ) represents the loss of the sound event classification task, and E2(θ) represents the loss of the acoustic scene classification task.

[0046] II. Improve model structure to enhance performance of acoustic scene classification tasks

[0047] 2.1 MMoE Model

[0048] By reviewing relevant literature on joint task analysis of ASC and SED, it was found that most of the models in previous work adopted... Figure 1 A similar model structure is used. This model structure is effective in most cases, but we observed that in some cases, its performance is actually worse than that of the standalone ASC task. The goal of this invention is to leverage additional sound event information to improve the performance of acoustic scene classification; therefore, we first consider changing the model structure to improve ASC classification results.

[0049] We believe the above problems are due to Figure 1 The baseline structure fails to adequately balance inter-task relationships because the baseline model shares most parameters, which is detrimental to mitigating the inherent conflicts arising from task differences, potentially harming prediction results for some tasks. MMoE explicitly models task relationships and learns task-specific features to leverage shared representations. It allows for automatic parameter allocation to capture shared task information or task-specific information without adding excessive new parameters for each task. Therefore, this invention introduces the MMoE model into the joint analysis task of ASC and SED.

[0050] Figure 2 This paper demonstrates the structure of the MMoE used in this invention, which differs slightly from the original paper. The relevant parameters were obtained through multiple experimental optimizations. The underlying model structure comprises three expert networks, each a feedforward network, with a gating unit introduced for each task. The gating unit outputs different weights through a softmax function, which are then weighted and summed with the outputs of each expert network to serve as inputs for two task branches. In this way, the gating networks for different tasks can learn the hybrid patterns of different expert sets, thereby capturing inter-task relationships. The output of task k is expressed by the formula:

[0051] y k =h k (f k (x)) (3)

[0052]

[0053] g k (x) = softmax(W) gk x) (5)

[0054] The expert network consists of multilayer perceptrons with ReLU activation functions, and the gated units are simple linear transformations with softmax layers. n is the number of expert networks, and d is the dimension of the input features.

[0055] 2.2 Cross_MMoE Model

[0056] This invention proposes a novel MMOE model—Cross_MMoE, such as... Figure 3 As shown, to better adapt to the tasks of this invention, the aim is to enhance the interactivity between tasks and improve the robustness of the model. We believe that MMOE makes full use of shared feature information at the underlying network structure level, but neglects the information interaction between the two task branches. Meanwhile, from the perspective of the dataset, considering that the dataset contains 29 sound event categories, the sound event information is very rich. To fully utilize this rich sound event information and focus on improving the performance of sound scene classification, we add the output of the sound event classification task branch to the output of the underlying network structure and use this sum as the input of the sound scene classification task branch. Similarly, we use the same method to enrich the input information of the sound event classification task branch. This design approach ( Figure 3 The cross-section (in the middle part) ensures the stability of the model learning process to a certain extent.

[0057] III. Class-Balanced Loss

[0058] Multi-task learning focuses on optimizing multiple objectives. The loss function in multi-task learning models often employs a weighted summation of the losses for each individual task. However, this approach has many problems, such as model performance being highly sensitive to weight selection, high cost of manually adjusting weight hyperparameters, difficulty in determining the optimal weight combination, and the inability to change weights during training. Previously, to overcome the limitations of traditional multi-task learning methods, researchers often used dynamic weight adaptation methods to dynamically change the balance between weights based on the performance of the ASC and SED tasks during model training, achieving some success.

[0059] However, the methods described above are relatively general and cannot take into account the specific characteristics of a particular task. For example, the dataset used in this invention, which contains 29 sound events, has a highly imbalanced data volume between categories, and we believe this factor has a greater impact on model performance. Therefore, we propose Class-Balanced Loss (CB Loss) based on the number of effective samples as an improved method in this invention.

[0060] CB Loss addresses the class imbalance problem by introducing a weighting factor inversely proportional to the number of valid samples. For an input sample x labeled y∈{1,2,...,C} (where C is the total number of classes), the model estimates the class probabilities as P=[p1,p2,...,p C ] T ,in Let the loss be denoted as L(P,y). Assume the number of samples in the i-th class is n. i Then the number of valid samples for the i-th class of data is N represents the volume of the set of all possible data in the feature space of a given class i (N≥1). Then, CB Loss can be expressed by the formula:

[0061]

[0062] The loss functions of ASC and SED in formula (1) are replaced by formula (6) for model training.

[0063] IV. Experimental Section

[0064] (1) Data set and evaluation metrics

[0065] The dataset is derived from the DCASE (Detection and Classification of Acoustic Scenes and Events) datasets TUT Sound Events 2016 / 2017 and TUT Acoustic Scenes 2016, containing a total of 6309 audio files covering 4 sound scene categories and 29 sound event categories. This invention divides the dataset for training, testing, and evaluation, with 70% used for training, 20% for testing, and 10% for evaluation. This invention selects the F1 score as the evaluation metric for the acoustic scene classification task, and for the sound event classification task, selects both the F1 score and the error rate (ER) as evaluation metrics.

[0066] (2) Model training settings

[0067] For feature extraction, this invention selects Mel Frequency Cepstrum Coefficient (MFCC) as the audio feature extraction method. The audio sampling rate is 22.05 kHz, and the Hamming window function is selected. The audio signal after frame-by-frame windowing is subjected to Fast Fourier Transform to obtain the frequency domain signal. After passing through a Mel filter and taking the logarithm, Discrete Cosine Transform (DCT) is then performed to obtain representative features.

[0068] During the training phase, the experiment uniformly set the training set batch size to 128, the test set batch size to 64, the model training iterations to 200, and the Adam optimizer. The experiment was conducted on an NVIDIA RTX2080 Ti, using the Tensorflow and Keras frameworks, and the operating system was Windows 10.

[0069] (3) Experimental Results

[0070] The baseline model constructed in this invention achieved an F1 score of 97.31% on the ASC task. Building upon this, the MMoE model was introduced into the acoustic field and applied, further improving the F1 score to 98.74%, a 1.43% improvement over traditional multi-task learning models. Furthermore, an improved Cross_MMoE model was proposed based on the MMoE model, further enhancing the F1 score to 99.04%.

[0071] Furthermore, to verify the effectiveness of the improved loss function selected in this invention, experiments were conducted on the baseline model, the MMoE model, and the Cross_MMoE model using CB Loss, and compared with the traditional cross-entropy loss function. The results are shown in Table 1. The results demonstrate that the CB Loss selected in this invention outperforms the traditional multi-task learning loss function on all three models, achieving the best F1 score of 99.31% on the Cross_MMoE model. This is attributed to the fact that CB Loss better handles class imbalance, focuses more on learning the minority classes, and improves the model's performance on the entire dataset.

[0072] Table 1

[0073]

[0074] Finally, to verify the superiority of the CB Loss selected in this invention, it was compared with two state-of-the-art loss functions in the field: dynamic weight adaptive loss based on dynamic weighted averaging and dynamic weight adaptive loss based on multifocal loss. To ensure experimental fairness, only the loss function was changed under the same experimental conditions to explore the impact of different loss functions. Table 2 shows the experimental comparison results. The results show that CB Loss achieved the highest F1 score in all three model structures; the dynamic weight adaptive method based on multifocal loss generally performed better, with slightly better results than the traditional multi-task learning loss function on both MMoE and Cross_MMoE models, showing some improvement; while the dynamic weight adaptive method based on dynamic weighted averaging performed the worst, showing improvement only on the baseline model. We believe that compared to the other two loss functions, CB Loss pays more attention to the classification performance of a few classes, and this type of dataset plays an equally important role in the task and cannot be ignored, while the other two loss functions focus more on the overall balance of the data. Of course, each loss function has its own advantages. The performance of loss functions also varies depending on factors such as different datasets, models, and tasks. It is undeniable that CB Loss has achieved the best performance on all three models, which shows that CB Loss is more versatile.

[0075] Table 2

[0076]

[0077] The methods described above according to the invention can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored on a remote recording medium or a non-transitory machine-readable medium and subsequently stored on a local recording medium, downloaded via a network. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the acoustic scene and sound event joint task analysis method described herein. Furthermore, when a general-purpose computer accesses the code used to implement the processing shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for performing the processing shown herein.

[0078] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A method for joint task analysis of acoustic scenes and sound events, characterized in that, Includes the following steps: 1) Audio feature extraction: Collect raw audio data and use Mel-spectrum cepstral coefficients to convert the raw audio data into audio features; 2) Baseline model construction: Construct a shared underlying network, including three sets of 3×3 convolutional and 2×2 max pooling layers. The baseline model uses a CRNN model for the sound event classification task branch and a CNN model for the acoustic scene classification task branch. 3) Baseline model training: Training is performed using the structure and parameters of the baseline model; the model is trained by inputting audio features and learning scene and event perception capabilities; 4) MMoE Model Construction: The initial version of the MMoE model is constructed using multiple expert networks and gating units on a shared underlying network; the CRNN model is used for the sound event classification task branch in the MMoE model, and the CNN model is used for the acoustic scene classification task branch in the MMoE model. Through experimentation and adjustments, the optimal number of expert networks was determined. 5) MMoE model training: Input audio features and train and optimize the MMoE model to learn scene and event perception capabilities; 6) Cross_MMoE model construction and training: Based on the MMoE model, two information interaction branches are added to obtain the Cross_MMoE model; audio features are input and the Cross_MMoE model is trained and optimized to learn scene and event perception capabilities. 7) Class-Balanced Loss Usage: Replace the cross-entropy loss function in the baseline model, MMoE model, and Cross_MMoE model with Class-Balanced Loss for evaluation of acoustic scene classification tasks; 8) Evaluation Model: Evaluate the baseline model, MMoE model and Cross_MMoE model, check their F1 scores on the acoustic scene perception task, and obtain the evaluation results.

2. The method for joint task analysis of acoustic scenes and sound events according to claim 1, characterized in that: In step 4), the underlying model structure of the MMoE model contains three expert networks, each of which is a feedforward network. Then, a gating unit is introduced for each task. The gating unit outputs different weights through the softmax function. The weights are then weighted and summed with the output of each expert network, which serves as the input for the two task branches: acoustic scene and sound event. The MMoE model can explicitly model task relationships and learn task-specific features to leverage shared representations, allowing for automatic parameter assignment to capture shared task information or task-specific information.

3. The method for joint task analysis of acoustic scenes and sound events according to claim 1, characterized in that: In step 6), the output of the sound event classification task branch is added to the output of the shared underlying network and used as the input of the sound scene classification task branch. The same method is used to enrich the input information of the sound event classification task branch and increase the interaction information between the two tasks.

4. The method for joint task analysis of acoustic scenes and sound events according to claim 1, characterized in that: In step 7), Class-Balanced Loss is used instead of the traditional cross-entropy loss function, and a weighting factor that is inversely proportional to the number of effective samples is introduced.

5. The method for joint task analysis of acoustic scenes and sound events according to claim 1, characterized in that: The Class-Balanced Loss is expressed by the following formula: In the formula, P represents the class probability vector estimated by the model, which represents the probability that a sample belongs to each class; the vector length is C, where C is the total number of classes; y represents the true label of the sample, which belongs to one of the categories {1,2,…,C}; Represents: the number of valid samples in the y-th class, indicating the number of valid samples in the target class; calculated using a weighting factor, which is determined by the parameter β; L(P,y) represents the loss function.

Images