An audio scene recognition method based on hybrid hierarchical classification

Abstract

The application provides an audio scene recognition method based on a mixed hierarchical classification, comprising the following steps: performing data enhancement on original audio data to obtain enhanced audio data; performing feature extraction on the enhanced audio data to obtain log-mel energy features as input features; inputting the log-mel energy features into a mixed hierarchical network to obtain feature vectors of different hierarchical layers and corresponding probability outputs; and performing weighted summation on the output probabilities of different hierarchical layers to obtain a final probability output and giving a prediction result. The application introduces a hierarchical classification method into the field of audio scene recognition, analyzes and models the hierarchical structure between audio scene classes, and makes the hierarchical structure conform to the human perception mode, so that the accuracy of audio scene recognition is improved.

Description

I. Technical Field

[0001] This invention belongs to the field of audio scene recognition, and in particular relates to an audio scene recognition method based on hybrid hierarchical classification. II. Background Technology

[0002] 1.1 Audio Scene Recognition

[0003] Audio scenes primarily refer to environmental audio recordings in natural or social environments; they are the overall ambient sound formed by the mixing of sounds from a specific real-world scene. Acoustic Scene Classification (ASC) analyzes the characteristic information of audio recordings to identify the audio scene in which they occur. This technology can be applied in fields such as visual assistance, smart homes, and security monitoring. For example, audio scenes are unaffected by light or obstructions, serving as a visual assistance system to help visually impaired individuals identify their location and surrounding environment to determine their next course of action. In the smart home field, audio scene recognition systems can automatically listen for and identify abnormal sounds such as broken windows and fire alarms, thereby enabling intelligent security monitoring and ensuring home safety.

[0004] With the development of deep learning, deep learning-based audio scene recognition methods have gradually become mainstream. Their main processes include data augmentation, data preprocessing, feature extraction, and classification. Currently, various methods have been proposed to improve the accuracy of audio scene recognition in areas such as data preprocessing, feature extraction, and modeling, achieving certain recognition performance.

[0005] However, audio scene recognition still faces several challenges: (1) The acoustic features of the same audio scene exhibit high diversity, while the acoustic features of different audio scenes share certain similarities, making it difficult to extract effective audio scene features. (2) Audio signals lack structured information, making modeling difficult. (3) The distribution of source data and target data is mismatched, such as multiple devices. (4) The traditional ASC algorithm is based on a set of mutually exclusive classes. However, in real-world data, there exists a hierarchical structure between audio scene classes. The existence of these problems results in relatively low accuracy in audio scene recognition.

[0006] 1.2 Multi-level classification method

[0007] Hierarchical classification methods are tree-structured hierarchical structures of rules for concepts defined on a partially ordered set (C, <). Here, C is a finite set listing all class concepts in the application domain, and the relation < represents an "IS-A" relationship. The "IS-A" relationship is asymmetric, reflexive, and transitive, and is mutually exclusive at the same level, and can be represented as:

[0008] (1) The only largest element “R” is the root of the tree;

[0009] (2) If c i <c j Then c i ≥c j ;

[0010] (3) c i ≥c i ;

[0011] (4) c k ∈C, if c i <c j And c j <c k Then there must be c. i <c k .

[0012] For audio scene recognition problems, hierarchical classification algorithms can be determined by the following four factors: First, the path of the predicted label, indicating that the algorithm can assign one or more paths to each data sample in the hierarchical structure. SPP represents a single path, and MPP represents multiple predicted labels for a certain category. Second, the prediction depth, determining whether the algorithm always assigns leaf classes. NMLNP indicates that the algorithm can assign classes (including leaf classes) at any level. Third, the type of class hierarchy, including trees and directed acyclic graphs (DAGs). The difference between trees and DAGs is that each node in a tree has only one parent node, while nodes in a DAG can have multiple parent nodes. Fourth, the type of hierarchical classification algorithm, including per-node local classifier (LCN), per-parent node local classifier (LCPN), and per-local classifier (LCL). The three hierarchical classification algorithms are detailed below:

[0013] (1) The LCN method trains a binary classifier for each node in the class hierarchy except the root node, requiring the selection of both positive and negative samples for training. Since each node in an LCN provides an output indicating whether the input belongs to its node class, it is multi-class. This algorithm uses multiple classifiers, which may lead to inconsistencies in predictions across class levels during testing, thus affecting the system's recognition accuracy. This can usually be addressed using a top-down prediction approach, where the prediction at the current level is determined by the class predicted at its parent node level.

[0014] (2) The LCPN method trains a multi-classifier for each parent node in the hierarchy and predicts its subclasses. Typically, the LCPN method uses the same classifier throughout the hierarchy. Like the LCN algorithm, its multi-classifier suffers from inconsistency in prediction, which needs to be addressed using a top-down prediction approach. However, the top-down prediction approach in LCPN leads to the propagation of errors between layers; a misclassification on one class node can propagate down to all its subclasses, thus affecting the accuracy of the recognition.

[0015] (3) The LCL method trains a multi-class classifier for each level in the hierarchical structure. When all leaf nodes have the same depth, the deepest classifier is a single-level classifier that only predicts the class at the leaf nodes. Since this algorithm has multiple classifiers at different levels, it may also lead to inconsistent class predictions, which is usually corrected using post-processing methods. III. Summary of the Invention

[0016] In view of this, the present invention proposes an audio scene recognition method based on hybrid hierarchical classification, which introduces hierarchical classification method into the field of audio scene recognition, analyzes and models the hierarchical structure between audio scene classes, and makes it conform to human perception patterns, thereby improving the accuracy of audio scene recognition.

[0017] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0018] An audio scene recognition method based on hybrid hierarchical classification includes the following steps:

[0019] Step 1: Perform data augmentation on the original audio data to obtain augmented audio data, thereby increasing the diversity of training samples and improving the generalization of the model;

[0020] Step 2: Extract features from the enhanced audio data and use its log-Mel energy features as input features;

[0021] Step 3: Input the log-Mel energy features into the hierarchical network to obtain the feature vectors of different layers and the corresponding probability outputs;

[0022] Step 4: Weight the output probabilities of different levels and sum them to obtain the final probability output, and give the prediction result.

[0023] Furthermore, there are various data augmentation methods in step 1, without limitation. Here are some examples of data augmentation methods: (1) Mixing. Randomly mix two audio data from the same audio scene class. (2) Pitch shifting. Randomly shift the pitch in a uniform distribution. (3) Time stretching. Randomly change the speed or duration of the audio signal without affecting the pitch. (4) Adding random noise. Randomly add Gaussian noise to the audio data. (5) Dynamic range compression. Reduce the volume of loud sounds or amplify quiet sounds in the audio to compress the dynamic range of the audio signal. (6) Spectral correction. Convert the given input spectrum to the spectrum of the reference device. (7) Random erasure. Randomly replace feature blocks in the audio signal with random numbers. (8) Spectral rolling. Randomly move the spectrum horizontally, vertically, or flattened. (9) Channel obfuscation. Randomly splice or use random weights to splice the harmonic and impulse components of the audio.

[0024] Furthermore, step 2 specifically includes the following steps:

[0025] Step 201: Perform frame segmentation and windowing processing on the data-enhanced audio signal;

[0026] Step 202: Perform a short-time Fourier transform on the short-time signal after frame-segmentation and windowing to convert the audio signal into the time-frequency domain, forming a 2D time spectrum;

[0027] Step 203: Pass the obtained 2D time spectrum through a Mel filter bank and take its logarithm to obtain the log-Mel energy characteristics. The conversion relationship between Mel frequency and frequency in Hz is as follows:

[0028]

[0029] Among them, f mel is the Mel frequency value, and f is the frequency value in Hz;

[0030] Furthermore, the hierarchical classification method described in step 3 can be described as an audio scene with n predefined categories and k levels P1, P2, P3, ..., P k The node at the i-th level is The audio scene classification problem, where m i This represents the total number of nodes in the i-th layer.

[0031] Furthermore, step 3 specifically includes the following steps:

[0032] Step 301: Design and construct a single-level classification model for the hybrid hierarchical classification model. The single-level classification model is designed as an n-classifier, which is non-hierarchical and directly determines the final audio scene class based on the input features;

[0033] Step 302: Design and construct the LCPN model for a hybrid hierarchical classification model. The LCPN model is based on the nodes of the i-th layer. The number of sub-classes is used to design a classifier with a corresponding number of categories. The parent class controls the training data of its sub-classes and a top-down prediction method is used to avoid inconsistencies in class predictions between class levels.

[0034] Step 303: Design and construct the LCL model for a hybrid hierarchical classification model. The LCL model also classifies data based on the nodes of the i-th layer. The number of child nodes corresponds to the number of categories in the classifier. Training is performed on the same dataset with different hierarchical classification labels, without using the hierarchical structure between the parent class and its next-level child classes as a constraint. The P1~P... of the LCPN system... k-1 The classifier of the layer is combined with the n-classifier of the single-level system to form the LCL system, which makes predictions on different level classification labels, thereby completing the construction of the LCL model.

[0035] Furthermore, step 4 specifically includes the following steps:

[0036] Step 401: Input the audio features into the single-level classification model, LCPN model and LCL model built above, train the classifiers of each level of the three models, and obtain the corresponding prediction outputs x1, x2 and x3.

[0037] Step 402: Sum the prediction outputs of the constructed single-level classification model, LCPN model, and LCL model according to their weights to give the final prediction result. The calculation formula is as follows:

[0038] x = ω1x1 + ω2x2 + ω3x3

[0039] Where ω1, ω2, and ω3 are the weight coefficients of the single-level classification model, the LCPN model, and the LCL model, respectively. IV. Description of the attached drawings

[0040] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0041] Figure 1 This is an overall block diagram of the method of the present invention;

[0042] Figure 2 This is a structural diagram illustrating the classification method and hierarchical class heuristic of the present invention.

[0043] Figure 3 This is a structural diagram of the single-level classification model proposed in this invention;

[0044] Figure 4 This is a structural diagram of the LCPN model proposed in this invention;

[0045] Figure 5 This is a structural diagram of the LCL model provided by the present invention;

[0046] Figure 6 This is a structural diagram of the classifier network model used in a specific implementation of the present invention. V. Detailed Implementation Methods

[0047] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] This invention provides an audio scene recognition method based on hybrid hierarchical classification. It summarizes and proposes three non-hierarchical or hierarchical ASC system structures, specifically including a single-level ASC system, an LCPN system, and an LCL system. For the first time, it proposes to combine the three ASC systems to construct a hybrid hierarchical classification system, build a hybrid hierarchical classification network model, make full use of the hierarchical information between audio scene classes, control the propagation error between layers, and effectively use prediction inconsistency to improve the final performance.

[0049] like Figure 1 As shown, the audio scene recognition method based on hybrid hierarchical classification of the present invention specifically includes the following steps:

[0050] Step 1:

[0051] Data augmentation is performed on the original audio to increase the diversity of training samples and improve the generalization of the model, and its log-Mel energy features are extracted as input features.

[0052] There are no restrictions on the data augmentation methods. Here are some examples: (1) Mixing. Randomly mix two audio data from the same audio scene class. (2) Pitch shifting. Randomly shift the pitch in a uniform distribution. (3) Time stretching. Randomly change the speed or duration of the audio signal without affecting the pitch. (4) Adding random noise. Randomly add Gaussian noise to the audio data. (5) Dynamic range compression. Reduce the volume of loud sounds or amplify quiet sounds in the audio to compress the dynamic range of the audio signal. (6) Spectral correction. Convert the given input spectrum to the spectrum of the reference device. (7) Random erasure. Randomly replace feature blocks in the audio signal with random numbers. (8) Spectral rolling. Randomly shift the spectrum horizontally, vertically, or after flattening. (9) Channel obfuscation. Randomly splice or use random weights to splice the harmonic and impulse components of the audio. You can choose one or more of these data augmentation methods to combine.

[0053] Then feature extraction is performed. The audio data is first segmented and windowed to obtain short-time stationary signal frames, and then transformed to the time-frequency domain by short-time Fourier transform to obtain a 2D time spectrum.

[0054] Finally, the obtained 2D time-frequency spectrum is passed through a Mel filter bank, and its logarithm is taken to obtain the logarithmic Mel energy characteristic of the audio signal. The conversion relationship between Mel frequency and frequency in Hz is as follows:

[0055]

[0056] Among them, f mel is the Mel frequency value, and f is the frequency value in Hz.

[0057] Step Two:

[0058] Three basic hierarchical classification models are constructed. Input features are fed into each hierarchical classification model, and feature extraction and probability prediction are performed on the three hierarchical models.

[0059] Specifically, according to Figure 2 The hierarchical structure theory shown constructs Figure 3 , Figure 4 and Figure 5 The three basic ASC models are: single-level classification model, LCPN model, and LCL model. The audio scene recognition problem is described as follows:

[0060] (1) n predefined audio scene categories;

[0061] (2) k levels P1, P2, P3, ..., P k ;

[0062] (3) The node at the i-th level is And there are Where, m i This represents the total number of nodes in the i-th layer.

[0063] Based on the above problem description, we designed and constructed single-level classification models, LCPN models, and LCL models. For example... Figure 3 As shown, the single-level ASC model is designed as an n-classifier, which is non-hierarchical. This classifier directly determines the final audio scene class based on the input features; for example... Figure 4 As shown, the LCPN model is based on the nodes of the i-th layer. The number of subclasses corresponds to the number of subclasses in the class hierarchy. The parent class controls the training data for its subclasses, and a top-down prediction method is used to avoid inconsistencies in class predictions across different hierarchical levels. Figure 5 As shown, the LCL model also depends on the nodes of the i-th layer. The number of child nodes corresponds to the number of categories in the classifier, which is then trained on the same dataset but with different hierarchical classification labels. This is achieved by using the P1~P... of the LCPN system. k-1 The classifier of the layer is combined with the n-classifier of the single-level system to form the LCL system, which makes predictions on different level classification labels.

[0064] Step 3:

[0065] First, the audio features are input into the aforementioned model to train the classifiers at each level of the three models, obtaining the corresponding prediction outputs. Then, the prediction outputs of the constructed single-level classification model, LCPN model, and LCL model are weighted and summed, as shown in the following formula:

[0066] x = ω1x1 + ω2x2 + ω3x3

[0067] Where ω1, ω2, and ω3 are the weight coefficients of the single-level classification model, the LCPN model, and the LCL model, respectively.

[0068] After the above steps, the final prediction result is obtained.

[0069] The following describes the implementation process of the present invention.

[0070] This invention was validated on the DCASE2020 development dataset. This dataset contains 10 audio scene categories, a sampling rate of 44.1 kHz, and 13,965 and 2,970 samples for training and validation sets, respectively, totaling 23,040 audio records, including data from devices A, B, C, and analog devices s1-s6. Each audio scene record is 10 seconds long and recorded via a single channel. The experiment used a two-layer hierarchical structure. The first layer was a coarse classification with three nodes: indoor, outdoor, and traffic. The second layer comprised 10 audio scene classes, where the parent node for airports, subway stations, and shopping malls was indoor; the parent node for parks, squares, sidewalks, and traffic lanes was outdoor; and the parent node for buses, subways, and trams was traffic. The specific process is as follows:

[0071] (1) The audio signal sampling rate is 44.1KHz, the number of points of the short-time Fourier transform (FFT) is 2048, the window size is 40ms, the frame coverage is 20ms, the log-Mel energy spectrum of 128 frequency bands is used as the basic feature, and deltas and deltas-deltas are used to supplement it, and finally the input feature vector with a size of 423×128×3 is obtained.

[0072] (2) A network is constructed for each classifier in the hybrid hierarchical classification model. This network is based on a fully convolutional neural network. For example... Figure 6The diagram shows the network model structure. The network begins with batch normalization, followed by nine zero-padding, stacked convolutional layers: two layers with 48 channels and a 5×5 kernel, using batch normalization and ReLU; five layers with 96 channels and a 3×3 kernel, using batch normalization and ReLU; one layer with 192 channels and a 3×3 kernel, using batch normalization and ReLU; and one layer with 10 channels and a 3×3 kernel, using batch normalization and ReLU. 2×2 max-pooling layers are added after the 2nd, 4th, and 8th stacked convolutional layers. Dropout with a size of 0.3 is applied after the 5th, 6th, and 7th stacked convolutional layers. After the 9th stacked convolutional layer, batch normalization, channel attention, global average pooling, and softmax processing are performed. The specific formulas for ReLU activation and softmax processing are as follows:

[0073] ReLU(f) = max(0, f)

[0074]

[0075] Where f is the feature vector from the convolutional layer. Output the predicted probability for the i-th audio scene.

[0076] (3) Using the constructed fully convolutional neural network to... Figure 1 Each classifier in the hybrid classification system was trained with 300 training epochs and a batch size of 32. Stochastic gradient descent was used for optimization, with a learning rate of 0.1. During training, data augmentation techniques including fusion, spectral correction, and random pruning were employed to improve data quality. Accuracy was used to evaluate the classification performance of the ASC system, with multi-cross-entropy loss also used as an auxiliary evaluation metric. Higher accuracy indicates better classification performance, as shown in the following formula.

[0077]

[0078] Among them, y i It is the one-hot encoding of the real label. This represents the predicted probability value of the model.

[0079] This invention verifies the performance of the proposed hybrid hierarchical classification system using a data augmentation strategy. The results show that the highest accuracy is achieved when the prediction output weight ratio of the single-level ASC system, LCPN system, and LCL system is 1:2:2, with a recognition accuracy of 75.39%.

Claims

1. An audio scene recognition method based on hybrid hierarchical classification, characterized in that: Includes the following steps: Step 1: Perform data augmentation on the original audio data to obtain augmented audio data, thereby increasing the diversity of training samples and improving the generalization of the model; Step 2: Extract features from the enhanced audio data and use its log-Mel energy features as input features; Step 3: Input the log-Mel energy features into the hierarchical network to obtain the feature vectors of different layers and the corresponding probability outputs; Step 4: Weight the output probabilities of different levels and sum them to obtain the final probability output, and give the prediction result; The hierarchical classification method in step 3 is described as a method for... A predefined category of audio scenes, Level , No. The nodes at each level are The audio scene classification problem, in which, Indicates the first The total number of nodes in the layer; specifically including: Step 301: Design and construct a single-level classification model for the hybrid hierarchical classification model; the single-level classification model is designed as a classifier, which is non-hierarchical and directly determines the final audio scene class through input features; Step 302: Design and construct the LCPN model for a hybrid hierarchical classification model; the LCPN model is based on the... Layer nodes The number of sub-classes is used to design a classifier with a corresponding number of categories. The parent class controls the training data of its sub-classes and a top-down prediction method is used to avoid inconsistencies in class predictions between class levels. Step 303: Design and construct the LCL model for the hybrid hierarchical classification model; the LCL model is also based on the first step... Layer nodes The number of child nodes corresponds to the number of categories in the classifier. The classifiers are trained on the same dataset and different hierarchical classification labels, without using the hierarchical structure between the parent class and its next-level child class for constraint. The classifiers of the layers of the LCPN system are combined with the classifiers of the single-level system to form the LCL system, which makes predictions on different hierarchical classification labels, thereby completing the construction of the LCL model.

2. The audio scene recognition method based on hybrid hierarchical classification according to claim 1, characterized in that: Step 2 specifically includes: Step 201: Perform frame segmentation and windowing processing on the data-enhanced audio signal; Step 202: Perform a short-time Fourier transform on the short-time signal after frame-segmentation and windowing to convert the audio signal into the time-frequency domain, forming a 2D time spectrum; Step 203: Pass the obtained 2D time spectrum through a Mel filter bank and take its logarithm to obtain the log-Mel energy feature.

3. The audio scene recognition method based on hybrid hierarchical classification according to claim 1, characterized in that: Step 4 specifically includes: Step 401: Input the audio features into the single-level classification model, LCPN model and LCL model mentioned above, train the classifiers of each level of the three models, and obtain the corresponding prediction outputs; Step 402: Sum the prediction outputs of the constructed single-level classification model, LCPN model and LCL model according to their weights to give the final prediction result.

Images