Sound scene classification method for double-branch foreground and background feature representation and fusion

Abstract

The invention provides a double-branch foreground and background feature representation and fusion sound scene classification method, which comprises a convolution calculation unit and a classifier, and is characterized by further comprising a foreground feature representation module, a background feature representation module and a self-contained feature cross fusion module, the method comprises the following steps that: a convolution calculation unit extracts time-frequency characteristics of an input audio to obtain an initial audio characteristic spectrogram; then a foreground feature representation module and a background feature representation module process the initial audio feature spectrogram in parallel to obtain foreground audio time-frequency features and background audio time-frequency features respectively; a self-contained feature cross fusion module carries out cross fusion on the foreground audio time-frequency features and the background audio time-frequency features to obtain available classified audio features; and the classifier classifies the sound scene to which the input audio belongs according to the received available classification audio features and then outputs a classification result. By adopting the method to classify the sound scenes, the classification precision of the sound scenes is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of machine hearing technology, and in particular to a double-branch foreground-background feature representation and fusion sound scene classification method. Background Art

[0002] The task of sound scene classification plays an important role in machine hearing research, and it also has great practical application value in the fields of robotic systems, mobile devices and sensors. Sound scene classification aims to use several information analysis methods to learn and model the obtained audio and corresponding labels, and after the new data is input into the model, it can analyze the feature information contained in the audio and identify the scene label corresponding to the audio.

[0003] Scene audio usually contains sounds from a variety of objects or creatures, which are usually divided into foreground event sounds and background event sounds. In scene audio, continuous environmental sounds with specific sound textures such as crowd sounds and traffic sounds usually constitute the background event sounds of the sound scene. On the contrary, some sparsely occurring and short-lasting sounds such as bird calls and human coughs are usually regarded as foreground event sounds. However, the sounds emitted by the same type of objects or creatures may also act as different categories of event sounds in different sound scenes depending on the duration. For example, the lengthy engine sound in the "train" scene audio is usually regarded as a background event sound, while the vehicle engine sound in the "pedestrian street" scene audio against a quiet background is regarded as a foreground event sound. Therefore, the complexity and variability of various event sounds in scene audio pose a huge challenge to the sound scene classification task.

[0004] In order to improve the performance of the sound scene classification model, researchers have done a lot of work on the feature extraction and utilization of foreground and background event sounds. Some researchers used clustering methods to design an additional module to identify representative foreground event sounds in the scene, so as to distinguish sound scenes with similar backgrounds; on the other hand, some researchers used robust principal component analysis to obtain the features of foreground event sounds and background event sounds respectively, and used subspace projection technology to improve the performance of sound scene classification.

[0005] However, the existing sound scene classification methods lack accurate extraction and fusion methods for audio foreground features and audio background features. At the same time, the existing methods use the same weights to perform simple feature splicing on the audio foreground and background features. This method ignores the fact that the degree of influence of audio foreground and background features in the sound scene classification task will vary with different audio scenes. Finally, the existing audio foreground and background feature fusion methods often ignore the nonlinear relationship between audio foreground features and background features. The above factors have resulted in greater room for improvement in the accuracy of sound scene classification in the existing technology. Summary of the invention

[0006] In view of the problems in the background technology, the present invention provides a dual-branch foreground-background feature characterization and fusion sound scene classification method to solve the problem of how to improve the accuracy of sound scene classification.

[0007] To achieve the purpose of the present invention, the present invention provides a dual-branch foreground and background feature characterization and fusion sound scene classification method, including a convolution calculation unit and a classifier, and its innovation lies in: also including a foreground feature characterization module, a background feature characterization module and a self-contained feature cross-fusion module; the foreground feature characterization module and the background feature characterization module are both connected to the convolution calculation unit, the foreground feature characterization module and the background feature characterization module are both connected to the self-contained feature cross-fusion module, and the self-contained feature cross-fusion module is connected to the classifier; the foreground feature characterization module includes a frame-level attention unit and a foreground feature extraction unit, and the foreground feature extraction unit includes a spatially separable convolution layer and a first global average pooling layer; the background feature characterization module includes a complementary frame-level attention unit and a background feature extraction unit, and the background feature extraction unit includes a feature extraction convolution layer and a second global average pooling layer; the self-contained feature cross-fusion module includes a linear dimensionality reduction unit and a linear feature cross unit; the frame-level attention unit is connected to the complementary frame-level attention unit;

[0008] The convolution calculation unit can perform preliminary feature extraction on the input audio; the classifier can classify the sound scene to which the audio belongs according to the input audio features;

[0009] The classification method includes:

[0010] 1) The convolution calculation unit extracts the time-frequency features of the input audio to obtain an initial audio feature spectrogram, and then the convolution calculation unit transmits the initial audio feature spectrogram to the foreground feature representation module and the background feature representation module respectively;

[0011] 2) the foreground feature characterization module processes the initial audio feature spectrogram according to method 1 to obtain the foreground audio time-frequency features, and then the foreground feature characterization module transmits the foreground audio and video features to the self-contained feature cross-fusion module;

[0012] The background feature characterization module processes the initial audio feature spectrogram according to method 2 to obtain the background audio time-frequency feature; then the background feature characterization module transmits the background audio time-frequency feature to the self-contained feature cross-fusion module;

[0013] 3) The self-contained feature cross-fusion module processes the received foreground audio time-frequency features and background audio time-frequency features according to method 3 to obtain usable classification audio features, and then the self-contained feature cross-fusion module transmits the usable classification audio features to the classifier;

[0014] 4) the classifier classifies the sound scene to which the input audio belongs according to the received available classification audio features, and then outputs the classification result;

[0015] The method 1 comprises:

[0016] 1) The frame-level attention unit processes the initial audio feature spectrogram by combining the frame-level attention mechanism and the squeeze-excitation mechanism to obtain the attention weight of the initial audio on the time frame; then the frame-level attention unit uses the attention weight to perform weighted processing on the initial audio feature spectrogram to highlight the foreground audio information, and obtains the foreground audio feature spectrogram;

[0017] The frame-level attention unit transmits the attention weight to the complementary frame-level attention unit; the frame-level attention unit transmits the foreground audio feature spectrogram to the foreground feature extraction unit;

[0018] 2) The foreground feature extraction unit controls the spatially separable convolutional layer to integrate and extract the foreground audio feature spectrogram to obtain the foreground audio time-frequency feature. The foreground feature extraction unit uses the first global average pooling layer to reduce the computational complexity of the foreground audio time-frequency feature integration extraction;

[0019] The second method comprises:

[0020] A) the complementary frame-level attention unit generates a complementary attention weight complementary to the attention weight for highlighting the background audio information according to the received attention weight; then the complementary frame-level attention unit performs weighted processing on the initial audio feature spectrogram to highlight the background audio information using the complementary attention weight, thereby obtaining a background audio feature spectrogram; then the complementary frame-level attention unit transmits the background audio feature spectrogram to the background feature extraction unit;

[0021] B) the background feature extraction unit controls the feature extraction convolution layer to extract the frequency information between each channel of the background audio feature spectrogram to obtain the background audio frequency features of multiple channels, and then the background feature extraction unit controls the second global average pooling layer to integrate the background audio frequency features of multiple channels along the frequency axis direction to obtain the background audio time-frequency features;

[0022] The third method comprises:

[0023] a) the self-contained feature cross-fusion module controls the linear dimension reduction unit to perform channel dimension reduction processing on the received foreground audio time-frequency features and background audio time-frequency features, respectively, to obtain reduced-dimensional foreground audio features and reduced-dimensional background audio features;

[0024] b) The self-contained feature cross-fusion module controls the linear feature cross unit to use the self-contained Cartesian product algorithm to cross-fuse the reduced-dimensionality foreground audio features and the reduced-dimensionality background audio features to obtain usable classification audio features.

[0025] The principle of the present invention is as follows:

[0026] In the sound scene classification task, although there are many classification methods that use foreground and background features, there are still many shortcomings: the existing foreground and background feature characterization methods are difficult to accurately extract the foreground and background features of the input audio, and do not consider the importance of the joint effect of foreground and background features for classification; in addition, in the foreground and background features under a scene label, some features and output scene categories do not always have a linear mapping relationship, but the feature fusion method in the prior art mainly considers the integration of feature information from different sources or intermediate layers, and does not consider the nonlinear relationship between foreground features and background features, which limits the further improvement of the classification model performance.

[0027] In order to accurately extract the foreground audio and background audio in the input audio, the present invention designs a dual-branch time-frequency feature representation structure. After the convolution calculation unit performs preliminary extraction on the input audio, the foreground feature representation module and the background feature representation module are used to extract the foreground audio time-frequency features and the background audio time-frequency features in the initial audio feature spectrogram respectively. Specifically:

[0028] In the foreground feature characterization module, considering that the foreground event sound has a short duration and is usually overlapped with the background event sound, and at the same time, the foreground event sound in audio files of different scenes has the characteristics of great variability and complexity, a frame-level attention unit is set in the foreground feature characterization module. This unit combines the frame-level attention mechanism and the squeeze-excitation mechanism, integrates all features of the initial audio feature spectrogram on the time frame, and obtains the attention weight of the foreground audio feature on the time frame. At the same time, the attention weight is used to perform weighted processing on the audio foreground information and background information to achieve the extraction of foreground audio features and the suppression of background audio features, so as to achieve the purpose of highlighting foreground features and suppressing background features.

[0029] In the background feature characterization module, in order to highlight the background audio features, it is first necessary to reduce the influence of the frame where the foreground event audio is located. In the present invention, on the premise that the attention weights for highlighting the foreground audio features have been obtained in the foreground feature characterization module, complementary attention weights complementary to the above attention weights are used in the background feature characterization module to filter transient noise and short-term frequency changes, so as to achieve the purpose of highlighting the background audio features. In addition, in the deep learning method, the average pooling operation provides a way to smooth feature representation, which helps the model capture more generalized global feature information rather than focusing too much on local texture or edge details. The background feature extraction unit designed by the present invention using this feature can well achieve the goal of reducing the interference of the foreground event sound and obtain the overall frequency characteristics of the background event sound that basically do not change with time.

[0030] The use of complementary attention weights in the foreground feature characterization module and the background feature characterization module takes into account the mutual influence and joint action of the foreground audio features and the background audio features in the sound scene classification. The extraction of features required for sound scene classification is more objective and comprehensive, and the classification accuracy is naturally higher.

[0031] In order to further improve the classification accuracy, the present invention also provides a self-contained feature cross-fusion module, which uses a self-contained Cartesian product calculation method to process the foreground audio time-frequency features and the background audio time-frequency features to obtain the weighted items and cross items of the foreground audio features and the background audio features. The above-mentioned weighted items are used to adaptively weigh the importance of the foreground audio features and the background audio features to the audio classification in different audio scenes, and the cross items help to capture the nonlinear relationship between the two features. Finally, the combined features of the weighted items and the cross items are directly sent to the classifier to enhance the network's ability to characterize nonlinear relationships and achieve accurate classification of audio scenes. While considering the nonlinear relationship between the foreground audio features and the background audio features, the self-contained feature cross-fusion module also adds the cross-fusion items of the foreground audio features and the background audio features to the feature vector that finally participates in the sound scene classification, achieving full consideration of all categories of audio features in the scene, thereby greatly improving the accuracy of sound scene classification.

[0032] It can be seen that the present invention has the following beneficial effects: using the classification method of the present invention to classify sound scenes can greatly improve the classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings of the present invention are as follows.

[0034] Attached Figure 1 A schematic diagram of the network architecture of the present invention.

[0035] In the figure: 1. Convolution calculation unit; 2. Classifier; 3. Foreground feature representation module; 4. Background feature representation module; 5. Self-contained feature cross-fusion module; 31. Frame-level attention unit; 32. Spatially separable convolution layer; 33. First global average pooling layer; 41. Complementary frame-level attention unit; 42. Feature extraction convolution layer; 43. Second global average pooling layer. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the embodiments.

[0037] As attached Figure 1As shown, the present invention involves a convolution calculation unit 1, a classifier 2, a foreground feature characterization module 3, a background feature characterization module 4 and a self-contained feature cross-fusion module 5; the foreground feature characterization module 3 and the background feature characterization module 4 are both connected to the convolution calculation unit 1, the foreground feature characterization module 3 and the background feature characterization module 4 are both connected to the self-contained feature cross-fusion module 5, and the self-contained feature cross-fusion module 5 is connected to the classifier 2; the foreground feature characterization module 3 includes a frame-level attention unit 31 and a foreground feature extraction unit, and the foreground feature extraction unit includes a spatially separable convolution layer 32 and a first global average pooling layer 33; the background feature characterization module 4 includes a complementary frame-level attention unit 41 and a background feature extraction unit, and the background feature extraction unit includes a feature extraction convolution layer 42 and a second global average pooling layer 43; the self-contained feature cross-fusion module 5 includes a linear dimensionality reduction unit 51 and a linear feature cross unit 52; the frame-level attention unit 31 is connected to the complementary frame-level attention unit 41;

[0038] The convolution calculation unit 1 can perform preliminary feature extraction on the input audio, and the convolution calculation unit can adopt the network structure in the prior art, such as an efficient convolution unit; the classifier 2 can classify the sound scene to which the audio belongs according to the input audio features. The classifier in the present invention adopts the network and hardware structure in the prior art;

[0039] The classification method includes:

[0040] 1) The convolution calculation unit extracts the time-frequency features of the input audio to obtain an initial audio feature spectrogram, and then the convolution calculation unit transmits the initial audio feature spectrogram to the foreground feature representation module and the background feature representation module respectively;

[0041] 2) the foreground feature characterization module processes the initial audio feature spectrogram according to method 1 to obtain the foreground audio time-frequency features, and then the foreground feature characterization module transmits the foreground audio and video features to the self-contained feature cross-fusion module;

[0042] The background feature characterization module processes the initial audio feature spectrogram according to method 2 to obtain the background audio time-frequency feature; then the background feature characterization module transmits the background audio time-frequency feature to the self-contained feature cross-fusion module;

[0043] 3) the self-contained feature cross-fusion module 5 processes the received foreground audio time-frequency features and background audio time-frequency features according to method 3 to obtain usable classification audio features, and then the self-contained feature cross-fusion module 5 transmits the usable classification audio features to the classifier 2;

[0044] 4) Classifier 2 classifies the sound scene to which the input audio belongs according to the received available classification audio features, and then outputs the classification result;

[0045] The method 1 comprises:

[0046] 1) The frame-level attention unit 31 processes the initial audio feature spectrogram by combining the frame-level attention mechanism and the squeeze-excitation mechanism to obtain the attention weight of the initial audio on the time frame; then the frame-level attention unit 31 uses the attention weight to perform weighted processing on the initial audio feature spectrogram to highlight the foreground audio information, and obtains the foreground audio feature spectrogram;

[0047] The frame-level attention unit 31 transmits the attention weight to the complementary frame-level attention unit 41; the frame-level attention unit 41 transmits the foreground audio feature spectrogram to the foreground feature extraction unit;

[0048] In fact, unlike the one-time compression of the squeeze excitation mechanism, the frame-level attention unit 31 includes two compression processes of the foreground audio feature spectrogram channel and frequency. The specific process is described as follows: Given the input feature (initial audio feature) graph F0∈R C×Fre×T , the number of channels is C, the frequency dimension size is Fre, and the number of time frames is T. First, in order to integrate information from multiple dimensions, the channels are compressed by 3×3 convolution φ(·) to integrate information between different channels and obtain the fusion feature F1∈R 1 ×Fre×T :

[0049] F1=φ(F0)

[0050] Secondly, in order to obtain the global information of the frequency dimension in each time frame of the audio feature spectrogram, we use global average pooling to fuse the frequency features and obtain the feature vector, Z t ∈R 1×1×T :

[0051]

[0052] Next, a multilayer perceptron consisting of two fully connected layers is used to fit Z t The correlation is calculated, and finally the attention weight A on each time frame of the audio feature spectrogram is obtained through the Sigmoid activation function:

[0053]

[0054] in, represents a multilayer perceptron, and θ(·) represents a Sigmoid activation function. The obtained attention weight A is then multiplied by F0 to obtain an audio feature spectrogram P1 that highlights the foreground audio feature information:

[0055] P1=A×F0

[0056] 2) The foreground feature extraction unit controls the spatially separable convolutional layer 32 to integrate and extract the foreground audio feature spectrogram, i.e., P1, to obtain the foreground audio time-frequency feature. The foreground feature extraction unit uses the first global average pooling layer 33 to reduce the computational complexity of the foreground audio time-frequency feature integration extraction; and finally outputs the feature vector M1∈R that highlights the foreground information. C , where σ(·) represents the convolutional block and pooling layer

[0057] M1=σ(P1)

[0058] The second method comprises:

[0059] A) the complementary frame-level attention unit 41 generates a complementary attention weight complementary to the attention weight for highlighting the background audio information according to the received attention weight; then the complementary frame-level attention unit 41 performs weighted processing on the initial audio feature spectrogram to highlight the background audio information using the complementary attention weight, and obtains the background audio feature spectrogram; then the complementary frame-level attention unit 41 transmits the background audio feature spectrogram to the background feature extraction unit;

[0060] That is to say: F0 passes through the attention vector (1-A∈R 1×1×T After weighting, the feature P2 that highlights the background information can be obtained. In addition, in the deep learning method, the average pooling operation provides a way to smooth the feature representation, which helps the model capture more generalized feature information instead of focusing too much on local texture or edge details. The present invention uses this feature to design a frequency feature representation module, which can reduce the interference of foreground event sounds and obtain the overall frequency feature P′2∈R of the background event sound that does not change over time. C×Fre×1 :

[0061]

[0062] B) The background feature extraction unit controls the feature extraction convolution layer 42 to extract the frequency information between each channel of the background audio feature spectrogram to obtain the background audio frequency features of multiple channels. In fact, the above P′2 is actually a two-dimensional feature map containing only channel dimension and frequency dimension. In order to obtain the frequency information between different channels, the frequency feature representation unit uses two convolutions with a convolution kernel of 1 and a Relu activation function. And through a layer of Sigmiod activation function θ(·) to get the final frequency feature S2∈R C′×Fre×1 :

[0063]

[0064] Among them, C′ is the channel dimension of the final output, C′≤C.

[0065] Then the background feature extraction unit controls the second global average pooling layer 43 to integrate the background audio frequency features of multiple channels along the frequency axis to obtain the background audio time-frequency feature M2:

[0066]

[0067] The third method comprises:

[0068] a) In order to remove the redundant information contained in the audio features and avoid the potential feature dimension explosion problem in subsequent feature fusion, we first send the feature vectors M1 and M2 obtained by the audio foreground and background feature representation module 4 to the linear dimension reduction unit ε(·) for processing, and obtain the reduced dimension foreground audio features M′1∈R n and the reduced-dimensional background audio feature M′2∈R n ;

[0069] M′1=ε(M1)

[0070] M′2=ε(M2)

[0071] b) The inventors found that the audio foreground features play a major role in the sound scene classification task, and the background features are relatively less important. Therefore, in the feature cross unit, a learnable parameter λ is firstly added to the end of the reduced-dimensional foreground audio feature vector M′1, and a fixed value 1 is added to the end of the reduced-dimensional background audio feature vector M′2, to obtain feature vectors M″1 and M″2:

[0072] M″1=[M′1,λ]∈R n+1

[0073] M″2=[M′2,1]∈R n+1

[0074] The learnable parameter λ added at the end of the reduced-dimensional foreground audio feature vector can be adjusted during the model training process to control the weight of the background audio features in the entire classification task.

[0075] Then, the feature crossover unit also uses the self-contained Cartesian product algorithm to calculate the cross-fusion term M of the reduced-dimensionality foreground audio features and the reduced-dimensionality background audio features:

[0076] M=M″1×(M″2) T

[0077] Finally, we concatenate the reduced-dimensionality foreground audio feature M′1 contained in M, the weight-adjustable reduced-dimensionality background audio feature λ⊙M′2, and the cross-fusion features of the two to form the final available classification audio feature vector S for classification:

[0078] S=[M′1,λ⊙M′2,κ(M′1×(M′2) T )]

[0079] where κ(·) is the dimension reduction and flattening operation.

[0080] In order to verify the effect of the scene classification method, the following comparative experiments were conducted on the method of the present invention and the classification method based on the baseline network in the prior art:

[0081] The above-mentioned baseline network is a SpSNet network, which only uses efficient convolution units to realize the extraction of time-frequency features of input audio; the method described in the present invention is hereinafter referred to as DFBFRF (Dual-branch Foreground and Background Feature Representation and Fusion for acoustic scene classification using dual-branch foreground and background feature representation and fusion sound scene classification method).

[0082] In order to test the classification performance of the proposed method in this example, we selected the following three audio classification datasets: DCASE2019, DCASE2021, and DCASE2023. These three datasets are the three TAU urban soundscape mobile development datasets officially released by DCASE in 2019, 2021, and 2023, respectively. Among the three datasets, the audio duration in DCASE2019 and DCASE2021 datasets is 10s, while the duration of all audio samples in DCASE2023 is only 1s.

[0083] Data processing and experimental setup include:

[0084] In the experiment, we used the same processing method for the DCASE2019 and DCASE2021 datasets. We converted all sample audio to mono and used a sampling rate of 44.1KHZ, a window length of 2048, a jump size of 1024, and a frequency dimension size of 256. The final feature map size of the input network was 1×431×256, where 1 represents the number of channels, 256 represents the number of frequency bands, and 431 represents the number of time frames. The audio samples in the DCSAE2023 dataset were also formatted as mono and 44.1kHz. The feature map size of the final input model was 1×44×256, where 1 represents the number of channels, 256 represents the number of frequency bands, and 44 represents the number of frames.

[0085] Specific experiments and analyses include:

[0086] The classification accuracy of the baseline network and DFBFRF in each sound scene category in the DCASE2019, DCASE2021, and DCASE2023 datasets is shown in Table 1 (Baseline Network and DFBFRF Sound Scene Classification Accuracy %). The sound scene classification performance of the DFBFRF overall network is improved by 2.1%, 3.9%, and 2.4% in the DCASE2019, DCASE2021, and DCASE2023 datasets, respectively. The classification accuracy of buses, subways, parks, and squares has improved in all three datasets, especially the network performance of the subway scene category has improved significantly, by 6.3%, 3.4%, and 19.1%, respectively.

[0087] Table 1

[0088]

[0089] The convolution calculation, classifier, frame-level attention mechanism, squeeze-excitation mechanism, audio weighted processing, channel dimensionality reduction processing, self-contained Cartesian product algorithm and SpSNet network used in the present invention are all very common processing methods in the prior art. Those skilled in the art can obtain relevant content from relevant literature of the prior art.

Claims

1. A dual-branch foreground-background feature characterization and fusion sound scene classification method, comprising a convolution calculation unit and a classifier, characterized in that: It also includes a foreground feature characterization module, a background feature characterization module and a self-contained feature cross-fusion module; both the foreground feature characterization module and the background feature characterization module are connected to the convolution calculation unit, both the foreground feature characterization module and the background feature characterization module are connected to the self-contained feature cross-fusion module, and the self-contained feature cross-fusion module is connected to the classifier; the foreground feature characterization module includes a frame-level attention unit and a foreground feature extraction unit, and the foreground feature extraction unit includes a spatially separable convolution layer and a first global average pooling layer; The background feature characterization module includes a complementary frame-level attention unit and a background feature extraction unit, wherein the background feature extraction unit includes a feature extraction convolution layer and a second global average pooling layer; the self-contained feature cross-fusion module includes a linear dimension reduction unit and a linear feature cross unit; the frame-level attention unit is connected to the complementary frame-level attention unit; The convolution calculation unit can perform preliminary feature extraction on the input audio; the classifier can classify the sound scene to which the audio belongs according to the input audio features; The classification method includes: 1) The convolution calculation unit extracts the time-frequency features of the input audio to obtain an initial audio feature spectrogram, and then the convolution calculation unit transmits the initial audio feature spectrogram to the foreground feature representation module and the background feature representation module respectively; 2) the foreground feature characterization module processes the initial audio feature spectrogram according to method 1 to obtain the foreground audio time-frequency features, and then the foreground feature characterization module transmits the foreground audio and video features to the self-contained feature cross-fusion module; The background feature characterization module processes the initial audio feature spectrogram according to method 2 to obtain the background audio time-frequency feature; then the background feature characterization module transmits the background audio time-frequency feature to the self-contained feature cross-fusion module; 3) The self-contained feature cross-fusion module processes the received foreground audio time-frequency features and background audio time-frequency features according to method 3 to obtain usable classification audio features, and then the self-contained feature cross-fusion module transmits the usable classification audio features to the classifier; 4) the classifier classifies the sound scene to which the input audio belongs according to the received available classification audio features, and then outputs the classification result; The method 1 comprises: 1) The frame-level attention unit processes the initial audio feature spectrogram by combining the frame-level attention mechanism and the squeeze-excitation mechanism to obtain the attention weight of the initial audio on the time frame; then the frame-level attention unit uses the attention weight to perform weighted processing on the initial audio feature spectrogram to highlight the foreground audio information, and obtains the foreground audio feature spectrogram; The frame-level attention unit transmits the attention weight to the complementary frame-level attention unit; the frame-level attention unit transmits the foreground audio feature spectrogram to the foreground feature extraction unit; 2) The foreground feature extraction unit controls the spatially separable convolutional layer to integrate and extract the foreground audio feature spectrogram to obtain the foreground audio time-frequency feature. The foreground feature extraction unit uses the first global average pooling layer to reduce the computational complexity of the foreground audio time-frequency feature integration extraction; The second method comprises: A) the complementary frame-level attention unit generates a complementary attention weight complementary to the attention weight for highlighting the background audio information according to the received attention weight; then the complementary frame-level attention unit performs weighted processing on the initial audio feature spectrogram to highlight the background audio information using the complementary attention weight, thereby obtaining a background audio feature spectrogram; then the complementary frame-level attention unit transmits the background audio feature spectrogram to the background feature extraction unit; B) the background feature extraction unit controls the feature extraction convolution layer to extract the frequency information between each channel of the background audio feature spectrogram to obtain the background audio frequency features of multiple channels, and then the background feature extraction unit controls the second global average pooling layer to integrate the background audio frequency features of multiple channels along the frequency axis direction to obtain the background audio time-frequency features; The third method comprises: a) the self-contained feature cross-fusion module controls the linear dimension reduction unit to perform channel dimension reduction processing on the received foreground audio time-frequency features and background audio time-frequency features, respectively, to obtain reduced-dimensional foreground audio features and reduced-dimensional background audio features; b) The self-contained feature cross-fusion module controls the linear feature cross unit to use the self-contained Cartesian product algorithm to cross-fuse the reduced-dimensionality foreground audio features and the reduced-dimensionality background audio features to obtain usable classification audio features.

Images