An audio scene classification method and apparatus

Abstract

The present application relates to the technical field of audio signal processing, and particularly relates to an audio scene classification method and device, the present application uses an audio classification model of Tucker decomposition and tensor regression, the model first enhances and expands a data set for log-mel data; redundancy of convolution weight is reduced by using Tucker decomposition, so that data can be more effectively and intuitively trained and feature extracted; the amount of calculation and the calculation complexity are reduced by using spatial separable convolution, and the running speed of the network is accelerated; then, a tensor regression layer is used to replace a traditional fully connected layer, the multi-mode structure of the data itself is retained, and the parameter quantity is reduced. The problems of high redundancy and parameter quantity of the traditional convolutional neural network, not enough intuitive and effective data features and loss of multi-mode structure information are solved, so that the accuracy of audio scene classification is improved.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, specifically to an audio scene classification method and apparatus. Background Technology

[0002] In recent years, with the increased utilization of audio signals and the massive collection of audio files, analyzing environmental information from large amounts of audio data has become a crucial task. One important aspect of this task is identifying audio features and categorizing them into specific scene types—Acoustic Scene Classification (ASC).

[0003] With the rapid development of neural networks, acoustic scene classification is being applied in a wider range of scenarios, such as context-aware wearable devices, hearing aids, healthcare, security monitoring, smart cities, and the Internet of Things.

[0004] Among them, the baseline system and the PRML (Pattern Recognition and Machine Learning) system are representative acoustic scene classification network models; although they are widely used in acoustic scene classification, their computational accuracy is not high. Summary of the Invention

[0005] This invention provides an audio scene classification method and apparatus, aiming to improve the accuracy of acoustic scene classification.

[0006] The first aspect of this invention provides an audio scene classification method, characterized in that the method includes:

[0007] Obtain the audio information to be categorized;

[0008] Extract the log-Mel features of the audio information to be classified;

[0009] An audio classification model using Tucker decomposition and tensor regression is employed to process log-Mel features, thereby outputting the classification results.

[0010] In one exemplary embodiment of this disclosure, the method for extracting log-Mel features of audio information to be classified includes: performing frame segmentation on the acquired audio information to be classified; then truncating the signal by adding a Haining window to each frame of audio information; performing a fast Fourier transform on the audio information; stacking the time-domain signals of each frame in time to obtain a spectrum; then taking the square of the absolute value of the spectrum to obtain the power spectrum, outputting the power spectrum features to a mel filter bank to obtain the mel spectrum; and finally taking the log of the mel spectrum to obtain the log-mel features.

[0011] In one exemplary embodiment of this disclosure, the Tucker decomposition and tensor regression audio classification model includes log-mel data augmentation, four TS layers, max pooling, and tensor regression layers;

[0012] In the log-mel data augmentation, the log-mel data is augmented and the dataset is expanded;

[0013] In the TS layer, channel features are extracted from the log-mel features, and the width and height of the features are processed;

[0014] In the max pooling process, the texture features of the audio are preserved.

[0015] In the tensor regression layer, the multi-mode structure of the features after average pooling compression is used to apply low-rank constraints to the weights, and the output is represented in a low-rank subspace to complete the classification.

[0016] In one exemplary embodiment of this disclosure, the log-mel data augmentation method includes using mixup data augmentation and SpecAugment augmentation methods to enhance log-mel features.

[0017] In one exemplary embodiment of this disclosure, the TS layer consists of Tucker decomposition convolution, spatially separable convolution, BN, ReLU activation function, and average pooling; training and feature extraction are performed through convolutional Tucker decomposition; the output is fed to spatially separable convolution to process the width and height of features, thereby accelerating the network's operation; then, the output is sequentially fed through BN, ReLU activation function, and average pooling, with average pooling used to preserve local audio features.

[0018] The convolutional layer weights are decomposed using Tucker decomposition. Factors along the input and output channels are used to parameterize 1×1 convolutions, while the remaining two factors are absorbed into the core and used to parameterize regular 2D convolutions. First, the number of input channels is reduced using 1×1 convolutions. Then, smaller convolutions are applied to extract easily classifiable features from the log-mel. Finally, a 1×1 convolution restores the number of channels to the output channel number. The two 1×1 convolutions are used to reduce and increase the channel dimensionality, respectively, primarily to reduce the number of parameters, thereby reducing computational cost. Furthermore, dimensionality reduction allows for more efficient and intuitive data training and feature extraction.

[0019] In one exemplary embodiment of this disclosure, spatially separable convolution is the outer product of a 3×3 kernel and a 3×1 kernel.

[0020] In one exemplary embodiment of this disclosure, the tensor regression layer first uses average pooling to reduce network parameters and feature dimensions, and then estimates the regression weight tensors for the input tensors while keeping the low rank and bias constant.

[0021] First, average pooling is used to reduce network parameters and feature dimensionality. Then, for the input tensor, the regression weight tensor is estimated under fixed low rank and bias. The output is a scalar value (audio category), replacing the inner product between the input and regression weights with tensor shrinkage along the matching pattern. A Tucker structure based on tensor decomposition is used here, but other tensor decomposition structures, such as CP and TT structures, can also be used. The Tensor Regression Layer (TRL) is equivalent to first reducing the dimensionality of the input tensor X using a factor matrix, then imposing low-rank constraints on the weights, and finally applying tensor regression to the result to obtain the classification result y.

[0022] A second aspect of the present invention provides an audio scene classification device, the device comprising:

[0023] The audio information acquisition module is configured to acquire audio information to be classified.

[0024] The acoustic feature extraction module is configured to extract log-Mel features from the audio information to be classified.

[0025] The classification feature enhancement module is configured to perform data augmentation on log-mel features using mixup data augmentation and SpecAugment enhancement methods;

[0026] The feature extraction module is configured to perform training and extract channel features from log-mel features using the TS layer; it is configured to process the width and height of the features using spatially separable convolutions.

[0027] The classification output module is configured to execute a multi-modal structure that uses tensor regression layers and average pooling to compress features, and output the classification result to complete the classification.

[0028] According to a third aspect of this disclosure, this application also provides an electronic device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the audio scene classification method as described above.

[0029] According to a fourth aspect of this disclosure, this application provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the audio scene classification method as described above.

[0030] The beneficial effects achieved by this invention are as follows: This invention solves the problem of complex audio scene classification by using a hybrid tensor decomposition and tensor regression acoustic scene classification model. This model consists of Tucker decomposition convolutions, spatially separable convolutions, and tensor regression layers. Tucker decomposition can reduce the redundancy of convolution weights, allowing for more effective and intuitive training and feature extraction. Spatially separable convolutions reduce computational load and complexity, accelerating network operation. Tensor regression layers replace traditional fully connected layers, preserving the multi-modal structure of the data while reducing the number of parameters. This solves the problems of high redundancy and parameter count, insufficiently intuitive and effective data features, and loss of multi-modal structural information in traditional convolutional neural networks, thereby improving the accuracy of audio scene classification. Attached Figure Description

[0031] Figure 1 This is a flowchart of an audio scene classification method provided in an embodiment of the present invention.

[0032] Figure 2 This is a flowchart of the method for extracting log-Mel features of audio information to be classified according to the present invention.

[0033] Figure 3 It is the extracted log-Mel feature map.

[0034] Figure 4 This is a flowchart of step three of the present invention.

[0035] Figure 5 It is the enhanced log-mel feature obtained after using mixup data augmentation and SpecAugment augmentation.

[0036] Figure 6 This is a structural diagram of the TS layer of the present invention.

[0037] Figure 7 This is a structural diagram of the Tucker decomposition convolution of the present invention.

[0038] Figure 8 This is a structural diagram of the tensor regression layer of this invention.

[0039] Figure 9 This is a structural block diagram of an audio scene classification device provided in an embodiment of the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining this application and are not intended to limit this application.

[0041] It should be noted that the audio scene classification method provided in this application can be implemented by an audio scene classification device. This device can be implemented as part or all of a computer device through software, hardware, or a combination of both. The computer device can be a server or a terminal. The server mentioned above can refer to a single server or a server cluster composed of multiple servers. The terminal mentioned above can be a desktop computer, tablet computer, laptop computer, or smartphone, etc. This application does not limit the specific type of terminal. In the following method embodiments, the implementation subject is always a computer device.

[0042] like Figure 1 The diagram illustrates a flowchart of an audio scene classification method provided in an embodiment of this application. The audio scene classification method includes the following steps:

[0043] Step 1: Obtain the audio information to be categorized;

[0044] Step 2: Extract the log-mel features of the audio information to be classified;

[0045] As an optional embodiment, such as Figure 2 The diagram shows a flowchart of a method for extracting log-Mel features from audio information to be classified. This method includes the following steps:

[0046] Step 201: Perform frame segmentation on the acquired audio information to be classified; each frame of audio information after processing can be regarded as a stationary signal; then, apply a Haining window to each frame of audio information to truncate the signal, with the frame length set to 46ms and the frame shift set to 23ms; the formula for the Haining window is as follows:

[0047]

[0048] Step 202: To obtain more obvious audio features, perform FFT (Fast Fourier Transform) on the audio information processed in Step 201 to convert the time-domain signal into a frequency-domain signal. Stack the time-domain signals of each frame in time to obtain the spectral features X(k) of the audio signal. FFT is a fast algorithm of DCT (Discrete Fourier Transform). The formula of DCT is as follows:

[0049]

[0050] Step 203: Take the absolute value of the spectral features to obtain the amplitude spectrum. Then, take the square of the amplitude spectrum and divide it by the number of points in the FFT (corresponding to the FFT) to obtain the power spectrum feature P(k). The calculation formula is as follows:

[0051]

[0052] Step 204: Generally speaking, the human ear has difficulty perceiving the difference between two very close linear frequencies. However, if we sum up the energy of a frequency range and only focus on how much energy is in that frequency range, the human ear can better distinguish them. Therefore, the power spectrum characteristic P(k) is output to a mel filter bank that conforms to the human hearing system to obtain the mel spectrum. The specific frequency of the filter is obtained through the following formula:

[0053]

[0054] Step 205: The human ear's perception of sound intensity is not linear. To align with the human auditory system, the mel spectrum is logarithmically reduced, and the energy is compressed, ultimately yielding the log-mel feature, as shown below. Figure 3 The image shown is the extracted log-Mel feature map.

[0055] Step 3: Using an audio classification model based on Tucker decomposition and tensor regression, extract the log-Mel feature map after processing in Step 2 to output the classification result.

[0056] As an optional embodiment, such as Figure 4 The flowchart for step 3 is shown below. This invention proposes an audio classification model based on Tucker decomposition convolution and Tensor Regression (TRL). This model consists of log-mel data augmentation, four TRL layers, max pooling, and one TRL layer. Each TRL layer comprises Tucker decomposition convolution, spatially separable convolution, batch normalization (BN), ReLU activation function, and average pooling. Tucker decomposition in the TRL layer reduces the redundancy of convolution weights, making training and feature extraction more effective and intuitive. Spatially separable convolution reduces computational cost and complexity, accelerating network operation. Average pooling better preserves local audio features. After four TRL layers, max pooling effectively preserves the texture features of the audio, making its overall features more prominent. Finally, a TRL layer is introduced to replace the traditional flattening and fully connected layers, preserving the multi-modal structure of the data and reducing the number of parameters, thus achieving better audio classification results.

[0057] As an optional embodiment, step 3 specifically includes the following steps:

[0058] Step 301: Enhance log-mel features using mixup data augmentation and SpecAugment augmentation methods;

[0059] Mixup data augmentation is performed on log-mel features to combine log-mel features from different classes, thereby expanding the dataset.

[0060] SpecAugment enhancement is applied to log-mel features;

[0061] Specifically, the log-mel features are normalized to zero mean using `xx.mean()`; the time axis of the log-mel features is distorted, which is equivalent to stretching and compressing the time dimension of the features, thereby achieving data augmentation; several consecutive rows in the log-mel features are randomly removed to enhance the time dimension; several consecutive columns in the log-mel features are randomly removed to enhance the frequency dimension. Figure 5 , respectively, are the original log-mel features, time-warped features, frequency-domain masking features, and time-masked log-mel features.

[0062] Step 302: Extract the channel features of the audio signal processed in step 301 through four consecutive TS layers:

[0063] like Figure 6 As shown, a TS layer consists of Tucker decomposition convolution, spatially separable convolution, BN, ReLU activation function, and average pooling. Among them, Tucker decomposition convolution in the TS layer can reduce the redundancy of convolution weights, making data training and feature extraction more effective and intuitive; spatially separable convolution can reduce the amount of computation and computational complexity, and speed up the network operation; average pooling can better preserve the features of local audio.

[0064] Log-MEL features are three-dimensional features, with one channel dimension and two spatial dimensions. In this case, the convolutional kernel dimensions are T×S×H×W, where S corresponds to the input channels and T corresponds to the output channels. The original convolutional layer operation is as follows:

[0065]

[0066] The convolution kernel W is decomposed using the Tucker method as shown below:

[0067]

[0068] The convolutional kernel is decomposed into a core tensor and factor matrices over four modes; the rank of the Tucker decomposition of the convolutional kernel is (R1, R2, R3, R4). One advantage of Tucker decomposition is that it does not require decomposing all modes at once. For log-mel features, the spatial information is already relatively small, so the spatial dimension modes consisting of width and height do not require low-rank approximation. Therefore, the spatial dimension matrix... and The tensor H generated after being absorbed into the core tensor G is shown below:

[0069]

[0070] The resulting tensor H is equivalent to not decomposing the spatial dimensional pattern consisting of the log-mel width and height; substituting it into the equation yields the formula for the Tucker decomposed convolution, and the Tucker decomposed convolution F is shown below:

[0071]

[0072] like Figure 7 As shown, the Tucker decomposition convolution first combines the s-dimensionality of X with... The shrinking process is equivalent to transforming the number of input channels of the processed log-mel feature from S to R2 using a 1×1 convolution, corresponding to the first step (pink operation) in the diagram. Then, an H×W parameterized 2D convolution is applied to extract key features, corresponding to the second step (blue operation) in the diagram. Finally, the shrinking process is combined with... Dimensionality reduction is equivalent to applying a 1×1 convolution to increase the number of channels from R2 to the target number of channels T, corresponding to the third step in the figure (orange operation). This reduces the number of parameters and computational cost, and allows for more effective and intuitive training and feature extraction of the data after dimensionality reduction.

[0073] After Tucker decomposes the convolutional processing, the output is the width and height of the spatially separable convolutional processing features, reducing computational cost.

[0074] Specifically, spatially separable convolution is the outer product of a 3×3 kernel and two 3×1 and 1×3 kernels. Each of these two convolutions requires 3 multiplications, reducing the number of multiplications from 9 to 6, thereby reducing the computational cost.

[0075] After spatially separable convolution processing, the output is sequentially processed through BN, ReLU activation functions, and average pooling. The average pooling operation downsamples the overall feature information, which can reduce the dimensionality parameters and pass the useful feature information to the next module.

[0076] Step 303: By processing the output features of step 302 through max pooling, the texture features of the audio can be better preserved, making its overall features more prominent.

[0077] Step 304: Using the multi-modal structure of the features compressed by average pooling through the tensor regression layer, apply low-rank constraints to the weights and represent the output in the low-rank subspace to complete the classification.

[0078] In one exemplary embodiment of this disclosure, such as Figure 8 The diagram shown is a schematic of the tensor regression layer structure;

[0079] By introducing a multimodal structure that preserves the data itself and reduces the number of parameters, tensor regression (TRL) layers can replace traditional flattening and fully connected layers, thus achieving better classification.

[0080] The processed features are represented as The output is Y∈R S×O S is the batch size, and O is the classification label of each batch. Given a fixed low-rank (R0,…,R3,R4) and a bias b, estimate the regression weight tensor. The classification formula is as follows:

[0081] Y =<X,W> 3+b

[0082] The key point of the formula is not the index of the final two tensors being shrunk; tensor regression retains a regression operation. The focus is on finding the optimal parameter values ​​for the final shrinkage. Next, the regression weights are decomposed using Tucker decomposition, expressed as:

[0083]

[0084] The factor matrix U is learned end-to-end with the rest of the network through gradient backpropagation; in this experiment, a variant of the classification formula is used as follows:

[0085] Y = <X×0V (0) ×1V (1 )×…×3V (3) G×4U (4) >3+b

[0086] in Most of the computation is performed in a low-rank subspace, rather than directly in the X dimension. Compressing the input tensor X using a factor matrix, and forcing it to be a low-rank tensor, reduces the storage requirements for key audio features; for example... Figure 8 As shown, the tensor regression layer can determine which category the input is more similar to, thus obtaining the classification result and completing the classification.

[0087] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0088] Experimental verification

[0089] This invention uses the dataset from DCASE2020 Challenge Task 1A, which contains ten audio scene categories and 23,035 audio segments. The training, validation, and test sets are divided in a 7:1.5:1.5 ratio. Unseen cities are used to test accuracy. The experiment was conducted on an Ubuntu 20.04 system using the PyTorch deep learning framework. The epoch was set to 300, the batch size to 16, the loss function to BELows (Cross Entropy Loss), and the Adam optimizer was used. The initial learning rate was set to 0.001.

[0090] The baseline system replaces the use of mel energy with OpenL3 embedding, replaces the CNN network architecture with two fully connected feedforward layers (sizes 512 and 128), sets the epoch to 200, the batch size to 64, the learning rate to 0.001, and uses the Adam optimizer.

[0091] In this experiment, accuracy was chosen as the evaluation method, representing the proportion of all correct predictions. The calculation formula is as follows:

[0092]

[0093] In this context, TP (True Positive) represents a correct positive example, where an instance is a positive class and is correctly identified as such. FN (False Negative) represents an incorrect negative example, a false negative, where an instance that is actually positive is incorrectly identified as negative. FP (False Positive) represents an incorrect positive example, a false positive, where an instance that is actually negative is incorrectly identified as such. TN (True Negative) represents a correct negative example, where an instance that is a negative class and is correctly identified as such.

[0094] The PRML system is an acoustic scene classification system based on Open13 embeddings and RESNET-18 with convolutional block attention modules (CBAM); the test results of each network model are shown in the table below.

[0095] Table 1 Experimental performance of each network model

[0096]

[0097]

[0098] As shown in the table, the acoustic scene classification model based on hybrid tensor decomposition and tensor regression proposed in this invention achieves an accuracy of 62.4% on the development set of DCASE2020 Task 1a, which is 8.3% higher than the baseline method and 2.8% higher than the PRML system. The proposed model using Tucker decomposition shows a slight decrease in average classification accuracy (0.4%) compared to the model without Tucker decomposition. This is because Tucker decomposition convolution sacrifices a small amount of performance for a reduction in the number of parameters. This demonstrates that replacing fully connected layers with tensor regression layers preserves the multimodal structure of the data. This structural information allows the model to better learn the features of each category, thereby improving classification accuracy.

[0099] The parameters of each model are shown in the table below.

[0100] Table 2 Parameter Quantities of Each Model

[0101]

[0102] As can be seen from the table, the proposed model has significantly fewer parameters than the baseline system and the PRML system, making it more suitable for use in lightweight devices. In the proposed model, the convolutions using Tucker decomposition have 23.3% more parameters than those without Tucker decomposition, indicating that Tucker decomposition has a significant impact on the number of parameters. However, as shown in Table 1, the average classification accuracy of the proposed model using Tucker decomposition only decreases by 0.6% compared to the model without Tucker decomposition. Tucker decomposition achieves a significant reduction in the number of parameters at the cost of less performance, demonstrating its superior performance.

[0103] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0104] Based on the same inventive concept, this application also provides an audio scene classification device for implementing the audio scene classification method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more audio scene classification device embodiments provided below can be found in the limitations of the audio scene classification method described above, and will not be repeated here.

[0105] In one embodiment, such as Figure 9 The diagram shown is a structural block diagram of an audio scene classification device provided in an embodiment of this application. The device includes:

[0106] The audio information acquisition module is configured to acquire audio information to be classified.

[0107] The acoustic feature extraction module is configured to extract log-Mel features from the audio information to be classified.

[0108] The classification feature enhancement module is configured to perform data augmentation on log-mel features using mixup data augmentation and SpecAugment enhancement methods;

[0109] The feature extraction module is configured to perform training and extract channel features from log-mel features using the TS layer.

[0110] The classification output module is configured to execute a multi-modal structure that uses tensor regression layers and average pooling to compress features, and output the classification result to complete the classification.

[0111] Each module in the aforementioned audio scene classification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0112] In one embodiment, the present invention also provides an electronic device, which may be a server, comprising a processor, a memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an audio scene classification method.

Claims

1. An audio scene classification method, characterized in that, The method includes: Obtain the audio information to be categorized; Extract the log-Mel features of the audio information to be classified; An audio classification model using Tucker decomposition and tensor regression is used to process log-Mel features, thereby outputting the classification results. The audio classification model based on Tucker decomposition and tensor regression consists of a log-mel data augmentation module, four TS layers, a max pooling layer, and a tensor regression layer connected in sequence. The TS layer consists of a Tucker decomposition convolution, a spatially separable convolution, a BN layer, a ReLU activation function, and an average pooling layer connected in sequence.

2. The audio scene classification method according to claim 1, characterized in that, The method for extracting log-Mel features from audio information to be classified includes: performing frame segmentation on the acquired audio information to be classified; then truncating the signal by adding a Haining window to each frame of audio information; performing a fast Fourier transform on the audio information; stacking the time-domain signals of each frame in time to obtain a spectrum; then taking the square of the absolute value of the spectrum to obtain the power spectrum, outputting the power spectrum features to the MEL filter bank to obtain the MEL spectrum; taking the log of the MEL spectrum to finally obtain the log-MEL features.

3. The audio scene classification method according to claim 1, characterized in that, In the log-mel data augmentation module, mixup data augmentation and SpecAugment augmentation methods are used to enhance log-mel features to expand the dataset; In the TS layer, channel features in log-mel features are extracted by Tucker decomposition convolution, the width and height of the features are processed by spatially separable convolution, and then the local audio features are output by passing through BN layer, ReLU activation function and average pooling layer in sequence. In the max pooling layer, the output features of the four TS layers are pooled to preserve the texture features of the audio. In the tensor regression layer, the multi-modal structure of the processed features is first compressed using average pooling, and then low-rank constraints are applied to the weights to represent the output in a low-rank subspace, thus completing the classification.

4. The audio scene classification method according to claim 3, characterized in that, Spatially separable convolutions are of size The convolution kernel is divided into one and The outer product of the convolution kernel.

5. An audio scene classification device, characterized in that, The device includes: The audio information acquisition module is configured to acquire audio information to be classified. The acoustic feature extraction module is configured to extract log-Mel features from the audio information to be classified. The classification feature enhancement module is configured to perform data augmentation on log-mel features using mixup data augmentation and SpecAugment enhancement methods; The feature extraction module is configured to extract channel features from log-mel features sequentially through four TS layers, process the width and height of the features through spatially separable convolution, and then retain the texture features of the audio through a max pooling layer. The TS layer consists of Tucker decomposition convolution, spatially separable convolution, BN layer, ReLU activation function and average pooling layer connected in sequence; The classification output module is configured to first use the multi-modal structure of the features after average pooling compression through a tensor regression layer, and then apply low-rank constraints to the weights to output the classification result to complete the classification.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images