A lightweight audio scene classification method and system based on deep learning
This lightweight audio scene classification method using deep learning solves the problems of redundant processes and inaccurate feature extraction in audio scene classification, achieving efficient and accurate audio scene classification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHENZHEN BOBEITE TECH DEV CO LTD
- Filing Date
- 2026-04-29
- Publication Date
- 2026-07-14
AI Technical Summary
Existing audio scene classification technologies suffer from process redundancy, generating a large amount of redundant data during feature extraction. They are unsuitable for low-computing-power scenarios and cannot accurately extract temporal correlation information from audio frames, resulting in low classification accuracy.
A lightweight audio scene classification method based on deep learning is adopted. By using sliding frame segmentation, tensor decomposition, context encoding, gradient attribution and importance sampling, a lightweight temporal feature vector is generated to accurately locate key frame sequences and improve classification efficiency and accuracy.
It reduces hardware resource consumption, improves the processing speed and accuracy of audio scene classification, and generates a stable and reliable category probability distribution to meet real-time classification requirements.
Smart Images
Figure CN122392498A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of audio classification technology, and in particular to a lightweight audio scene classification method and system based on deep learning. Background Technology
[0002] Existing audio scene classification technologies suffer from redundancy in signal processing and feature extraction. The framing of the original audio signal is not streamlined or optimized, resulting in a large amount of redundant data during feature extraction. The overall processing is cumbersome and time-consuming, consuming excessive hardware resources and making it unsuitable for low-computing-power scenarios. Furthermore, current technologies lack lightweight mechanisms for processing audio temporal features, leading to feature dimensions and data volume exceeding actual classification requirements. The inefficiency of feature transfer and transformation makes it difficult to meet the needs of real-time audio scene classification.
[0003] Existing audio scene classification technologies cannot accurately extract temporal correlation information from audio frames, nor can they accurately determine the actual role of each audio frame in the classification result. The selection of keyframes lacks scientific basis. The criteria for determining the importance of audio frames in the sampling process are vague, retaining invalid audio frames while omitting core valid frames. This results in low-quality input data for subsequent classification inference, insufficient confidence in the classification results, and ultimately low accuracy in audio scene classification. The overall classification performance fails to meet stable and reliable usage standards. Therefore, improving the efficiency and accuracy of audio scene classification has become an urgent problem to be solved. Summary of the Invention
[0004] This invention provides a lightweight audio scene classification method and system based on deep learning to solve the problems mentioned in the background art.
[0005] To achieve the above objectives, this invention provides a lightweight audio scene classification method based on deep learning, comprising:
[0006] C1. Obtain the original audio signal of the audio scene, and perform sliding frame division on the original audio signal to obtain the temporal audio frame sequence of the audio scene;
[0007] C2. Perform tensor decomposition on the audio frames in the temporal audio frame sequence to obtain the lightweight temporal feature vector of the audio scene;
[0008] C3. Input the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and have the lightweight deep learning network perform context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame.
[0009] C4. Based on the deep temporal feature map, perform gradient attribution on the audio frame to obtain the scene classification contribution of the audio frame;
[0010] C5. Based on the scene classification contribution, perform importance sampling on the temporal audio frame sequence to obtain the key frame sequence of the audio scene;
[0011] C6. Input the keyframe sequence into the classification layer of the lightweight deep learning network, and have the classification layer perform confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determine the final classification result of the audio scene based on the category probability distribution.
[0012] In a preferred embodiment, acquiring the original audio signal of the audio scene and performing sliding frame division on the original audio signal to obtain the temporal audio frame sequence of the audio scene includes:
[0013] Acquire the raw audio signal of the audio scene;
[0014] The original audio signal is converted from analog to digital to obtain the digital audio signal of the audio scene;
[0015] Based on a preset sliding window length, the digital audio signal is segmented into overlapping frames to obtain the overlapping initial audio frames of the audio scene.
[0016] Based on the time index of the original audio signal, the overlapping initial audio frames are reassembled in time to obtain the time-domain audio frame sequence of the audio scene.
[0017] In a preferred embodiment, the step of performing tensor decomposition on the audio frames in the temporal audio frame sequence to obtain the lightweight temporal feature vector of the audio scene includes:
[0018] Level detection is performed on the audio frames in the time-domain audio frame sequence to obtain the amplitude of the sampling points of the audio frames;
[0019] Signal extraction is performed on the audio frames in the time-domain audio frame sequence to obtain the time-domain waveform segments of the audio frames;
[0020] The time-domain waveform segment is vectorized and encoded to obtain a one-dimensional initial tensor of the audio frame;
[0021] Based on the amplitude of the sampling points, a linear projection is performed on the one-dimensional initial tensor to obtain the principal component tensor of the one-dimensional initial tensor.
[0022] Based on the time index of the time-domain audio frame sequence, feature aggregation is performed on the principal component tensor to obtain a lightweight time-domain feature vector of the audio scene.
[0023] In a preferred embodiment, the step of inputting the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and having the lightweight deep learning network perform context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame, includes:
[0024] Construct a pre-trained lightweight deep learning network, which consists of depthwise separable convolutional layers, one-dimensional temporal convolutional layers, and classification layers stacked sequentially.
[0025] The lightweight temporal feature vector is input into the input layer of the lightweight deep learning network;
[0026] In the input layer, the lightweight time-domain feature vector is decoupled sequentially to obtain the time-step feature sub-vector of the lightweight time-domain feature vector;
[0027] A point-based transformation is performed on the time-step feature vector to obtain the spatially enhanced feature vector of the time-step feature vector.
[0028] Based on the time step order of the time step feature vectors, the spatial enhancement feature vectors are recombined in time dimension to obtain a two-dimensional spatial enhancement feature map of the spatial enhancement feature vectors.
[0029] Inter-frame pooling is performed on the two-dimensional spatial enhancement feature map to obtain a preliminary temporal feature map of the two-dimensional spatial enhancement feature map, and hierarchical encoding is performed on the preliminary temporal feature map to obtain a depth temporal feature map of the audio frame.
[0030] In a preferred embodiment, the construction of a pre-trained lightweight deep learning network, wherein the lightweight deep learning network is composed of depthwise separable convolutional layers, one-dimensional temporal convolutional layers, and classification layers stacked sequentially, includes:
[0031] Frame-level mapping is performed on the training audio samples in the training dataset of the audio scene to obtain the training temporal feature vector of the training audio samples. The training dataset contains the training audio samples and the real scene category labels of the training audio samples.
[0032] The training temporal feature vector is input into an untrained lightweight deep learning network, and hierarchical temporal filtering is performed on the training temporal feature vector by a depthwise separable convolutional layer and a one-dimensional temporal convolutional layer to obtain the training deep temporal feature map of the training audio sample.
[0033] The training deep temporal feature map is input into the classification layer, and the classification layer performs category inference on the training deep temporal feature map to obtain the predicted category probability distribution of the training audio sample;
[0034] A deviation analysis is performed between the predicted category probability distribution and the real scene category label to obtain the training loss value of the training audio sample;
[0035] Based on the training loss value, the convolutional layer weight parameters in the lightweight deep learning network are iteratively updated to obtain the final converged network of the lightweight deep learning network, and the final converged network is used as the pre-trained lightweight deep learning network.
[0036] In a preferred embodiment, the step of performing gradient attribution on the audio frame based on the deep temporal feature map to obtain the scene classification contribution of the audio frame includes:
[0037] The deep temporal feature map is passed to the classification layer in the lightweight deep learning network, and the classification layer performs a fully connected transformation on the deep temporal feature map to obtain the original classification score of the audio scene.
[0038] The audio frames in the depth temporal feature map are segmented to obtain a subset of the feature map of the audio frames.
[0039] The gradient attribution initial value of the audio frame is obtained by performing difference estimation between the feature map subset and the original classification score of the category;
[0040] Frame order statistics are performed on the audio frames to obtain the time index order and audio frame sequence length of the audio frames;
[0041] Based on the time index order and the audio frame sequence length, the initial gradient attribution value is mapped to importance to obtain the scene classification contribution of the audio frame.
[0042] In a preferred embodiment, the formula for calculating the scene classification contribution is as follows:
[0043] ;
[0044] In the formula, The time index number is the audio frame in the time-domain audio frame sequence. The first time-domain audio frame in the sequence of audio frames The scene classification contribution of each audio frame. The highest scalar score among the original classification scores of the category output by the classification layer. The first in the depth-time feature map Global aggregation value of a subset of feature maps from each audio frame This is a temporary index variable for the summation operation in the calculation formula. The first in the depth-time feature map Global aggregation value of a subset of feature maps from each audio frame Original classification score for the category For the feature map subset The absolute value of the gradient, Original classification score for the category For the feature map subset The absolute value of the gradient, Original classification score for the category The minute changes The global aggregation value of the feature map subset The minute changes The global aggregation value of the feature map subset The minute changes The frame sequence length of the time-domain audio frame sequence.
[0045] In a preferred embodiment, the step of sampling the temporal audio frame sequence based on the scene classification contribution to obtain the keyframe sequence of the audio scene includes:
[0046] Based on the contribution of the scene classification, the temporal audio frame sequence is prioritized to obtain the contribution ranking sequence of the temporal audio frame sequence;
[0047] Based on the contribution ranking sequence, the temporal audio frame sequence is saliency sampled to obtain the candidate keyframe sequence of the audio scene;
[0048] Based on the time index of the time-domain audio frame sequence, the candidate keyframe sequence is reconstructed in order to obtain the keyframe sequence of the audio scene.
[0049] In a preferred embodiment, the step of inputting the keyframe sequence into the classification layer of the lightweight deep learning network, having the classification layer perform confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determining the final classification result of the audio scene based on the category probability distribution, includes:
[0050] The keyframe sequence is input into the classification layer of the lightweight deep learning network, which is composed of a global pooling sub-layer and a fully connected sub-layer connected in sequence.
[0051] The global pooling sublayer spatially compresses the depth temporal feature map of the key frames in the key frame sequence to obtain the global feature value of the key frame.
[0052] The global feature values are encoded to obtain the global feature description vector of the key frame;
[0053] The global feature description vector is passed to the fully connected sub-layer, which performs a linear transformation on the global feature description vector to obtain the original score vector of the global feature description vector.
[0054] The original score vector is normalized to obtain the category probability distribution of the audio scene;
[0055] Perform peak classification on the category probability distribution to obtain the category with the highest probability of the category probability distribution;
[0056] Based on the category with the highest probability, the sum of the probability values of all other categories in the category probability distribution is evaluated to obtain the final classification result of the audio scene.
[0057] To address the aforementioned problems, this invention also provides a lightweight audio scene classification system based on deep learning, the system comprising:
[0058] The audio framing module is used to acquire the original audio signal of the audio scene and perform sliding framing on the original audio signal to obtain the temporal audio frame sequence of the audio scene.
[0059] The tensor decomposition module is used to perform tensor decomposition on the audio frames in the temporal audio frame sequence to obtain a lightweight temporal feature vector of the audio scene.
[0060] The context encoding module is used to input the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and the lightweight deep learning network performs context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame.
[0061] The gradient attribution module is used to perform gradient attribution on the audio frame based on the deep temporal feature map to obtain the scene classification contribution of the audio frame.
[0062] An importance sampling module is used to perform importance sampling on the temporal audio frame sequence based on the scene classification contribution to obtain the key frame sequence of the audio scene;
[0063] The confidence inference module is used to input the keyframe sequence into the classification layer of the lightweight deep learning network, and the classification layer performs confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determines the final classification result of the audio scene based on the category probability distribution.
[0064] Compared with the prior art, the present invention has the following beneficial effects:
[0065] 1. This invention can directly generate a dimension-adaptive lightweight temporal feature vector by performing normalized sliding frame division and tensor decomposition on the original audio signal, thereby reducing the data volume and transmission time of audio features, reducing the hardware resource consumption in the audio signal processing process, maintaining the stability and smoothness of the audio feature extraction and transmission process, improving the overall processing speed of audio scene classification, and meeting the real-time classification processing requirements of continuous audio signals.
[0066] 2. This invention relies on deep temporal feature maps to complete gradient attribution and importance sampling of audio frames, accurately locking the key frame sequences that play a dominant role in the classification results, ensuring the effectiveness and relevance of the classification input data, and generating a stable and reliable category probability distribution through confidence inference of the classification layer, thereby improving the accuracy and confidence level of audio scene classification results and making the judgment results of audio scene classification more stable and practical. Attached Figure Description
[0067] Figure 1 This is a flowchart illustrating a lightweight audio scene classification method based on deep learning, provided in an embodiment of the present invention.
[0068] Figure 2 A functional block diagram of a lightweight audio scene classification system based on deep learning provided in an embodiment of the present invention;
[0069] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0070] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0071] This application provides a lightweight audio scene classification method based on deep learning. The execution entity of this lightweight audio scene classification method based on deep learning includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application embodiment: a server, a terminal, etc. In other words, the lightweight audio scene classification method based on deep learning can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.
[0072] Reference Figure 1 The diagram shown is a flowchart illustrating a lightweight audio scene classification method based on deep learning, according to an embodiment of the present invention. In this embodiment, the lightweight audio scene classification method based on deep learning includes:
[0073] C1. Obtain the original audio signal of the audio scene, and perform sliding frame division on the original audio signal to obtain the temporal audio frame sequence of the audio scene;
[0074] In this embodiment of the invention, the step of acquiring the original audio signal of the audio scene and performing sliding frame division on the original audio signal to obtain the temporal audio frame sequence of the audio scene includes:
[0075] Acquire the raw audio signal of the audio scene;
[0076] The original audio signal is converted from analog to digital to obtain the digital audio signal of the audio scene;
[0077] Based on a preset sliding window length, the digital audio signal is segmented into overlapping frames to obtain the overlapping initial audio frames of the audio scene.
[0078] Based on the time index of the original audio signal, the overlapping initial audio frames are reassembled in time to obtain the time-domain audio frame sequence of the audio scene.
[0079] The pickup unit completely receives the continuous analog sound wave signal propagating in the audio scene without any signal loss or interruption, stably capturing the audio signal source to be classified and obtaining the original audio signal of the audio scene.
[0080] The original audio signal is input into the analog-to-digital converter module, which performs point-by-point quantization and encoding of the instantaneous amplitude of the analog signal based on a fixed sampling duration. This converts the continuous analog waveform into a discrete digital encoding form, maintaining the integrity of the signal amplitude characteristics throughout the process, and obtaining the digital audio signal of the audio scene.
[0081] Using a preset fixed sliding window length as the truncation reference, the window is uniformly translated along the time axis of the digital audio signal. Each time the translation is completed, the signal segment covered by the window is extracted. Adjacent signal segments are overlapped and truncated at a fixed ratio to completely cover the entire digital audio signal, thus obtaining the overlapping initial audio frame of the audio scene.
[0082] According to the order of the time index generated during the acquisition of the original audio signal, all overlapping initial audio frames are arranged and integrated in sequence, strictly maintaining the time correspondence between each frame and the original signal, completing the regular arrangement of the frame sequence, and obtaining the time domain audio frame sequence of the audio scene.
[0083] The beneficial effects are that through standardized signal acquisition, analog-to-digital conversion, frame stacking and temporal reassembly processes, a temporally accurate and structurally regular temporal audio frame sequence can be generated, ensuring the integrity and consistency of audio signal preprocessing, providing standardized basic data for subsequent feature extraction and classification operations, and improving the stability and reproducibility of the overall processing flow.
[0084] C2. Perform tensor decomposition on the audio frames in the temporal audio frame sequence to obtain the lightweight temporal feature vector of the audio scene;
[0085] In this embodiment of the invention, the step of performing tensor decomposition on the audio frames in the temporal audio frame sequence to obtain the lightweight temporal feature vector of the audio scene includes:
[0086] Level detection is performed on the audio frames in the time-domain audio frame sequence to obtain the amplitude of the sampling points of the audio frames;
[0087] Signal extraction is performed on the audio frames in the time-domain audio frame sequence to obtain the time-domain waveform segments of the audio frames;
[0088] The time-domain waveform segment is vectorized and encoded to obtain a one-dimensional initial tensor of the audio frame;
[0089] Based on the amplitude of the sampling points, a linear projection is performed on the one-dimensional initial tensor to obtain the principal component tensor of the one-dimensional initial tensor.
[0090] Based on the time index of the time-domain audio frame sequence, feature aggregation is performed on the principal component tensor to obtain a lightweight time-domain feature vector of the audio scene.
[0091] For each audio frame in the time-domain audio frame sequence, the instantaneous value of the signal level is detected point by point along the time axis of the audio frame, with the preset sampling point timing position as the detection benchmark. The signal strength data corresponding to each sampling point is completely recorded to ensure that the sampling point and amplitude data correspond one-to-one and there is no missing data, so as to obtain the sampling point amplitude of the audio frame.
[0092] From a single audio frame in a time-domain audio frame sequence, the effective signal content is completely extracted according to the fixed time boundary of the audio frame. Blank segments and interference signals without actual feature expression within the frame are removed, while the core time-domain features such as the waveform fluctuation trajectory and timing correspondence of the signal are completely preserved, thus obtaining the time-domain waveform segment of the audio frame.
[0093] All sampling point data within the time-domain waveform segment are encoded in one-dimensional order according to the chronological order of acquisition, transforming continuous waveform features into single-dimensional discrete structured data. Each data bit precisely corresponds to the feature information of a sampling point, forming a regular one-dimensional data structure, and thus obtaining the one-dimensional initial tensor of the audio frame.
[0094] Using the amplitude of the sampling points as the weight basis for feature mapping, the one-dimensional initial tensor is linearly mapped to the direction with the highest feature contribution. The core components that play a dominant role in the expression of audio features in the tensor are selected and retained, while redundant feature components without practical representational significance are removed, resulting in the principal component tensor of the one-dimensional initial tensor.
[0095] According to the original time index of the temporal audio frame sequence, the principal component tensors corresponding to all audio frames are sequentially fused and sequentially concatenated to integrate the scattered frame-level principal component features into scene-level unified features, compress redundant feature dimensions, and obtain a lightweight temporal feature vector of the audio scene.
[0096] The beneficial effects are that through refined level detection, accurate signal extraction, standardized vectorized encoding, directional linear projection and temporal feature aggregation, the core temporal features of the audio scene can be fully preserved, the dimension and volume of feature data can be greatly reduced, and lightweight temporal feature vectors with high representativeness and low redundancy can be generated, providing concise and reliable feature data for subsequent context encoding, and improving feature processing efficiency and feature expression accuracy.
[0097] C3. Input the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and have the lightweight deep learning network perform context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame.
[0098] In this embodiment of the invention, the step of inputting the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and having the lightweight deep learning network perform context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame, includes:
[0099] Construct a pre-trained lightweight deep learning network, which consists of depthwise separable convolutional layers, one-dimensional temporal convolutional layers, and classification layers stacked sequentially.
[0100] The lightweight temporal feature vector is input into the input layer of the lightweight deep learning network;
[0101] In the input layer, the lightweight time-domain feature vector is decoupled sequentially to obtain the time-step feature sub-vector of the lightweight time-domain feature vector;
[0102] A point-based transformation is performed on the time-step feature vector to obtain the spatially enhanced feature vector of the time-step feature vector.
[0103] Based on the time step order of the time step feature vectors, the spatial enhancement feature vectors are recombined in time dimension to obtain a two-dimensional spatial enhancement feature map of the spatial enhancement feature vectors.
[0104] Inter-frame pooling is performed on the two-dimensional spatial enhancement feature map to obtain a preliminary temporal feature map of the two-dimensional spatial enhancement feature map, and hierarchical encoding is performed on the preliminary temporal feature map to obtain a depth temporal feature map of the audio frame.
[0105] The construction of a pre-trained lightweight deep learning network, comprising a depthwise separable convolutional layer, a one-dimensional temporal convolutional layer, and a classification layer stacked sequentially, includes:
[0106] Frame-level mapping is performed on the training audio samples in the training dataset of the audio scene to obtain the training temporal feature vector of the training audio samples. The training dataset contains the training audio samples and the real scene category labels of the training audio samples.
[0107] The training temporal feature vector is input into an untrained lightweight deep learning network, and hierarchical temporal filtering is performed on the training temporal feature vector by a depthwise separable convolutional layer and a one-dimensional temporal convolutional layer to obtain the training deep temporal feature map of the training audio sample.
[0108] The training deep temporal feature map is input into the classification layer, and the classification layer performs category inference on the training deep temporal feature map to obtain the predicted category probability distribution of the training audio sample;
[0109] A deviation analysis is performed between the predicted category probability distribution and the real scene category label to obtain the training loss value of the training audio sample;
[0110] Based on the training loss value, the convolutional layer weight parameters in the lightweight deep learning network are iteratively updated to obtain the final converged network of the lightweight deep learning network, and the final converged network is used as the pre-trained lightweight deep learning network.
[0111] Following the hierarchical connection specifications of depthwise separable convolutional layers, one-dimensional temporal convolutional layers, and classification layers, the structure of each layer is built, signal channels are connected, and functions are adapted in sequence, forming an untrained lightweight deep learning network infrastructure with complete forward propagation capabilities.
[0112] For all training audio samples in the audio scene training dataset, the temporal features of the samples are mapped one by one to frame-level structured feature data according to the temporal boundary and feature correspondence rules of the audio frames of the samples, ensuring that the features and sample frames are completely matched without information loss, and thus obtaining the training temporal feature vector of the training audio samples.
[0113] The training temporal feature vectors are fed into the input of an untrained lightweight deep learning network. The depthwise separable convolutional layer performs pointwise filtering and core feature extraction on the spatial dimension of the features. The one-dimensional temporal convolutional layer performs frame-wise filtering and temporal feature enhancement on the extracted features in the temporal dimension. After multi-level filtering, the features are integrated to form a complete deep temporal feature representation, resulting in the training deep temporal feature map of the training audio samples.
[0114] The training deep temporal feature map is transmitted to the classification layer of the lightweight deep learning network. The classification layer judges the features one by one according to the preset scene category, generates the corresponding probability value of each scene category, and all the values are combined in order of category to form a complete probability set, thus obtaining the predicted category probability distribution of the training audio sample.
[0115] The values of each item in the predicted category probability distribution are precisely compared with the real scene category labels corresponding to the training audio samples. The quantitative difference value between the two is calculated and recorded. This difference value directly reflects the degree of deviation between the network output and the real label, and the training loss value of the training audio samples is obtained.
[0116] Based on the training loss value as the parameter adjustment basis, the internal weight parameters of the depthwise separable convolutional layer and the one-dimensional temporal convolutional layer in the lightweight deep learning network are successively corrected and adjusted in a targeted manner. The adjustment operation is repeated until the network output no longer changes and reaches a stable convergence state, thus obtaining the final converged network of the lightweight deep learning network. This final converged network is then formally set as the pre-trained lightweight deep learning network.
[0117] The lightweight temporal feature vectors obtained from the previous processing are completely transmitted and accurately connected to the input layer of the pre-trained lightweight deep learning network, ensuring that the feature data is connected without loss or misalignment.
[0118] Inside the input layer of the pre-trained lightweight deep learning network, the overall feature vector is decomposed into independent feature segments corresponding to each time step according to the fixed time step division standard of the lightweight temporal feature vector. Each segment retains all feature information of the corresponding time step, thus obtaining the time step feature sub-vector of the lightweight temporal feature vector.
[0119] Point-to-point feature transformation and enhancement processing is performed on each time step feature sub-vector to enhance the discriminative ability and expressive accuracy of the features within the sub-vector, without changing the temporal attributes and core feature content of the sub-vector, resulting in a spatially enhanced feature vector of the time step feature sub-vector.
[0120] Following the original temporal order of the time step feature vectors, all spatial augmentation feature vectors are arranged and concatenated along the time dimension to construct a two-dimensional structured feature data that simultaneously contains the time dimension and the spatial feature dimension, thus obtaining a two-dimensional spatial augmentation feature map of the spatial augmentation feature vectors.
[0121] The two-dimensional spatial augmented feature map is subjected to feature mean integration processing between adjacent frames to compress redundant features between frames and retain core temporal correlation information, resulting in a preliminary temporal feature map of the two-dimensional spatial augmented feature map. Then, multi-level feature encoding and depth extraction are performed on the preliminary temporal feature map to continuously mine high-order contextual correlation features between frames, resulting in a deep temporal feature map of the audio frame.
[0122] The beneficial effects are that through standardized network construction, iterative training and convergence, and step-by-step context encoding operations, it is possible to efficiently extract deep temporal features of audio based on a lightweight network structure, accurately mine the contextual association information between audio frames, and generate deep temporal feature maps with strong representation capabilities and low redundancy. While reducing the consumption of computing resources, it can significantly improve the accuracy and processing efficiency of audio feature encoding.
[0123] C4. Based on the deep temporal feature map, perform gradient attribution on the audio frame to obtain the scene classification contribution of the audio frame;
[0124] In this embodiment of the invention, the step of performing gradient attribution on the audio frame based on the deep temporal feature map to obtain the scene classification contribution of the audio frame includes:
[0125] The deep temporal feature map is passed to the classification layer in the lightweight deep learning network, and the classification layer performs a fully connected transformation on the deep temporal feature map to obtain the original classification score of the audio scene.
[0126] The audio frames in the depth temporal feature map are segmented to obtain a subset of the feature map of the audio frames.
[0127] The gradient attribution initial value of the audio frame is obtained by performing difference estimation between the feature map subset and the original classification score of the category;
[0128] Frame order statistics are performed on the audio frames to obtain the time index order and audio frame sequence length of the audio frames;
[0129] Based on the time index order and the audio frame sequence length, the initial gradient attribution value is mapped to importance to obtain the scene classification contribution of the audio frame.
[0130] The formula for calculating the contribution of the scene classification is as follows:
[0131] ;
[0132] In the formula, The time index number is the audio frame in the time-domain audio frame sequence. The first time-domain audio frame in the sequence of audio frames The scene classification contribution of each audio frame. The highest scalar score among the original classification scores of the category output by the classification layer. The first in the depth-time feature map Global aggregation value of a subset of feature maps from each audio frame This is a temporary index variable for the summation operation in the calculation formula. The first in the depth-time feature map Global aggregation value of a subset of feature maps from each audio frame Original classification score for the category For the feature map subset The absolute value of the gradient, Original classification score for the category For the feature map subset The absolute value of the gradient, Original classification score for the category The minute changes The global aggregation value of the feature map subset The minute changes The global aggregation value of the feature map subset The minute changes The frame sequence length of the time-domain audio frame sequence.
[0133] The deep temporal feature map is transmitted completely and without loss to the classification layer of the lightweight deep learning network. The classification layer performs linear combination and score mapping of the temporal features of each dimension of the deep temporal feature map through the internal fully connected structure, converting the feature information into the initial judgment score corresponding to each audio scene. The initial scores of all scene categories are summarized in a fixed category order to form a complete score set, and the original classification score of the audio scene category is obtained.
[0134] It is the highest scalar score value among the original classification scores obtained by inputting the deep temporal feature map into the classification layer of a lightweight deep learning network and performing a fully connected transformation. This score value is generated by the forward inference of the trained network and reflects the highest confidence of the current feature map in the corresponding scene classification.
[0135] Based on the fixed feature region boundaries corresponding to a single audio frame in the deep temporal feature map, a region truncation operation with strict boundary alignment is performed on the deep temporal feature map. Only the feature map content exclusive to the target audio frame is retained, and the feature region information corresponding to other audio frames is completely removed. This ensures that the truncated content corresponds one-to-one with the target audio frame and there is no feature confusion, thus obtaining a subset of the feature map of the audio frame.
[0136] Perform global aggregation on the extracted feature map subset to obtain the global aggregation value of the feature map subset. This refers to the deep temporal feature map, and the one with the sequence number. After the feature region corresponding to the audio frame is segmented, the feature map subset obtained by global aggregation processing is then processed. The deep temporal feature map is generated by the lightweight deep learning network by context encoding the lightweight temporal feature vector. This value only reflects the overall feature information of the audio frame.
[0137] In the deep temporal feature map, with the sequence number... After extracting frame segments from the feature regions corresponding to the audio frames, a global aggregation process is performed to obtain a subset of the feature map with a global aggregation value. The method for obtaining this value is similar to... Consistency is used in subsequent statistical calculations.
[0138] Using the changes in feature values of feature map subsets as the analytical benchmark and the changes in the original classification scores of categories as the analytical results, the correlation difference between the feature values of feature subsets and the classification scores is calculated one by one. All the calculated correlation differences are integrated into a unified initial attribution value, which directly reflects the basic influence of feature subsets on classification scores, thus obtaining the initial gradient attribution value of audio frames.
[0139] It is the original classification score of the category. The minute changes, caused by small variations in a subset of the feature map, are fundamental data for analyzing the degree to which features affect classification results.
[0140] It is the global aggregation value of a subset of feature maps. The small change reflects the first The minute adjustments made to a subset of feature maps from each audio frame during the analysis process.
[0141] It is the global aggregation value of a subset of feature maps. The small change reflects the first The minute adjustments made to a subset of feature maps from each audio frame during the analysis process.
[0142] Through analysis and The absolute value of the gradient obtained from the correspondence is the value that reflects the first... The extent to which feature changes in each audio frame affect the highest classification score.
[0143] Through analysis and The absolute value of the gradient obtained from the correspondence is the value that reflects the first... The extent to which feature changes in each audio frame affect the highest classification score.
[0144] All audio frames are sequentially numbered according to the original acquisition time sequence to complete the sequential recording of the time index. At the same time, the total number of all audio frames is counted as the fixed length of the sequence to ensure that the time index order, frame sequence length and original time domain audio frame sequence are completely consistent, thus obtaining the time index order and audio frame sequence length of the audio frames.
[0145] It is a time index number obtained by continuously numbering the audio frames in the time-domain audio frame sequence according to the time sequence of the original audio signal, starting from 1. This index number corresponds one-to-one with the position of the audio frame in the sequence and is used to identify the temporal position of the audio frame.
[0146] It is the total length of the frame sequence obtained by counting all audio frames in the time-domain audio frame sequence. This sequence is generated by analog-to-digital conversion, frame stacking and temporal reconstruction of the original audio signal, and the value is completely consistent with the total number of audio frames contained in the sequence.
[0147] This is a temporary index variable used in subsequent summation operations. The value range of this variable is consistent with the sequence number range of the audio frames, from 1 to... This is used to iterate through each audio frame in the time-domain audio frame sequence to accumulate and count the absolute values of the gradients for all audio frames.
[0148] The absolute values of the gradients corresponding to all audio frames in the sequence are summed to obtain the total influence value of all audio frames on the classification score, providing a denominator benchmark for subsequent calculation of the gradient proportion of individual audio frames.
[0149] The temporal position weight is calculated based on the audio frame number and sequence length. This part assigns weights related to the sequence position to audio frames at different positions, taking into account the temporal characteristics of the audio.
[0150] The weight allocation is based on the temporal position corresponding to the time index order, and the total amount is calculated based on the length of the audio frame sequence. The gradient attribution initial value is combined with the temporal weight and the proportion of the total sequence to perform a directional numerical mapping transformation, generating a quantitative value that can accurately reflect the actual effect of a single audio frame on the scene classification result, and thus obtaining the scene classification contribution of the audio frame.
[0151] It is the first The scene classification contribution of an audio frame is obtained by multiplying the gradient proportion by the temporal position weight. The gradient proportion is the ratio of the gradient influence value of a single audio frame to the sum of the gradient influence values of all audio frames. This value directly reflects the degree of contribution of a single audio frame to the scene classification result.
[0152] This calculation method provides a quantitative basis for gradient attribution of audio frames, realizes the objective quantification of the scene classification contribution of each audio frame, provides a clear judgment standard for the importance sampling of subsequent temporal audio frame sequences, and ensures that the sampled key frame sequences can retain the most valuable information for classification, thereby improving the efficiency and accuracy of subsequent classification.
[0153] The beneficial effects are as follows: a complete process is completed by generating accurate classification scores through fully connected transformation of the classification layer, accurately extracting and globally aggregating single-frame features, quantifying the correlation between features and scores through differential estimation, obtaining standard time-series parameters through frame sequence statistics, and completing contribution measurement through gradient accumulation and weight mapping. This process can accurately and objectively calculate the actual contribution of each audio frame to scene classification, providing a quantitative and reproducible basis for subsequent key frame screening, and improving the accuracy and reliability of audio frame importance determination.
[0154] C5. Based on the scene classification contribution, perform importance sampling on the temporal audio frame sequence to obtain the key frame sequence of the audio scene;
[0155] In this embodiment of the invention, the step of sampling the temporal audio frame sequence based on the scene classification contribution to obtain the keyframe sequence of the audio scene includes:
[0156] Based on the contribution of the scene classification, the temporal audio frame sequence is prioritized to obtain the contribution ranking sequence of the temporal audio frame sequence;
[0157] Based on the contribution ranking sequence, the temporal audio frame sequence is saliency sampled to obtain the candidate keyframe sequence of the audio scene;
[0158] Based on the time index of the time-domain audio frame sequence, the candidate keyframe sequence is reconstructed in order to obtain the keyframe sequence of the audio scene.
[0159] Using the specific value of the scene classification contribution of each audio frame in the time-domain audio frame sequence as the sorting criterion, all audio frames are arranged in descending order of contribution value. During the arrangement process, the original feature information, time index identifier and scene classification contribution value of each audio frame are completely preserved without changing the inherent attributes of any audio frame, forming an ordered frame sequence set based on contribution value as the core criterion, thus obtaining the contribution ranking sequence of the time-domain audio frame sequence.
[0160] Based on the order of the contribution ranking sequence, audio frames are filtered according to a preset fixed effective contribution threshold. Audio frames with scene classification contribution greater than or equal to the preset threshold are extracted from the sequence, and low-value audio frames with scene classification contribution less than the preset threshold are directly removed. After the filtering is completed, a preliminary keyframe set retaining the core effective features is formed, and a candidate keyframe sequence for the audio scene is obtained.
[0161] Using the time index generated during the original acquisition of the time-domain audio frame sequence as the reconstruction standard, all audio frames in the candidate keyframe sequence are rearranged according to their corresponding original time index positions. During the reconstruction process, the feature and contribution information of the audio frames are not adjusted, the natural temporal arrangement structure of the audio frames is restored, and the temporal logic is kept consistent with the original signal, thus obtaining the keyframe sequence of the audio scene.
[0162] The beneficial effects are that, through a standardized process of prioritizing scene classification contribution, sampling with preset threshold significance, and reconstructing the original time index order, effective audio frames that provide core support for audio scene classification can be accurately selected, while redundant audio frames with no classification value are eliminated. This significantly reduces the volume of audio frame sequence data while fully preserving core classification features, providing concise, efficient, and temporally regular input data for subsequent classification layer confidence inference, and significantly improving the running efficiency and result stability of subsequent classification processing.
[0163] C6. Input the keyframe sequence into the classification layer of the lightweight deep learning network, and have the classification layer perform confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determine the final classification result of the audio scene based on the category probability distribution.
[0164] In this embodiment of the invention, the step of inputting the keyframe sequence into the classification layer of the lightweight deep learning network, having the classification layer perform confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determining the final classification result of the audio scene based on the category probability distribution, includes:
[0165] The keyframe sequence is input into the classification layer of the lightweight deep learning network, which is composed of a global pooling sub-layer and a fully connected sub-layer connected in sequence.
[0166] The global pooling sublayer spatially compresses the depth temporal feature map of the key frames in the key frame sequence to obtain the global feature value of the key frame.
[0167] The global feature values are encoded to obtain the global feature description vector of the key frame;
[0168] The global feature description vector is passed to the fully connected sub-layer, which performs a linear transformation on the global feature description vector to obtain the original score vector of the global feature description vector.
[0169] The original score vector is normalized to obtain the category probability distribution of the audio scene;
[0170] Perform peak classification on the category probability distribution to obtain the category with the highest probability of the category probability distribution;
[0171] Based on the category with the highest probability, the sum of the probability values of all other categories in the category probability distribution is evaluated to obtain the final classification result of the audio scene.
[0172] The selected keyframe sequences are transmitted completely and without loss to the input port of the classification layer of the lightweight deep learning network, ensuring that the temporal structure and feature information of the keyframe sequences are not misaligned or lost. The classification layer adopts a fixed structure in which the global pooling sublayer and the fully connected sublayer are connected sequentially. The output of the global pooling sublayer is directly connected to the input of the fully connected sublayer, realizing seamless transfer of feature data and ensuring the coordinated operation of the functions of each sublayer of the classification layer.
[0173] The global pooling sublayer iterates through the deep temporal feature map corresponding to each keyframe in the keyframe sequence, fully covering all spatial dimension feature data of the feature map. It performs a mean integration operation on the spatial dimension of the deep temporal feature map of each keyframe, extracts the core values that can accurately represent the overall features of the keyframe, completely removes meaningless local redundant features and interference information in the feature map, and completes the effective compression of the feature dimension to obtain the global feature value of the keyframe.
[0174] The global feature values corresponding to each keyframe are transformed into an ordered structure according to a preset fixed feature encoding rule. All global feature values are arranged and integrated into a single vector according to the original time index order of the keyframes. This ensures that the position of each value in the vector corresponds one-to-one with the temporal position of the keyframe, thus fully preserving the expressive power and temporal correlation information of the global features, and obtaining the global feature description vector of the keyframe.
[0175] The generated global feature description vector is accurately passed to the fully connected sub-layer of the classification layer. The fully connected sub-layer performs a directional linear mapping transformation on the global feature description vector through comprehensive association operations between all internal nodes and each dimension of the global feature description vector. The feature information is mapped one by one to the initial score data corresponding to each preset audio scene. The initial scores of all scenes are combined in a fixed category order to obtain the original score vector of the global feature description vector.
[0176] Numerical range normalization is performed on all initial score data in the original score vector. Through numerical transformation, all scores are uniformly mapped to a fixed probability range of 0 to 1, ensuring that the sum of scores for all scene categories is 1. This completely eliminates the magnitude difference between scores for different scenes, allowing the scores of each category to be directly used for probability comparison, and obtaining the category probability distribution of the audio scene.
[0177] Perform a comparison operation on the probability values corresponding to all scene categories included in the category probability distribution, identify the probability item with the highest value from all probability values, determine the specific audio scene category corresponding to the highest probability item, complete the accurate matching of the probability peak and the corresponding category, and obtain the maximum probability category of the category probability distribution.
[0178] Using the determined highest probability category as the core criterion, the probability values of all scene categories other than the highest probability category in the category probability distribution are first accumulated one by one to obtain the sum of the probability values of all other categories. Then, the probability value of the highest probability category is precisely compared with the sum. When the probability value of the highest probability category is greater than the sum, the highest probability category is determined to be a valid classification result, and the final classification result of the audio scene is obtained.
[0179] The beneficial effects are that, through the collaborative operation of each sub-layer of the classification layer, the spatial compression, encoding, linear transformation, normalization and category determination of keyframe features are completed in sequence. It can achieve accurate confidence inference based on concise and efficient keyframe sequences, generate a standardized and reliable category probability distribution, and ensure the accuracy and stability of the final classification result through peak classification and advantage evaluation. At the same time, it significantly reduces the computational power consumption of classification inference and improves the overall efficiency and practicality of audio scene classification.
[0180] like Figure 2 The diagram shown is a functional block diagram of a lightweight audio scene classification system based on deep learning provided in an embodiment of the present invention.
[0181] The lightweight audio scene classification system 100 based on deep learning described in this invention can be installed in an electronic device. Depending on the functions implemented, the lightweight audio scene classification system 100 based on deep learning may include an audio framing module 101, a tensor decomposition module 102, a context encoding module 103, a gradient attribution module 104, an importance sampling module 105, and a confidence inference module 106. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.
[0182] In this embodiment, the functions of each module / unit are as follows:
[0183] The audio framing module 101 is used to acquire the original audio signal of the audio scene and perform sliding framing on the original audio signal to obtain the temporal audio frame sequence of the audio scene.
[0184] The tensor decomposition module 102 is used to perform tensor decomposition on the audio frames in the temporal audio frame sequence to obtain a lightweight temporal feature vector of the audio scene.
[0185] The context encoding module 103 is used to input the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and the lightweight deep learning network performs context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame.
[0186] The gradient attribution module 104 is used to perform gradient attribution on the audio frame based on the deep temporal feature map to obtain the scene classification contribution of the audio frame.
[0187] The importance sampling module 105 is used to perform importance sampling on the temporal audio frame sequence according to the scene classification contribution to obtain the key frame sequence of the audio scene;
[0188] The confidence inference module 106 is used to input the keyframe sequence into the classification layer of the lightweight deep learning network, and the classification layer performs confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determines the final classification result of the audio scene based on the category probability distribution.
[0189] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.
[0190] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0191] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.
[0192] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0193] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.
[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A lightweight audio scene classification method based on deep learning, characterized in that, The method includes: C1. Obtain the original audio signal of the audio scene, and perform sliding frame division on the original audio signal to obtain the temporal audio frame sequence of the audio scene; C2. Perform tensor decomposition on the audio frames in the temporal audio frame sequence to obtain the lightweight temporal feature vector of the audio scene; C3. Input the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and have the lightweight deep learning network perform context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame. C4. Based on the deep temporal feature map, perform gradient attribution on the audio frame to obtain the scene classification contribution of the audio frame; C5. Based on the scene classification contribution, perform importance sampling on the temporal audio frame sequence to obtain the key frame sequence of the audio scene; C6. Input the keyframe sequence into the classification layer of the lightweight deep learning network, and have the classification layer perform confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determine the final classification result of the audio scene based on the category probability distribution.
2. The lightweight audio scene classification method based on deep learning as described in claim 1, characterized in that, The step of acquiring the original audio signal of the audio scene and performing sliding frame division on the original audio signal to obtain the temporal audio frame sequence of the audio scene includes: Acquire the raw audio signal of the audio scene; The original audio signal is converted from analog to digital to obtain the digital audio signal of the audio scene; Based on a preset sliding window length, the digital audio signal is segmented into overlapping frames to obtain the overlapping initial audio frames of the audio scene. Based on the time index of the original audio signal, the overlapping initial audio frames are reassembled in time to obtain the time-domain audio frame sequence of the audio scene.
3. The lightweight audio scene classification method based on deep learning as described in claim 1, characterized in that, The step of performing tensor decomposition on the audio frames in the temporal audio frame sequence to obtain the lightweight temporal feature vector of the audio scene includes: Level detection is performed on the audio frames in the time-domain audio frame sequence to obtain the amplitude of the sampling points of the audio frames; Signal extraction is performed on the audio frames in the time-domain audio frame sequence to obtain the time-domain waveform segments of the audio frames; The time-domain waveform segment is vectorized and encoded to obtain a one-dimensional initial tensor of the audio frame; Based on the amplitude of the sampling points, a linear projection is performed on the one-dimensional initial tensor to obtain the principal component tensor of the one-dimensional initial tensor. Based on the time index of the time-domain audio frame sequence, feature aggregation is performed on the principal component tensor to obtain a lightweight time-domain feature vector of the audio scene.
4. The lightweight audio scene classification method based on deep learning as described in claim 1, characterized in that, The step of inputting the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and having the lightweight deep learning network perform context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame, includes: Construct a pre-trained lightweight deep learning network, which consists of depthwise separable convolutional layers, one-dimensional temporal convolutional layers, and classification layers stacked sequentially. The lightweight temporal feature vector is input into the input layer of the lightweight deep learning network; In the input layer, the lightweight time-domain feature vector is decoupled sequentially to obtain the time-step feature sub-vector of the lightweight time-domain feature vector; A point-based transformation is performed on the time-step feature vector to obtain the spatially enhanced feature vector of the time-step feature vector. Based on the time step order of the time step feature vectors, the spatial enhancement feature vectors are recombined in time dimension to obtain a two-dimensional spatial enhancement feature map of the spatial enhancement feature vectors. Inter-frame pooling is performed on the two-dimensional spatial enhancement feature map to obtain a preliminary temporal feature map of the two-dimensional spatial enhancement feature map, and hierarchical encoding is performed on the preliminary temporal feature map to obtain a depth temporal feature map of the audio frame.
5. The lightweight audio scene classification method based on deep learning as described in claim 4, characterized in that, The construction of a pre-trained lightweight deep learning network, comprising a depthwise separable convolutional layer, a one-dimensional temporal convolutional layer, and a classification layer stacked sequentially, includes: Frame-level mapping is performed on the training audio samples in the training dataset of the audio scene to obtain the training temporal feature vector of the training audio samples. The training dataset contains the training audio samples and the real scene category labels of the training audio samples. The training temporal feature vector is input into an untrained lightweight deep learning network, and hierarchical temporal filtering is performed on the training temporal feature vector by a depthwise separable convolutional layer and a one-dimensional temporal convolutional layer to obtain the training deep temporal feature map of the training audio sample. The training deep temporal feature map is input into the classification layer, and the classification layer performs category inference on the training deep temporal feature map to obtain the predicted category probability distribution of the training audio sample; A deviation analysis is performed between the predicted category probability distribution and the real scene category label to obtain the training loss value of the training audio sample; Based on the training loss value, the convolutional layer weight parameters in the lightweight deep learning network are iteratively updated to obtain the final converged network of the lightweight deep learning network, and the final converged network is used as the pre-trained lightweight deep learning network.
6. The lightweight audio scene classification method based on deep learning as described in claim 4, characterized in that, The step of performing gradient attribution on the audio frame based on the deep temporal feature map to obtain the scene classification contribution of the audio frame includes: The deep temporal feature map is passed to the classification layer in the lightweight deep learning network, and the classification layer performs a fully connected transformation on the deep temporal feature map to obtain the original classification score of the audio scene. The audio frames in the depth temporal feature map are segmented to obtain a subset of the feature map of the audio frames. The gradient attribution initial value of the audio frame is obtained by performing difference estimation between the feature map subset and the original classification score of the category; Frame order statistics are performed on the audio frames to obtain the time index order and audio frame sequence length of the audio frames; Based on the time index order and the audio frame sequence length, the initial gradient attribution value is mapped to importance to obtain the scene classification contribution of the audio frame.
7. The lightweight audio scene classification method based on deep learning as described in claim 6, characterized in that, The formula for calculating the contribution of the scene classification is as follows: ； In the formula, The time index number is the audio frame in the time-domain audio frame sequence. The first time-domain audio frame in the sequence of audio frames The scene classification contribution of each audio frame. The highest scalar score among the original classification scores of the category output by the classification layer. The first in the depth temporal feature map Global aggregation value of a subset of feature maps from each audio frame This is a temporary index variable for the summation operation in the calculation formula. The first in the depth-time feature map Global aggregation value of a subset of feature maps from each audio frame Original classification score for the category For the feature map subset The absolute value of the gradient, Original classification score for the category For the feature map subset The absolute value of the gradient, Original classification score for the category The minute changes The global aggregation value of the feature map subset The minute changes The global aggregation value of the feature map subset The minute changes The frame sequence length of the time-domain audio frame sequence.
8. The lightweight audio scene classification method based on deep learning as described in claim 1, characterized in that, The step of sampling the temporal audio frame sequence based on the scene classification contribution to obtain the keyframe sequence of the audio scene includes: Based on the contribution of the scene classification, the temporal audio frame sequence is prioritized to obtain the contribution ranking sequence of the temporal audio frame sequence; Based on the contribution ranking sequence, the temporal audio frame sequence is saliency sampled to obtain the candidate keyframe sequence of the audio scene; Based on the time index of the time-domain audio frame sequence, the candidate keyframe sequence is reconstructed in order to obtain the keyframe sequence of the audio scene.
9. A lightweight audio scene classification method based on deep learning as described in claim 1, characterized in that, The step of inputting the keyframe sequence into the classification layer of the lightweight deep learning network, having the classification layer perform confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determining the final classification result of the audio scene based on the category probability distribution, includes: The keyframe sequence is input into the classification layer of the lightweight deep learning network, which is composed of a global pooling sub-layer and a fully connected sub-layer connected in sequence. The global pooling sublayer spatially compresses the depth temporal feature map of the key frames in the key frame sequence to obtain the global feature value of the key frame. The global feature values are encoded to obtain the global feature description vector of the key frame; The global feature description vector is passed to the fully connected sub-layer, which performs a linear transformation on the global feature description vector to obtain the original score vector of the global feature description vector. The original score vector is normalized to obtain the category probability distribution of the audio scene; Perform peak classification on the category probability distribution to obtain the category with the highest probability. Based on the category with the highest probability, the sum of the probability values of all other categories in the category probability distribution is evaluated to obtain the final classification result of the audio scene.
10. A lightweight audio scene classification system based on deep learning, characterized in that, The system for implementing the lightweight audio scene classification method based on deep learning as described in claim 1 includes: The audio framing module is used to acquire the original audio signal of the audio scene and perform sliding framing on the original audio signal to obtain the temporal audio frame sequence of the audio scene. The tensor decomposition module is used to perform tensor decomposition on the audio frames in the temporal audio frame sequence to obtain a lightweight temporal feature vector of the audio scene. The context encoding module is used to input the lightweight temporal feature vector into a pre-trained lightweight deep learning network, and the lightweight deep learning network performs context encoding on the lightweight temporal feature vector to obtain the deep temporal feature map of the audio frame. The gradient attribution module is used to perform gradient attribution on the audio frame based on the deep temporal feature map to obtain the scene classification contribution of the audio frame. An importance sampling module is used to perform importance sampling on the temporal audio frame sequence based on the scene classification contribution to obtain the key frame sequence of the audio scene; The confidence inference module is used to input the keyframe sequence into the classification layer of the lightweight deep learning network, and the classification layer performs confidence inference on the original audio signal to obtain the category probability distribution of the audio scene, and determines the final classification result of the audio scene based on the category probability distribution.