Self-supervision audio pre-training method and system based on multi-resolution and representation regularization

Abstract

The invention discloses a self-supervision audio pre-training method and system based on multi-resolution and representation regularization, and the method comprises the steps: carrying out the preprocessing of an original audio signal, and generating a time-frequency spectrogram; synchronously extracting multi-scale audio features of the spectrogram in a layered manner through a convolutional audio Transform encoder integrated with a multi-resolution convolutional block, and fusing the multi-scale audio features into unified representation; introducing a representation regularization technology, and carrying out semantic alignment on representation of an intermediate layer of a convolutional audio Transform encoder and audio embedding of an external pre-training audio encoder; predicting main task loss, global representation loss and representation alignment loss based on a mask, and jointly optimizing model parameters to complete audio self-supervision pre-training; the self-supervised audio pre-training system comprises a preprocessing module, a multi-resolution feature extraction module, a representation alignment module, a double-branch training module and a loss calculation module. According to the method, the comprehensiveness and discrimination of audio representation are remarkably improved, the convergence speed is improved by 5 times, and the multi-scene audio understanding requirement is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence deep learning, in particular to a self-supervised audio pre-training method and system based on multi-resolution and representation regularization. BACKGROUND

[0002] Self-Supervised Learning (SSL) has become a powerful paradigm for representation learning, enabling models to learn meaningful patterns from unlabelled data. In the field of audio, SSL methods have achieved remarkable success in transferable audio representation learning, and the learned representations can provide strong support for various downstream tasks such as audio classification and speech recognition. Among them, the methods based on bootstrapping (such as data2vec, EAT and M2D) show excellent performance. These methods use a teacher-student framework, where the student network learns to predict the representation generated by the slowly updated teacher network, achieving efficient audio representation learning.

[0003] Despite the above progress of self-supervised learning in the field of audio, existing audio self-supervised learning methods based on bootstrapping still have obvious limitations: On the one hand, existing methods have the limitation of single granularity representation. Different types of audio events require different granularity of feature representation in time and frequency dimensions. For example, high-frequency transient events require fine time-frequency resolution, while low-frequency continuous environmental sound requires broader context awareness. However, existing audio self-supervised learning methods usually only operate and capture representations at a single time and spectral granularity level, which makes it difficult for the model to effectively capture hierarchical features of audio events across multiple time scales and frequency resolutions, resulting in incomplete audio representations and limiting the performance upper bound of downstream tasks.

[0004] On the other hand, existing methods have the problem of low training efficiency and excessive resource consumption. The acquisition of high-quality audio representations often requires a large amount of computational resources and a long training period, which makes the computational cost of the model development process high, and the development efficiency is seriously affected.

[0005] Therefore, under the premise of not sacrificing the quality of representation, exploring more efficient training strategies has become an active and important research direction in the field of audio self-supervised learning. SUMMARY

[0006] In view of the above defects of the prior art, the present application at least solves the following technical problems: 1. Existing audio self-supervised models only extract features at a single time and spectral granularity level, which cannot effectively capture the hierarchical information inherent in audio signals across different time scales and frequency resolutions, resulting in limitations in the generated audio representations, which in turn limits the performance ceiling of downstream tasks such as audio classification and speech recognition; 2. Existing audio self-supervised pre-training methods have slow convergence speed, and require a large amount of computing resources and a long training period to obtain high-quality audio representations, resulting in high model development costs and low development efficiency, making it difficult to meet the engineering landing and large-scale promotion needs in practical applications.

[0007] The application discloses a multi-resolution and representation regularization self-supervised audio pre-training method, which comprises the following steps: S1: Preprocessing the original audio signal to generate a time-frequency spectrogram; S2: Through a convolutional audio Transformer encoder integrating multi-resolution convolutional blocks, multi-scale audio features of the spectrogram are extracted from fine to coarse granularity in layers, and are fused into a unified representation; S3: Introducing representation regularization technology, aligning the representation of the intermediate layer of the convolutional audio Transformer encoder with the audio embedding of the external pre-trained audio encoder in semantics; S4: Based on the mask prediction main task loss, global representation loss and representation alignment loss, jointly optimizing the model parameters to complete the audio self-supervised pre-training; Further, the multi-resolution convolutional block contains a plurality of resolution sub-blocks, each of which realizes feature embedding through convolution operation with different convolution kernels and step lengths, and synchronously captures audio features of different time scales and frequency resolutions; Further, the convolution module in each resolution sub-block adopts a three-layer convolution layer structure, which contains 1x1 convolution layer, 5x5 convolution layer and 1x1 convolution layer in sequence, respectively used for cross-channel information integration, spatial feature extraction and channel dimension adjustment, and the feature map shape remains unchanged; Further, the representation regularization projects the 768-dimensional CLS representation output by the last layer of the student model through a single fully connected layer to the corresponding dimension of the audio embedding generated by the audio encoder of the external CLAP model, and calculates the difference between the two using a mean square error loss function; Further, the weight coefficients of the mask prediction main task loss, global representation loss and representation alignment loss are all 1.0, which together constitute the overall objective of model optimization; Further, the preprocessing specifically comprises: resampling the original audio to 16kHz single channel, generating a spectrum graph with a dimension of T*128 by adopting a 25ms Hanning window, a 10ms frame shift and 128 mel filter banks; Further, the resolution parameters of the multi-resolution convolution block are configured as 4, 8 and 16, and each resolution sub-block output is unified in scale after a downsampling operation, and then a fusion representation is generated through summation and flattening operations. The application further discloses a multi-resolution and representation regularization self-supervised audio pre-training system based on the self-supervised audio pre-training method. The preprocessing module is configured to perform resampling and mel spectrum conversion on the original audio signal to generate a spectrum graph. The multi-resolution feature extraction module is integrated in the front end of the convolutional audio Transformer encoder and is configured to extract fine-grained to coarse-grained multi-scale audio features in layers and synchronize the features to generate a unified representation. The representation alignment module is configured to perform semantic alignment between the intermediate layer representation in the model and the representation of the external pre-trained audio encoder. The double-branch training module includes a student branch and a teacher branch, and the parameters of the teacher branch are updated asynchronously through an exponential moving average. The loss calculation module is configured to calculate mask block prediction loss, global representation loss and representation alignment loss, and jointly guide model optimization. Further, the student branch includes a convolutional audio Transformer encoder, a projector and a representation alignment head, the projector is composed of five convolutional networks and a fully connected layer, and is configured to map the student encoder output to the teacher target prediction result. Further, the teacher branch includes a convolutional audio Transformer encoder, and the parameters of the convolutional audio Transformer encoder are derived from the exponential moving average update of the student branch parameters.

[0008] The application successfully solves the core pain points of the existing audio self-supervised pre-training technology through the synergistic innovation of the multi-resolution convolution block and the representation regularization technology, and achieves remarkable technical effects: In terms of audio representation quality, traditional mainstream audio representation learning models usually operate on a single level or fixed scale of the network, and the perceptual field of feature extraction is relatively fixed. This makes it difficult for the model to effectively capture the time-frequency structures with large differences in audio signals at the same time: for example, high-frequency transient events (such as keyboard clicks) require fine time-frequency resolution, while low-frequency sustained environmental sounds (such as background noise) require broader context perception. The multi-resolution convolutional block introduced in the present application, as the core component of the convolutional audio Transformer encoder front-end, can extract audio features of different granularity levels simultaneously at the same level of the network by deploying branches with different convolution kernels and step lengths in parallel. Specifically, it can preserve high-frequency details and accurate time-frequency positioning of short-time events at the fine-grained level, and integrate extensive context information at the coarse-grained level to form a macroscopic understanding of the global acoustic scene. This cross-scale, hierarchical feature fusion mechanism enables the model to adaptively process various types of information in audio from micro to macro, significantly enhancing the model's representation ability and classification robustness in complex, multi-source mixed real environments. Through multiple internationally recognized audio understanding benchmark tests, after pre-training on the AudioSet dataset (AS-2M) containing 2 million samples, the mAP of the model supervised by fine-tuning on the AS-20K balanced subset reaches 47.8%, which is 6.9 percentage points higher than the previous best model; the average precision of audio event classification on the AudioSet official test set reaches 50.2%; on the environmental sound classification benchmark ESC-50, a 98.6% accuracy is achieved through five-fold cross-validation, breaking the accuracy record of the dataset; and in the Speech Commands V2 speech command recognition task, an accuracy of 98.3% is achieved, which is comparable to the current optimal result.

[0009] In terms of training efficiency, traditional self-supervised learning relies entirely on the model learning representations from raw data, which usually requires extremely large datasets and lengthy training periods, resulting in high computational costs and slow convergence. The representation regularization technique introduced in the training process of the present invention introduces an additional auxiliary representation alignment loss function in addition to the main task of audio self-supervised training. This function forces the feature representation of the intermediate layer of the student model to align with the embedding space of a high-quality external audio encoder (CLAP) learned on a large number of audio-text pairs. By simply adding the representation alignment loss function during model training, the model is provided with direct supervision signals from powerful pre-trained models that are rich in semantic information. This explicit regularization guides the model parameters to optimize towards more semantically discriminative directions, significantly accelerating the training convergence process and effectively improving the quality of the learned representations, achieving high-performance models in less training resources and shorter time. The convergence speed of the complete model is improved by up to 5 times compared to the ablation variant, and the final performance of the model can surpass the strong baseline model after only about 20,000 steps of pre-training, significantly reducing the computational resources and training time required to achieve the same performance and effectively reducing the research and development costs.

[0010] In terms of engineering practicality, the technical solution of the present invention is completely based on the mainstream deep learning framework PyTorch, which can be directly deployed in a general GPU computing environment without the need for special hardware support, with low landing costs. The trained model has wide applicability and can be used for voice control command recognition and abnormal sound monitoring in smart home, acoustic diagnosis of equipment failure and safety production monitoring in industrial internet, audio classification, label generation and content review in content industry, and enhanced environmental sound perception and hazard event recognition in autonomous driving systems. At the same time, this technology can be used as a basic encoder for audio modalities, combined with large language models to build multi-modal AI systems for deeper audio content understanding and voice interaction experience, with diversified business conversion paths and sustained market competitiveness. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 The present invention is based on a multi-resolution and representation regularization self-supervised audio pre-training method architecture diagram; Figure 2 The present invention is based on a multi-resolution and representation regularization self-supervised audio pre-training method architecture diagram; Figure 3 The present invention is based on a multi-resolution and representation regularization self-supervised audio pre-training method architecture diagram; DETAILED DESCRIPTION

[0012] The technical content of the present application will be more clearly understood and facilitated to be understood by referring to the following description of preferred embodiments of the present application with reference to the accompanying drawings. The present application can be embodied in many different forms and the protection scope of the present application is not limited to the embodiments described herein.

[0013] In the drawings, components of the same structure are denoted by the same reference numerals, and components having similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawing clearer, the thickness of the components is appropriately exaggerated in some places in the drawing.

[0014] The present application discloses a self-supervised audio pre-training method based on multi-resolution and representation regularization, the overall architecture of which is shown in Figure 1 The method comprises the following steps: S1: pre-processing the original audio signal to generate a time-frequency spectrogram; S2: extracting fine-to-coarse multi-scale audio features of the spectrogram in a hierarchical and synchronous manner through a convolutional audio Transformer encoder integrating multi-resolution convolutional blocks, and fusing them into a unified representation; S3: introducing representation regularization technology to semantically align the representation of the intermediate layer of the convolutional audio Transformer encoder with the audio embedding of the external pre-trained audio encoder; S4: based on the mask prediction main task loss, global representation loss and representation alignment loss, jointly optimizing the model parameters to complete the audio self-supervised pre-training.

[0015] The specific implementation of each embodiment will be described in detail below with reference to the accompanying drawings: The self-supervised audio pre-training method of the present application designs a double-branch architecture based on the Bootstrap paradigm, which includes a student branch and a teacher branch. The original audio signal is first resampled to 16kHz single-channel format, and then converted to a time-frequency representation through Mel-spectral analysis, which uses a 25ms Hanning window, a 10ms frame shift and 128 Mel filter banks, finally generating a spectrogram with a dimension of Tx128 as the model input.

[0016] The teacher branch is composed of a convolutional audio transformer (CAT) encoder alone, and the parameters thereof are asynchronously updated by an exponential moving average of the student branch parameters. The student branch comprises three core components: a CAT encoder, a projector, and a representation alignment head. The CAT encoder adopts a cascaded structure, and a multi-resolution feature extraction block is arranged at the front end, and a plurality of transformer modules are connected at the rear end; the projector is composed of five convolutional networks and a fully connected layer, and is responsible for mapping the output of the student encoder to the prediction of the teacher target; and the representation alignment head is a simple fully connected layer, and is used for realizing semantic alignment of the intermediate representation and an external model.

[0017] During the training process, the student branch input adopts an inverse block mask strategy, and the spectral graph blocks are randomly masked at a proportion of 80%. The model training target is guided by three loss functions: the prediction loss based on the masked block adopts a mean square error to calculate the difference between the prediction of the student model for the masked area and the teacher target; the global representation loss is realized by comparing the mean square error between the CLS representation of the final output of the student model and the mean value of the hidden states of the teacher model; and the representation regularization loss is calculated by measuring the alignment degree between the output of a specific layer of the student model and the audio embedding generated by the external CLAP encoder. The three loss terms are respectively assigned a weight coefficient of 1.0, and collectively constitute the overall target of model optimization. The model is pre-trained on the AS-2M subset of the AudioSet, and the Adam optimizer is used in combination with the cosine annealing learning rate scheduling, the batch size is set to 48, and the total training step number reaches 400,000 steps.

[0018] After the complete pre-training process, the performance of the present application and the existing mainstream methods are compared as shown in Table 1: Table 1: Performance comparison table of the present application and the existing mainstream pre-training models

[0019] On the AudioSet dataset containing 2 million samples, the model after supervised fine-tuning reaches an average precision of 50.2% on the official test set, and the mAP on the AS-20K subset reaches 47.8%, which is improved by 6.9 percentage points compared with the previous best model; on the ESC-50 dataset, the accuracy reaches 98.6%, which refreshes the record of the dataset. On the speech command recognition task SpeechCommands V2, the model also performs excellently, and reaches an accuracy of 98.3%, which is the same as the current optimal result.

[0020] In terms of training efficiency, the present application can exceed the final performance of the strong baseline model only by about 20,000 pre-training steps, and the convergence speed of the complete model is improved by 5 times compared with the ablation variant, as shown in Figure 2As shown, the synergistic effect of the multi-resolution architecture and the representation regularization technique is fully demonstrated.

[0021] The multi-resolution convolution block is a hierarchical processing architecture, including three resolution levels, and the corresponding hyperparameter settings are {4, 8, 16}, as shown in Figure 3 As shown, each resolution block is composed of two key components: the patch embedding layer uses a convolution operation with a convolution kernel and a step size of r k+1 / r k , to realize the resolution size and channel dimension transformation of the feature map; the convolution module adopts a three-layer convolution layer structure, which includes 1x1 convolution for cross-channel information integration, 5x5 convolution for spatial feature extraction, and 1x1 convolution for channel dimension adjustment, which can keep the feature map shape unchanged.

[0022] The intermediate representations generated by each resolution block are scaled by specially designed down-sampling blocks, which use a convolution layer with a convolution kernel and a step size of r n / r k , to project the feature maps of different resolutions to a unified size, and finally generate a comprehensive representation that integrates multi-scale information through summation operation and flattening operation, as the input of the subsequent Transformer module.

[0023] Through ablation experiments, under the condition of keeping other conditions unchanged, the performance of different resolution configurations on the AS-20K dataset is compared, and the experimental results are shown in Table 2: Table 2: Performance comparison table of different resolution configurations of the multi-resolution convolution block of the present application

[0024] The model using the {4, 8, 16} resolution set with 11-layer Transformer achieves an mAP performance of 41.6% on the AS-20K dataset when the parameter amount is 91M; while the control model using only a single resolution {16} has a parameter amount of 88M, but the performance decreases to 40.2%; when using a more fine-grained {4, 8, 16, 32} resolution set, the parameter amount increases to 128M, but the performance actually degrades to 38.1%, proving that the multi-resolution configuration used in the present application can bring a performance improvement of 1.4 percentage points at a minimal cost of only increasing 3M parameters, achieving an optimal balance between model complexity and expression ability, and revealing the performance bottleneck that too coarse feature granularity can cause information loss.

[0025] The representation regularization technique improves the training efficiency and model performance by establishing the association between the internal representation and the external prior knowledge. In the specific implementation process, the audio encoder of the CLAP model pre-trained on the audio-text pair is selected as the external teacher, and the high-quality audio embedding generated by the external teacher is used as the optimization target. The representation alignment operation is located on the CLAP embedding of the CLS feature output by the last Transformer block of the student model, and the 768-dimensional feature of the token is projected to the corresponding dimension of the CLAP embedding through a single-layer fully connected network, and then the mean square error between the two is calculated as the regularization loss.

[0026] Through detailed ablation research verification: in the alignment layer selection aspect, when the regularization loss acts on the final Transformer layer, the effect is optimal, and the mAP reaches 47.3% on AS-20K; When the alignment position is moved forward to the middle layer, the performance shows a decreasing trend layer by layer. In the loss weight debugging, the best performance is obtained when the representation loss learning weight λ=1.0, and the performance decays when the value deviates from 5.0 or 0.1. The loss function comparison shows that the mean square error is better than other measurement methods such as cross entropy, L1 loss and cosine similarity, as shown in Table 3: Table 3: Performance comparison table of different parameter settings of the representation alignment loss of the present application

[0027] It is particularly worth noting that after introducing the representation regularization, the model only needs about 20,000 pre-training steps to exceed the final performance of the baseline model, and the convergence speed is improved by 5 times. These experimental conclusions consistently show that the representation regularization scheme adopted by the present application can effectively utilize the external semantic supervision signal, accelerate the training process, and significantly improve the model representation quality.

[0028] The above describes the preferred embodiments of the present application in detail. It should be understood that those skilled in the art can make many modifications and changes to the present application without creative labor based on the concept of the present application. Therefore, any technical solution obtained by logical analysis, reasoning or limited experiment based on the prior art according to the concept of the present application should be within the protection scope determined by the claims.

Claims

1. A self-supervised audio pre-training method based on multi-resolution and representation regularization, characterized in that, The method includes the following steps: S1: Preprocess the original audio signal to generate a time-frequency spectrum; S2: By integrating a convolutional audio Transformer encoder with multi-resolution convolutional blocks, fine-grained to coarse-grained multi-scale audio features of the spectrogram are extracted hierarchically and synchronously, and fused into a unified representation. S3: Introduce representation regularization technology to semantically align the representation of the intermediate layer of the convolutional audio Transformer encoder with the audio embedding of the external pre-trained audio encoder. S4: Based on the mask prediction main task loss, global representation loss and representation alignment loss, the model parameters are jointly optimized to complete the audio self-supervised pre-training.

2. The self-supervised audio pre-training method based on multi-resolution and representation regularization as described in claim 1, characterized in that, The multi-resolution convolutional block contains multiple resolution sub-blocks. Each resolution sub-block embeds features through convolution operations with different kernels and strides, simultaneously capturing audio features at different time scales and frequency resolutions.

3. The self-supervised audio pre-training method based on multi-resolution and representation regularization as described in claim 2, characterized in that, The convolutional module in each resolution sub-block adopts a three-layer convolutional layer structure, which includes a 1×1 convolutional layer, a 5×5 convolutional layer and a 1×1 convolutional layer in sequence, respectively used for cross-channel information integration, spatial feature extraction and channel dimension adjustment, while keeping the feature map shape unchanged.

4. The self-supervised audio pre-training method based on multi-resolution and representation regularization as described in claim 1, characterized in that, The representation regularization projects the 768-dimensional CLS representation output by the last Transformer layer of the student model to the corresponding dimension of the audio embedding generated by the audio encoder of the external CLAP model through a single fully connected layer, and calculates the difference between the two using the mean squared error loss function.

5. The self-supervised audio pre-training method based on multi-resolution and representation regularization as described in claim 1, characterized in that, The weight coefficients of the mask prediction main task loss, global representation loss, and representation alignment loss are all 1.0, which together constitute the overall objective of model optimization.

6. The self-supervised audio pre-training method based on multi-resolution and representation regularization as described in claim 1, characterized in that, The preprocessing specifically includes: resampling the original audio to 16kHz mono, using a 25ms Hanning window, a 10ms frame shift, and 128 Mel filter banks to generate a spectrogram with a dimension of T×128.

7. The self-supervised audio pre-training method based on multi-resolution and representation regularization as described in claim 2, characterized in that, The resolution parameters of the multi-resolution convolutional blocks are configured as 4, 8, and 16. After the outputs of each resolution sub-block are unified in scale through downsampling, a fused representation is generated through summation and flattening operations.

8. A self-supervised audio pre-training system based on multi-resolution and representation regularization, characterized in that, A method for implementing the self-supervised audio pre-training method as described in any one of claims 1 to 7 includes: The preprocessing module is used to resample and perform Mel-frequency conversion on the original audio signal to generate a spectrogram; The multi-resolution feature extraction module, integrated into the front end of the convolutional audio Transformer encoder, is used to extract multi-scale audio features from fine-grained to coarse-grained levels in a hierarchical and synchronous manner and fuse them into a unified representation. The representation alignment module is used to semantically align the representations of the intermediate layers of the model with the representations of the external pre-trained audio encoder. The dual-branch training module includes a student branch and a teacher branch, wherein the parameters of the teacher branch are updated asynchronously through an exponential moving average. The loss calculation module is used to calculate the mask block prediction loss, global representation loss, and representation alignment loss, and jointly guide model optimization.

9. The self-supervised audio pre-training system based on multi-resolution and representation regularization as described in claim 8, characterized in that, The student branch includes a convolutional audio Transformer encoder, a projector, and a representation alignment head. The projector consists of a five-layer convolutional network and a fully connected layer, used to map the student encoder output to the teacher's target prediction result.

10. The self-supervised audio pre-training system based on multi-resolution and representation regularization as described in claim 8, characterized in that, The teacher branch contains a convolutional audio Transformer encoder whose parameters are derived from an exponential moving average update of the student branch parameters.

Images