Cross-domain small sample class incremental audio classification method based on domain generalization representation

Abstract

The invention discloses a cross-domain small sample class incremental audio classification method based on domain generalization representation. The method comprises the following steps: extracting a logarithmic Mel spectrum from an input audio sample; initializing a residual convolutional neural network, wherein the residual convolutional neural network comprises a domain generalization representation extractor and a classifier; in the basic link, a supervised training method and a saliency weighted adversarial training method are successively adopted to train the residual convolutional neural network; extracting characterization from the source domain audio category training sample of the basic link, and calculating and storing a mean vector of the same category characterization for updating the classifier in the increment link; in an increment link, a classifier is updated by adopting an increment audio category training sample of a target domain and a mean vector represented by an old audio category; and inputting a to-be-tested audio sample into the residual convolutional neural network to obtain a category to which the to-be-tested audio sample belongs. According to the method, the domain generalization representation extractor and the confrontation training method of significance weighting are adopted, so that the cross-domain classification performance of the model is effectively improved while overfitting of the model to a new class and forgetting of the model to an old class are relieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of audio signal processing, and particularly relates to a cross-domain small sample class incremental audio classification method based on domain generalization representation. BACKGROUND

[0002] As a core technology of acoustic signal intelligent processing, audio classification has important application value in medical auxiliary diagnosis, ecological monitoring, smart city and other fields. Traditional audio classification methods face two important challenges: first, the fixed class training paradigm is difficult to adapt to the actual needs of dynamically adding classes; second, the dependence of model training on large-scale labeled data is in sharp conflict with the small sample reality scene.

[0003] To cope with the above challenges, researchers have proposed a small sample class incremental audio classification method. This method can continuously learn new classes under the condition of a small number of samples while retaining the memory of base class knowledge. In the basic link, the model is trained using a large number of base class samples; while in the incremental link, only a small number of samples are needed to learn new classes. This method needs to solve two major problems: base class knowledge forgetting and incremental class overfitting. Existing solutions usually adopt an architecture that separates the representation extractor and the classifier, i.e., the representation extractor is fixed after training in the basic link to retain base class features; the classifier is continuously updated to adapt to new classes. Current mainstream methods can be divided into four categories: dynamic small sample learning, prototype classification, random classification, and fully connected classification methods.

[0004] Although existing methods can achieve the dual goals of incremental learning and base class memory, there are still significant limitations. For example, these methods assume that all data comes from a single domain distribution, while in actual scenarios, audio data from different sources often has significant domain shift phenomena. For example, when the model is trained based on laboratory environment data, the distribution changes due to device differences, environmental noise, and other factors when deploying new data. This data heterogeneity in cross-domain scenarios makes it difficult for existing single-domain assumption methods to maintain stable performance. In order to solve the above problems, it is urgent to propose a cross-domain small sample class incremental audio classification method. SUMMARY

[0005] The main purpose of the present application is to overcome the shortcomings and deficiencies of the prior art, and to provide a cross-domain small sample class incremental audio classification method based on domain generalization representation, which organically combines domain generalization representation extractors and saliency weighted adversarial training, so that intelligent audio terminals can continuously learn incremental classes under cross-domain conditions while not forgetting base classes.

[0006] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows: A cross-domain small sample class incremental audio classification method based on domain generalization representation, the audio classification method comprising the following steps: S1, extracting log-mel-spectrogram features from input audio samples, wherein the input audio samples are divided into a basic segment audio category and an incremental segment audio category which are mutually exclusive in category, wherein the basic segment audio category is referred to as a base category, and the base category is from a source domain, and the incremental segment audio category is referred to as an incremental category, and the incremental category is from a target domain; S2, initializing a residual convolutional neural network, wherein the residual convolutional neural network comprises a domain generalization representation extractor and a classifier connected in sequence; S3, in the basic segment, using a supervised training method and a saliency weighted adversarial training method in sequence to train the residual convolutional neural network, and optimizing parameters of the domain generalization representation extractor and the classifier until convergence, and fixing the parameters of the domain generalization representation extractor after training is completed; S4, inputting log-mel-spectrogram features of the base category audio samples into the domain generalization representation extractor to obtain base category representations, and then calculating and saving mean vectors of representations of the same category, which are used to update the classifier in the incremental segment; S5, in the incremental segment, inputting log-mel-spectrogram features of the incremental category audio samples into the domain generalization representation extractor to obtain incremental category representations, and using the incremental category representations, the mean vectors of the base category representations saved in the previous incremental segment, and an incremental loss function to train the residual convolutional neural network, and optimizing parameters of the classifier until convergence; S6, extracting log-mel-spectrogram features from a to-be-tested audio sample, and inputting the log-mel-spectrogram features into the domain generalization representation extractor to obtain a representation of the to-be-tested audio sample, calculating cosine similarity between the representation and weights of each category of the classifier, and the category corresponding to the weight with the highest similarity is the category to which the to-be-tested audio sample belongs, thereby obtaining an audio classification result.

[0007] Further, the extraction process of the log-mel-spectrogram features in step S1 is as follows: S1.1, pre-emphasizing the audio sample using a first-order high-pass filter, and a transfer function of the filter is wherein represents a coefficient of the high-pass filter, and the value range is 0-1; S1.2, dividing the pre-emphasized audio sample into short-time segments, and obtaining windowed short-time audio frames through window function weighting; S1.3, implementing discrete Fourier transform on the windowed short-time audio frames to generate corresponding linear spectra; S1.4, converting the linear spectra into mel spectra using mel filters, and obtaining log-mel spectra by taking logarithm of the mel spectra.

[0008] Further, the structure of the domain generalization representation extractor comprises 1 input convolutional layer, 4 residual layers and 1 average pooling layer connected in sequence, wherein each residual layer is composed of 2 residual blocks with the same structure, each residual block comprises 2 convolutional layers, each convolutional layer is connected with a self-attention normalization layer and a ReLU activation layer in sequence, and feature fusion is realized by bitwise addition operation; the structure of the classifier comprises 1 fully connected layer and 1 Softmax layer.

[0009] The input convolutional layer comprises 1 64-channel 3x3 convolutional kernel, 1 self-attention normalization layer, 1 ReLU activation layer and 1 max-pooling layer connected in sequence. Convolution operation converts the input feature map into a high-dimensional feature representation, while gradually reducing the spatial resolution, and retains key feature information by using nonlinear transformation and feature selection mechanism, thereby enhancing the expression ability of the network to the input features and optimizing the feature extraction process of the subsequent layers.

[0010] The 4 residual layers have independent and different parameters, and each residual layer is composed of two residual blocks with the same structure. Each residual block comprises 2 convolutional layers, each convolutional layer is connected with a self-attention normalization layer and a ReLU activation layer in sequence, and feature fusion is realized by bitwise addition operation. Through this hierarchical structure, the network can gradually extract deep semantic features, effectively improving the representation ability of the network to the audio features while increasing the depth of the network.

[0011] The average pooling layer is used to perform average pooling operation. By performing dimension reduction operation on the feature map, the feature dimension and the computational complexity are significantly reduced, which enhances the generalization performance of the model while effectively reducing the number of model parameters.

[0012] The self-attention normalization layer comprises 1 feature transformation layer and 1 attention feature fusion layer, wherein the structure of the feature transformation layer comprises three parallel branches, i.e. batch normalization branch, instance normalization branch and convolutional pooling branch, which are respectively used to obtain the batch normalization feature, instance normalization feature and convolutional pooling feature of the input feature. The attention feature fusion layer is used to calculate the similarity of the batch normalization feature and the instance normalization feature and generate dynamic weights, and the batch normalization feature and the instance normalization feature are weighted and fused to output the attention batch instance normalized feature.

[0013] Further, the step S3 comprises: S3.1, using the base class audio samples of the source domain, pre-training the residual convolutional neural network by using a supervised training method, the process is as follows: S3.1.1, randomly sampling a batch of training samples from the base class audio samples of the source domain , wherein represents the log-mel spectrum of the audio sample in the training sample, represents the category label; S3.1.2, the domain generalization representation extractor obtains the domain generalization representation. The self-attention normalization layer in the domain generalization representation extractor first performs batch normalization, instance normalization and convolution pooling on the features output by the convolution layer in parallel to obtain batch normalization features, instance normalization features and convolution pooling features, respectively, then calculates the attention weights of the batch normalization features and the instance normalization features, and obtains the domain generalization features through feature weighted fusion.

[0014] S3.1.3, the cosine similarity between the audio sample representations of different categories in the domain generalization representation and the classifier weights is calculated, and the parameters of the residual convolutional neural network are updated iteratively by minimizing the cosine cross-entropy loss; S3.1.4, repeating steps S3.1.1-S3.1.3 until the network converges; S3.2, using the base class audio samples of the source domain, the residual convolutional neural network is trained using the saliency weighted adversarial training method, the process is as follows: S3.2.1, randomly sampling a batch of training samples from the base class audio samples of the source domain , wherein represents the log mel spectrum of the audio sample in the source domain training sample, represents the category label; S3.2.2, fixing the parameters of the residual convolutional neural network, performing spectral perturbation on to obtain the log mel spectrum of the adversarial sample , wherein the spectral perturbation is realized by two identical random convolution operations; S3.2.3, inputting into the residual convolutional neural network to calculate the cosine cross-entropy loss of the adversarial sample, and calculating the saliency weight of by back propagation , updating iteratively by saliency weight modulation of gradient amplitude to minimize the adversarial loss, and taking the updated after multiple rounds as the log mel spectrum feature of the pseudo target domain sample; S3.2.4, activating the parameters of the residual convolutional neural network, concatenating and and inputting them into the domain generalization representation extractor to obtain the training sample representation, calculating the cosine similarity between the representations of audio samples of different categories and the weights of the classifier fully connected layer, and updating the parameters of the residual convolutional neural network iteratively by minimizing the cosine cross-entropy loss; S3.2.5, repeating steps S3.2.1-S3.2.4 until the network converges.

[0015] ​Furthermore, the calculation formula for instance normalization in step S3.1.2 is as follows:

[0016]

[0017]

[0018] set up It is the feature of a batch of training samples as input, where B, C, H, and W represent the batch size, number of channels, height, and width of the feature map, respectively. express The element in the h-th row and w-th column of the c-th channel of the b-th sample feature. It is a very small non-zero constant used to prevent the variance from being zero. and These represent the mean and variance of a single sample in the spatial dimension, respectively. Instance normalization normalizes only the spatial dimension of a single sample, which can improve the domain generalization of the representation but reduces its discriminativeness. Batch normalization relies on the statistical measures of a batch of samples to extract discriminative representations, but its domain generalization is weak.

[0019] The formula for batch normalization is as follows:

[0020]

[0021]

[0022] The formula for calculating attention weights is:

[0023] In the formula, , , These are the instance-normalized features, batch-normalized features, and convolutional pooling features of the input, respectively, where D is the dimension of the input vector. Scaling factor This represents the matrix transpose operation. It is a trainable linear transformation matrix. These represent the weights of the batch-normalized features and the instance-normalized features, respectively. and represents the mean and variance of all samples in a batch in channel c, respectively; the attention mechanism generates weights by calculating the similarity between instance normalized features and batch normalized features and combining them with convolutional pooling features. It can dynamically generate adaptive weights based on the input features of different domains, which helps the model adapt to the distribution differences of different input domains.

[0024] The calculation for feature-weighted fusion is as follows:

[0025] In the formula This represents the normalized features of the attention batch instances. This represents the Hadamard product. The weighted fusion operation, by adaptively merging instance-normalized features and batch-normalized features, helps improve the domain generalization and class discriminativeness of the representation.

[0026] Furthermore, the cosine cross-entropy loss function used for training the network in steps S3.1 and S3.2 is defined as follows:

[0027] In the formula This represents the parameters of the residual convolutional neural network. These represent the fully connected layer parameters of the domain generalization representation extractor and the classifier, respectively. Let represent the log-Mel spectrum and class label of the i-th audio training sample, respectively; , and Representing temperature coefficient, The weights of the fully connected layer of the classifier corresponding to each class The representation corresponding to the sample; This represents the cosine distance between the training sample representation and the weights of the fully connected layer of the classifier.

[0028] Furthermore, the significance weight in step S3.2.3 The calculation formula is as follows:

[0029] in, The update formula is as follows:

[0030] In the formula, Indicates the learning rate; Indicates the adjustable coefficient; Indicates to The gradient; " indicates an assignment operation; Representing adversarial loss and defining it as follows:

[0031] In the formula Let the log-Mel spectrum and class label of the i-th source domain audio training sample be represented respectively. The log-Mel spectrum of the pseudo-target domain samples is represented. Represents semantic constraint coefficients. The Wasserstein distance of the source domain sample and the pseudo target domain sample in the representation space is represented, and is defined as follows:

[0032] In the formula, And Respectively represent the representation of the source domain sample and the adversarial sample, The square of the L2 norm is represented. The adversarial loss is used to generate a semantically consistent adversarial sample, and the diversity of the adversarial sample can be improved by significantly modulating the disturbance amplitude of the adversarial sample. The generated adversarial sample is used as a pseudo target domain sample and a source domain sample to jointly train the model, which helps to learn a domain generalization representation.

[0033] Further, the mean vector of the representation of all samples of the same category in the step S4 is calculated according to the following formula:

[0034] In the formula, The representation of the mean vector of the category The logarithmic mel spectrum of the i-th training sample is represented, i=1,2,..,K, and K represents the number of samples of the category The representation of the i-th training sample is represented.

[0035] Further, the incremental loss function of the m-th incremental link in the step S5 is defined as follows:

[0036] In the formula, The adjustable coefficient is represented; The cosine cross-entropy loss of the mean vector of the training data and the old class representation in the m-th incremental link is represented.

[0037] Further, the audio classification process to be measured in the step S6 is as follows: S6.1, extract the logarithmic mel spectrum features of the audio sample to be measured; S6.2, input the logarithmic mel spectrum features of the audio sample to be measured into the trained domain generalization representation extractor to obtain the domain generalization representation; S6.3, calculate the cosine similarity between the domain generalization representation and the weight of each category in the classifier, and the category corresponding to the weight with the largest similarity is the category to which the audio sample to be measured belongs, and the audio classification result is obtained.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects: (1) The present application proposes a cross-domain small sample class incremental audio classification method based on domain generalization representation, which can solve the domain offset problem of base class and incremental class in small sample class incremental audio classification. ​​

[0039] (2) The application replaces the batch normalization layer in the extractor with a self-attention normalization layer to obtain a domain generalization representation extractor, and improves the discriminability and domain generalization of the representation by adaptively fusing batch normalization features and instance normalization features.

[0040] (3) The application organically combines the saliency weighted adversarial training with the domain generalization representation extractor, improves the diversity of the training samples by generating pseudo target domain samples, and further improves the domain generalization of the representation.

[0041] (4) In the incremental link, the parameters of the domain generalization representation extractor are fixed, the representation of the incremental class and the mean vector of the old class representation are used to update the classifier, and the forgetting of the model to the old class and the overfitting to the incremental class are alleviated.

[0042] (5) The experimental results show that the method of the application obtains higher average precision and lower forgetting degree on the LS2NS and NS2LS data sets, and effectively solves the domain shift problem of the base class and the incremental class in the small sample class incremental audio classification. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 is a flow chart of a cross-domain small sample class incremental audio classification method based on domain generalization representation disclosed in an embodiment of the present application; Figure 2 is a residual convolutional neural network structure schematic diagram in an embodiment of the present application; Figure 3 is a self-attention normalization layer structure schematic diagram in an embodiment of the present application; Figure 4 is a saliency weighted adversarial training process schematic diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] Reference to an "embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. As will be apparent to those of ordinary skill in the art, embodiments described in this application can be combined with other embodiments in various ways.

[0047] Embodiment 1 Figure 1 A flowchart of an embodiment of a cross-domain small sample class incremental audio classification method based on domain generalization representation. As shown in Figure 1 The cross-domain small sample class incremental audio classification method based on domain generalization representation includes the following steps: S1, extracting log-mel spectrum features from input audio samples, wherein the input audio samples are divided into mutually exclusive basic link audio categories (referred to as base classes) and a plurality of incremental link audio categories (referred to as incremental classes). Wherein the base classes come from the source domain, and the incremental classes come from the target domain; In this embodiment, the extraction of log-mel spectrum in step S1 includes the following steps: S1.1, pre-emphasizing the audio sample using a first-order high-pass filter, and the transfer function of the filter is wherein represents the coefficient of the high-pass filter, and the value is 0.95; S1.2, frame the pre-emphasized audio sample into short-time segments, and obtain the windowed short-time audio frame by weighting with a window function; S1.3, performing discrete Fourier transform on the windowed short-time audio frame to generate the corresponding linear spectrum; S1.4, converting the linear spectrum to a mel spectrum using a mel filter, and taking the logarithm of the mel spectrum to obtain the log-mel spectrum.

[0048] In this embodiment, NSynth-100 and LS-100 are used to construct cross-domain audio data sets LS2NS and NS2LS, wherein NSynth-100 contains 100 kinds of musical instrument sounds, and LS-100 contains 100 kinds of speaker sounds.

[0049] The download links of the two data sets are https: / / www.modelscope.cn / datasets / pp199124903 / NSynth-100 / summary and https: / / www.modelscope.cn / datasets / pp199124903 / LS-100 / summary respectively.

[0050] Tables 1 and 2 provide detailed information on the LS2NS and NS2LS cross-domain datasets, respectively.

[0051] As shown in Tables 1 and 2, both the LS2NS and NS2LS datasets are divided into source domain base class dataset D0 and target domain incremental class dataset D0, which are class-disjoint. m (1 ≤ m ≤ (M-1)). D0 contains the basic training dataset. and basic validation dataset D m Includes incremental training dataset and incremental validation dataset . Used for training residual convolutional neural networks Used to update the weights of the classifier. In LS2NS, the source domain base class dataset D0 comes from LS-100, and the target domain incremental class dataset D... m From NSynth-100. In NS2LS, the source domain base class dataset D0 comes from NSynth-100, and the target domain incremental class dataset D... m From LS-100.

[0052] Table 1. Detailed information about the LS2NS dataset

[0053] Table 2. Detailed information about the NS2LS dataset

[0054] S2. Initialize the residual convolutional neural network, which includes a domain generalization representation extractor and a classifier. In this embodiment, the parameters of the residual convolutional neural network are randomly initialized. The domain generalization representation extractor is the backbone of ResNet18. The batch normalization layer after the convolutional layer in the backbone is replaced with a self-attention normalization layer. The classifier is initialized as follows: A 512-dimensional random vector, where The number of categories of basic components. Figure 2 and Figure 3 These are schematic diagrams of the residual convolutional neural network and the self-attention normalization layer structure, respectively, according to embodiments of the present invention.

[0055] like Figure 2 As shown, the structure of a residual convolutional neural network includes a domain generalization representation extractor and a classifier. The domain generalization representation extractor consists of one sequentially connected input convolutional layer, four residual layers, and one average pooling layer.

[0056] The input convolutional layer includes sequentially connected 1 64-channel 3*3 convolutional kernel, 1 self-attention normalization layer, 1 ReLU activation layer and one maximum pooling layer.

[0057] The 4 residual layers have independent and different parameters, and each layer is composed of two residual blocks with the same structure. Each residual block includes two convolutional layers, and the self-attention normalization layer and the ReLU activation layer are sequentially connected after each convolutional layer, and the feature fusion is realized by the bit-by-bit addition operation.

[0058] The average pooling layer is used to perform the average pooling operation.

[0059] As shown in Figure 3 , the self-attention normalization layer includes 1 feature transformation module and 1 attention feature fusion module, wherein the structure of the feature transformation module includes three parallel branches, namely a batch normalization branch, an instance normalization branch and a convolutional pooling branch, which are respectively used to obtain the batch normalization feature Q, the instance normalization feature K and the convolutional pooling feature V of the input feature. The attention feature fusion layer is used to calculate the similarity of the batch normalization feature Q and the instance normalization feature K and generate dynamic weights W BN and W IN , the batch normalization feature Q and the instance normalization feature K are weighted and fused through the weights, and the attention batch instance normalized feature is output.

[0060] S3, in the basic link, the supervised training method and the saliency weighted adversarial training method are used in sequence to train the residual convolutional neural network, and the parameters of the domain generalization feature extractor and the classifier are optimized until convergence, and the parameters of the domain generalization feature extractor are fixed after the training is completed; In this embodiment, the training steps of the basic link are as follows: S3.1, using the base class audio samples of the source domain, pre-training the residual convolutional neural network by using the supervised training method and the cosine cross-entropy loss function, wherein the training batch size is 128; S3.1.1, randomly sampling a batch of training samples from the base class audio samples of the source domain , wherein represents the log mel spectrum of the audio sample in the training sample, represents the category label; S3.1.2, inputting into the domain generalization feature extractor to obtain the domain generalization feature. Wherein, the self-attention normalization layer in the feature extractor first performs batch normalization, instance normalization and convolutional pooling on the features output by the convolutional layer to obtain batch normalization feature Q, instance normalization feature K and convolutional pooling feature V in parallel, and then calculates the attention weight W BN and W INFinally, the weights are used to perform a weighted fusion of the batch normalized features and the instance normalized features to obtain the domain generalization features. .

[0061] The formula for instance normalization is as follows:

[0062]

[0063]

[0064] set up It is the feature of a batch of training samples as input, where B, C, H, and W represent the batch size, number of channels, height, and width of the feature map, respectively. express The element in the h-th row and w-th column of the c-th channel of the b-th sample feature. It is a very small non-zero constant used to prevent the variance from being zero. and These represent the mean and variance of a single sample in the spatial dimension, respectively.

[0065] The formula for batch normalization is as follows:

[0066]

[0067] .

[0068] The formula for calculating attention weights is:

[0069] In the formula, , , These are the instance-normalized features, batch-normalized features, and convolutional pooling features of the input, respectively, where D is the dimension of the input vector. Scaling factor This represents the matrix transpose operation. It is a trainable linear transformation matrix. These represent the weights of the batch-normalized features and the instance-normalized features, respectively. and Let represent the mean and variance of all samples in a batch on channel c, respectively.

[0070] The calculation for feature-weighted fusion is as follows:

[0071] In the formula denotes the normalized feature of the attention batch instance, denotes the Hadamard product.

[0072] S3.1.3, calculate the cosine similarity between the audio sample representations of different classes in the domain generalization representation and the classifier weights, and update the parameters of the residual convolutional neural network by minimizing the cosine cross-entropy loss iteratively; The cosine cross-entropy loss function for training the network is defined as follows:

[0073] wherein denotes the parameters of the residual convolutional neural network, denotes the parameters of the domain generalization representation extractor and the classifier, respectively. denote the log mel-spectrogram and the class label of the i-th audio training sample, respectively; , and denote the temperature coefficient, the classifier weight corresponding to the class, and the representation corresponding to the sample, respectively. denotes the cosine distance between the training sample representation and the classifier weight.

[0074] S3.1.4, repeat steps S3.1.1-S3.1.3 until the network converges.

[0075] S3.2, using the base class audio samples of the source domain, training the residual convolutional neural network using a saliency weighted adversarial training method, wherein the training batch size is 128; S3.2.1, randomly sampling a batch of training samples from the base class audio samples of the source domain wherein denotes the log mel-spectrogram of the source domain training sample, denotes the class label; S3.2.2, fixing the parameters of the residual convolutional neural network, performing spectral perturbation on the input to obtain the log mel-spectrogram of the adversarial sample ; wherein the spectral perturbation is realized by two identical random convolution operations. At each iteration, a value is randomly selected from [1, 3, 5, 7, 11] as the convolution kernel size, and then the parameters of the random convolution layer are initialized with a normal distribution. S3.2.3, inputting

[0076] into the residual convolutional neural network, calculating the cosine cross-entropy loss of the adversarial sample, calculating the saliency weight of by backpropagation , and updating by saliency weight modulation of the gradient amplitude iteratively The updated formula of is as follows: The log-mel-spectrogram feature of the pseudo target domain sample; Wherein, The calculation formula of is as follows:

[0077] Wherein, The updated formula of is as follows:

[0078] In the formula, Indicates the learning rate; Indicates the adjustable coefficient; Indicates the gradient of to Indicates the assignment operation; Indicates the adversarial loss and is defined as follows: In the formula

[0079] Indicates the log-mel-spectrogram and the class label of the i th source domain audio training sample respectively; Indicates the log-mel-spectrogram of the pseudo target domain sample; Indicates the semantic constraint coefficient; Indicates the Wasserstein distance of the source domain sample and the pseudo target domain sample in the feature space, and is defined as follows: In the formula

[0080] And Indicate the features of the source domain sample and the adversarial sample respectively, Indicates the square of the L2 norm.

[0081] S3.2.4, the parameters of the activated residual convolutional neural network are spliced and input into the domain generalization feature extractor to obtain the training sample feature, the cosine similarity between the features of audio samples of different categories and the weights of the full connection layer of the classifier is calculated, and the parameters of the residual convolutional neural network are updated iteratively by minimizing the cosine cross-entropy loss; S3.2.5, repeat steps S3.2.1-S3.2.4 until the network converges.

[0082] S4, input the log-mel-spectrogram feature of the base class audio sample into the domain generalization feature extractor to obtain the base class feature, and then calculate and save the mean vector of the same class feature; In the formula

[0083] In the formula​​​ representing a class a mean vector of the representation, representing a log mel-spectrogram of the i-th training sample, i = 1, 2,.., K, K representing a number of samples of the class representing a number of samples of the class representing a representation of the i-th training sample; S5, in the incremental link, input the log mel-spectrogram features of the incremental class audio sample into the domain generalization representation extractor to obtain the incremental class representation, use the incremental class representation and the mean vector of the old class representation saved in the previous incremental link and the incremental loss function to train the residual convolutional neural network, and optimize the parameters of the classifier until convergence.

[0084] wherein the incremental loss function is defined as follows:

[0085] wherein is set to 0.6; respectively represent the cosine cross-entropy loss of the training data and the mean vector of the old class representation in the m-th incremental link.

[0086] S6, extract the log mel-spectrogram features of the test audio sample, input it into the trained residual convolutional neural network, obtain the representation through the domain generalization representation extractor, calculate the cosine similarity between the representation and the weight of each class in the classifier, and the class corresponding to the weight with the highest similarity is the class to which the test audio sample belongs, thereby obtaining the audio classification result.

[0087] In the embodiment, the specific steps of testing audio classification are as follows: S6.1, extract the log mel-spectrogram features of the test audio sample; S6.2, input the log mel-spectrogram features of the test audio sample into the trained domain generalization representation extractor to obtain the domain generalization representation; S6.3, calculate the cosine similarity between the domain generalization representation and the weight of each class in the classifier, and the class corresponding to the weight with the highest similarity is the class to which the test audio sample belongs, thereby obtaining the audio classification result.

[0088] The embodiment is compared with the PCR method (Y. Tan, and X. Xiang, “Cross-domain few-shot incremental learning for point-cloud recognition,” in Proc. of IEEE / CVF Computer Vision, 2024, pp. 2296-2305.) on two different cross-domain audio datasets LS2NS and NS2LS. The PCR adopts a double-branch structure of a base branch and a domain-specific branch. The method first discriminates whether the test sample belongs to the base class or the new class, and then classifies it using the corresponding branch. The effectiveness of the method is proved by calculating the average accuracy of the model on all classes and the forgetting degree in multiple incremental links.

[0089] wherein the classification accuracy of the mth link is defined as follows: , wherein the number of samples of all classes classified correctly is denoted as N, and H represents the total number of samples of all classes.

[0090] wherein the average accuracy AA of the model on all classes is defined as follows: , wherein M represents the number of links.

[0091] wherein the forgetting degree PD of the model on all classes is defined as follows: , wherein M represents the number of links, the classification accuracy of the mth link, denotes the classification accuracy of the last link.

[0092] As shown in Tables 3 and 4, the method of the present application obtains an average accuracy of 80.13% and 79.25% on the LS2NS and NS2LS datasets, respectively, which are higher than the corresponding values of the PCR method. The forgetting degree of the method of the present application on the LS2NS and NS2LS datasets is 21.49% and 37.47%, respectively, which are lower than the corresponding values of the PCR method. The main reason why the method of the present application is superior to the PCR method is that in the base class link, the method of the present application adopts the self-attention normalization and saliency weighting adversarial training method to improve the discriminability and domain generalization of the representation; in the incremental link, the method of the present application fixes the parameters of the representation extractor, and updates the classifier using the mean vector of the old class representation and the new class representation, which alleviates the overfitting of the model to the new class and the forgetting of the old class. In summary, the method proposed in the present application can effectively improve the cross-domain classification performance of the model while avoiding overfitting to the new class and forgetting the old class.

[0093] ​Table 3. Performance comparison results table on LS2NS dataset

[0094] Table 4. Performance comparison results table on NS2LS dataset

[0095] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not contradict, they should be considered as within the scope of the present disclosure.

[0096] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods and should be within the scope of the present application.

Claims

1. A cross-domain few-sample incremental audio classification method based on domain generalization representation, characterized in that, The audio classification method includes the following steps: S1. Extract log-Mel spectrum features from the input audio samples, wherein the input audio samples are divided into mutually exclusive base segment audio categories and incremental segment audio categories, wherein the base segment audio categories are referred to as base classes and are derived from the source domain, and the incremental segment audio categories are referred to as incremental classes and are derived from the target domain; S2. Initialize the residual convolutional neural network, which includes a sequentially connected domain generalization representation extractor and a classifier. S3. In the basic stage, the residual convolutional neural network is trained by supervised training method and saliency-weighted adversarial training method. The parameters of the domain generalization representation extractor and classifier are optimized until convergence. After training, the parameters of the domain generalization representation extractor are fixed. S4. Input the log-Mel spectrum features of the base class audio samples into the domain generalization representation extractor to obtain the base class representation, and then calculate and save the mean vector of the same class representation for updating the classifier in the incremental step. S5. In the incremental stage, the log-Mel spectrum features of the incremental audio samples are input into the domain generalization representation extractor to obtain the incremental representation. The mean vector of the incremental representation and the base class representation saved in the previous incremental stage, as well as the incremental loss function, are used to train the residual convolutional neural network. The parameters of the classifier are optimized until convergence. S6. Extract the log-Mel spectrum features of the audio sample to be tested and input them into the domain generalization representation extractor to obtain the representation of the audio sample to be tested. Calculate the cosine similarity between the representation and the weights of each category in the classifier. The category corresponding to the weight with the highest similarity is the category to which the audio sample to be tested belongs, and the audio classification result is obtained.

2. The cross-domain few-sample incremental audio classification method based on domain generalization representation according to claim 1, characterized in that, The extraction process of the log-Mel spectrum features in step S1 is as follows: S1.

1. A first-order high-pass filter is used to pre-emphasize the audio samples. The transfer function of the filter is: ,in These represent the coefficients of the high-pass filter, with values ​​ranging from 0 to 1. S1.

2. Divide the pre-emphasized audio samples into short-time segments, and obtain windowed short-time audio frames by weighting them using a window function; S1.

3. Perform a discrete Fourier transform on the windowed short-time audio frame to generate the corresponding linear spectrum; S1.

4. A Mel filter is used to convert the linear spectrum into a Mel spectrum, and the logarithm of the Mel spectrum is taken to obtain the logarithmic Mel spectrum.

3. The cross-domain small-sample incremental audio classification method based on domain generalization representation according to claim 1, characterized in that, The structure of the domain generalization representation extractor includes one input convolutional layer, four residual layers, and one average pooling layer connected in sequence. Each residual layer consists of two residual blocks with identical structures. Each residual block contains two convolutional layers. Each convolutional layer is followed by a self-attention normalization layer and a ReLU activation layer in sequence, and feature fusion is achieved through bitwise addition. The structure of the classifier includes one fully connected layer and one softmax layer.

4. The cross-domain small sample incremental audio classification method based on domain generalization representation according to claim 1, characterized in that, Step S3 includes: S3.1 Using base class audio samples from the source domain, a supervised training method is used to pre-train a residual convolutional neural network, as follows: S3.1.1 Randomly sample a batch of training samples from the base class audio samples in the source domain. ,in This represents the log-Mel spectrum of the audio samples in the training samples. Indicates category label; S3.1.2, will The input domain generalization representation extractor yields the domain generalization representation; Among them, the self-attention normalization layer in the domain generalization representation extractor first performs batch normalization, instance normalization and convolutional pooling in parallel on the features output by the convolutional layer to obtain batch normalized features, instance normalized features and convolutional pooling features, respectively. Then, the attention weights of the batch normalized features and instance normalized features are calculated, and the domain generalization features are obtained by feature weighted fusion. S3.1.3 Calculate the cosine similarity between the audio sample representations of different categories and the classifier weights in the generalization representation of the computation domain, and iteratively update the parameters of the residual convolutional neural network by minimizing the cosine cross-entropy loss; S3.1.4 Repeat steps S3.1.1-S3.1.3 until the network converges; S3.2 Using base class audio samples from the source domain, train a residual convolutional neural network using a saliency-weighted adversarial training method, as follows: S3.2.1 Randomly sample a batch of training samples from the base class audio samples in the source domain. ,in This represents the log-Mel spectrum of audio samples in the source domain training samples. Indicates category label; S3.2.2, Fix the parameters of the residual convolutional neural network, for Log-Mel spectra of adversarial examples are obtained by performing spectral perturbation. In this case, the spectral perturbation is achieved by two identical random convolution operations; S3.2.3, will Input the residual convolutional neural network, calculate the cosine cross-entropy loss of adversarial examples, and then calculate the loss through backpropagation. Significance weight Iterative updates of gradient magnitude are achieved through saliency-weighted modulation. To minimize adversarial losses, multiple rounds of updates Log-Mel spectrum features as samples of the pseudo-target domain; S3.2.4, Activate the parameters of the residual convolutional neural network, and The training sample representations are concatenated and input into the domain generalization representation extractor. The cosine similarity between the representations of different categories of audio samples and the weights of the fully connected layer of the classifier is calculated. The parameters of the residual convolutional neural network are iteratively updated by minimizing the cosine cross-entropy loss. S3.2.5 Repeat steps S3.2.1-S3.2.4 until the network converges.

5. The cross-domain small sample incremental audio classification method based on domain generalization representation according to claim 4, characterized in that, The calculation formula for instance normalization in step S3.1.2 is as follows: set up It is the feature of a batch of training samples as input, where B, C, H, and W represent the batch size, number of channels, height, and width of the feature map, respectively. express The element in the h-th row and w-th column of the c-th channel of the b-th sample feature. It is a very small non-zero constant used to prevent the variance from being zero. and These represent the mean and variance of a single sample in the spatial dimension, respectively. The formula for batch normalization is as follows: The formula for calculating attention weights is: In the formula, , , These are the instance-normalized features, batch-normalized features, and convolutional pooling features of the input, respectively, where D is the dimension of the input vector. Scaling factor This represents the matrix transpose operation. It is a trainable linear transformation matrix. These represent the weights of the batch-normalized features and the instance-normalized features, respectively. and Let represent the mean and variance of all samples in channel c within a batch, respectively. The calculation for feature-weighted fusion is as follows: In the formula This represents the normalized features of the attention batch instances. It represents the Hadamardi (or Hadama) stack.

6. The cross-domain few-sample incremental audio classification method based on domain generalization representation according to claim 4, characterized in that, The cosine cross-entropy loss function used to train the network in steps S3.1 and S3.2 is defined as follows: In the formula This represents the parameters of the residual convolutional neural network. These represent the fully connected layer parameters of the domain generalization representation extractor and the classifier, respectively. Let represent the log-Mel spectrum and class label of the i-th audio training sample, respectively; , and Representing temperature coefficient, The weights of the fully connected layer of the classifier corresponding to each class The representation corresponding to the sample; This represents the cosine distance between the training sample representation and the weights of the fully connected layer of the classifier.

7. The cross-domain few-sample incremental audio classification method based on domain generalization representation according to claim 4, characterized in that, The significance weight in step S3.2.3 The calculation formula is as follows: in, The update formula is as follows: In the formula, Indicates the learning rate; Indicates the adjustable coefficient; Indicates to gradient; " indicates an assignment operation; Representing adversarial loss and defining it as follows: In the formula Let the log-Mel spectrum and class label of the i-th source domain audio training sample be represented respectively. The log-Mel spectrum of the pseudo-target domain samples is represented. Represents semantic constraint coefficients. The Wasserstein distance between the source domain sample and the pseudo-target domain sample in the representation space is defined as follows: In the formula and These represent the source domain samples and adversarial samples, respectively. This represents the square of the L2 norm.

8. The cross-domain small sample incremental audio classification method based on domain generalization representation according to claim 1, characterized in that, The formula for calculating the mean vector of all samples of the same category in step S4 is as follows: In the formula Indicates category The representation of the mean vector, Let K represent the log-Mel spectrum of the i-th training sample, where i = 1, 2, ..., K, and K represents the class. The number of samples, This represents the representation of the i-th training sample.

9. The cross-domain small-sample incremental audio classification method based on domain generalization representation according to claim 1, characterized in that, The loss function for the m-th incremental step in step S5 is defined as follows: In the formula Indicates the adjustable coefficient; Let represent the cosine cross-entropy loss of the mean vectors of the training data and the old class representation in the m-th incremental step, respectively.

10. The cross-domain small-sample incremental audio classification method based on domain generalization representation according to claim 1, characterized in that, The audio classification process in step S6 is as follows: S6.1 Extract the log-Mel spectrum features of the audio sample to be tested; S6.2 Input the log-Mel spectrum features of the audio sample to be tested into the trained domain generalization representation extractor to obtain the domain generalization representation. S6.3 Calculate the cosine similarity between the domain generalization representation and the weights of each category in the classifier. The category corresponding to the weight with the highest similarity is the category to which the audio sample to be tested belongs, and the audio classification result is obtained.

Images