Acoustic Scene Classification Method Based on Paired Feature Fusion

Abstract

The present invention belongs to the technical field of pattern recognition and artificial intelligence, and discloses an acoustic scene classification method based on paired feature fusion, including: establishing a paired feature fusion algorithm based on spectrum correction and spectrum subtraction, obtaining the paired features of the sample by calculating the original information of the sample; establishing a lightweight neural network model based on lightweight neural network architecture search; inputting the paired features into the lightweight neural network model to obtain the classification probability of the sample. The present invention proposes a paired feature fusion algorithm, and uses the mean operation to extract the common features in the spectrum graph of each device for the conversion of device samples, which effectively alleviates the problem of mismatch in the number of samples of multiple devices while quickly shortening the training time. By proposing a lightweight attention module, the neural network model can focus on the extraction of global features during training, thereby weakening the influence of noise signals and effectively improving the classification accuracy of difficult samples.

Description

Technical Field

[0001] The present invention relates to the technical field of pattern recognition and artificial intelligence, and in particular to an acoustic scene classification method based on paired feature fusion. Background Art

[0002] In everyday life, when we hear the roar of an airplane engine or the honking of a car horn, we conjure up images of the corresponding aircraft or car. This distinct audio segment is called an "acoustic event" (AE) in the field of sound. This audio segment corresponds to one or more specific pieces of information about an event. The process of inferring and analyzing the scene and surrounding environment surrounding an acoustic event is known as the derivation from "acoustic event" to "acoustic scene" (AS) in the field of acoustics.

[0003] With the advancement of deep learning and the growth of GPU computing power, single-device sound scene classification has achieved promising results. However, in multi-device sound scene classification, due to the varying usage of each device, the samples collected by each device can vary significantly, making it difficult for single-device methods to maintain the same results. Furthermore, many current techniques increase network accuracy by deepening the network structure or designing more complex network models. Excessively deep neural networks place significant demands on storage and computing resources, making them impractical for use on common portable devices. Therefore, designing lightweight models is crucial. Summary of the Invention

[0004] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is: the problem of low classification accuracy and lightweight of multiple devices in the prior art.

[0007] To solve the above technical problems, the present invention provides the following technical solution: an acoustic scene classification method based on paired feature fusion, comprising:

[0008] A paired feature fusion algorithm is established based on spectrum correction and spectrum subtraction, and the paired features of the samples are obtained by calculating the original information of the samples;

[0009] Establish a lightweight neural network model based on lightweight neural network architecture search;

[0010] The paired features are input into the lightweight neural network model to obtain the classification probability of the sample.

[0011] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the feature fusion algorithm includes:

[0012] The original information of the sample is converted into a 1025×431 two-dimensional tensor through fast Fourier transform. The two vectors of the two-dimensional tensor are subtracted to obtain the feature difference of different devices on the same sample after filtering the background noise. The absolute value of the feature difference after subtraction is taken, and the difference is expressed as:

[0013] X m =||f(s a )|-|f(s m )||

[0014] Among them, s a A sample representing the original device, s m represents the sample of the reference device, and the f function is the fast Fourier transform.

[0015] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the feature fusion algorithm includes:

[0016] The sample differences of all reference devices are calculated and averaged to obtain the calibration characteristics between the original device and the reference device. The calibration characteristics are expressed as:

[0017]

[0018] Where N represents the number of reference devices and i represents the i-th sample

[0019] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the feature fusion algorithm further includes:

[0020] The correction feature is subtracted from the sample of the original device to obtain a new sample with the reference device feature. The new sample with the reference device feature is expressed as:

[0021]

[0022] in, It can be considered as a new sample with the characteristics of the reference device;

[0023] The new sample with the reference device characteristics is subjected to a modulo square and logarithm operation to obtain a logarithmic Melton spectrum feature as an input feature of the lightweight neural network model.

[0024] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the neural network model includes:

[0025] The features after paired feature fusion are used as input, and are passed through the multi-channel fusion module and the lightweight attention module. After passing through three lightweight modules, and finally through global average pooling and Softmax, the classification probability of the sample is obtained.

[0026] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the multi-channel fusion module includes:

[0027] A maximum pooling layer, an average pooling layer and two convolutional layers are spliced in the time domain to form the multi-channel fusion module.

[0028] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the multi-channel fusion module further includes:

[0029] The multi-channel fusion module performs 3×3 convolution with a stride of 1, 3×3 convolution with a stride of 2, 2×2 maximum pooling and average pooling on the input features respectively; the input feature size after splicing is 64×1059×32, and after downsampling and dimensionality upscaling by three lightweight modules, the output dimension becomes 8×133×64; the features output from the lightweight attention module are downsampled and upscaled, and element-by-element added to the output of the lightweight module as the input of the next stage; finally, the number of channels is reduced to 10 through 1×1 convolution, and the classification probability of the sample is obtained after global average pooling and Softmax.

[0030] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the lightweight module includes three inverted bottleneck modules;

[0031] The three inverted bottleneck modules are connected in a stacked manner to form a lightweight module. The first bottleneck module of each lightweight module downsamples the input spectrum graph so that the time domain and frequency domain dimensions are halved. The following two bottleneck modules supplement the original sample information of the input through residual connections.

[0032] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, in the lightweight neural network model, the downsampling of the spectrum graph dimension is achieved by depthwise separable convolution; when downsampling, the depthwise separable convolution kernel size is 3, the step size is 2, and the padding is 1; the depthwise separable convolution kernel size in other modules is 3, the step size is 1, and the padding is 1; finally, 1×1 convolution is used for dimensionality reduction so that the number of output channels is consistent with the number of input channels.

[0033] As a preferred solution of the acoustic scene classification method based on paired feature fusion described in the present invention, the lightweight attention module includes:

[0034] The lightweight attention module compresses the frequency domain of each channel to obtain the average frequency domain features, and then obtains the global information of part of the frequency domain to the entire time domain through the calculation of the self-attention mechanism; then expands the frequency domain to the original dimension to obtain the complete global features; finally, the residual connection is used at the tail to supplement part of the input features.

[0035] Beneficial effects of the present invention: The present invention provides a paired fusion-based acoustic scene classification method. By proposing a paired feature fusion algorithm, the common features in the spectrograms of each device are extracted using a mean operation for device sample conversion. This method effectively alleviates the problem of mismatched sample quantities across multiple devices while rapidly shortening training time. By proposing a lightweight attention module, the neural network model can focus on the extraction of global features during training, thereby weakening the influence of noise signals and effectively improving the classification accuracy of difficult samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0037] Figure 1 A process diagram of feature extraction based on a paired fusion acoustic scene classification method provided by one embodiment of the present invention;

[0038] Figure 2 Spectrograms of the same sample on different devices based on the paired fusion acoustic scene classification method provided by one embodiment of the present invention;

[0039] Figure 3 An overall structural diagram of a lightweight neural network based on a paired fusion acoustic scene classification method provided by one embodiment of the present invention;

[0040] Figure 4 An inverted bottleneck structure diagram of a lightweight module based on a paired fusion acoustic scene classification method provided by one embodiment of the present invention;

[0041] Figure 5 A comparison chart of classification results of different methods based on the paired fusion acoustic scene classification method provided in one embodiment of the present invention;

[0042] Figure 6 A diagram showing the classification accuracy at different λ values for a paired fusion acoustic scene classification method provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0043] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0044] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0045] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0046] The present invention is described in detail with reference to schematic diagrams. For ease of illustration, cross-sectional views of device structures may be partially enlarged and not to scale when describing embodiments of the present invention. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0047] In the description of the present invention, it should be noted that the terms "upper, lower, inner, and outer" and other references to orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0048] In this disclosure, unless otherwise specified or limited, the terms "mounted," "connected," and "connected" should be interpreted broadly. For example, they may refer to fixed, removable, or integral connections. They may also refer to mechanical, electrical, or direct connections, indirect connections through an intermediary, or internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in this disclosure.

[0049] Example 1

[0050] Reference Figure 1 —4 is an embodiment of the present invention, which provides an acoustic scene classification method based on paired fusion, including:

[0051] S1: A paired feature fusion algorithm is established based on spectrum correction and spectrum subtraction, and the paired features of the samples are obtained by calculating the original information of the samples;

[0052] Device A, which has the largest number of samples, is designated as the original device, while the other devices are designated as reference devices. The goal is to transform the features of the original device into those of the reference device through a paired feature fusion algorithm. Our method operates after the fast Fourier transform, and the algorithm inputs are paired samples from the reference device and the original device.

[0053] Furthermore, the feature fusion algorithm includes: converting the original information of the sample into a 1025×431 two-dimensional tensor through fast Fourier transform, subtracting the two vectors of the two-dimensional tensor to obtain the feature difference of different devices on the same sample after filtering the background noise; taking the absolute value of the feature difference after subtraction, and expressing the difference as:

[0054] X m =||f(s a )|-|f(s m )||

[0055] Among them, s a represents the sample of device A, s m Represents the samples from device B to S3, where the f function is the fast Fourier transform.

[0056] The sample differences of all reference devices are calculated and averaged to obtain the calibration characteristics between the original device and the reference device. The calibration characteristics are expressed as:

[0057]

[0058] Where N represents the number of reference devices and i represents the i-th sample.

[0059] The correction feature is subtracted from the sample of the original device to obtain a new sample with the feature of the reference device. The new sample with the feature of the reference device is expressed as:

[0060]

[0061] in, It can be considered as a new sample with the characteristics of the reference device;

[0062] The new samples with the reference device characteristics are subjected to the modulo square and logarithm operations to obtain the logarithmic Melton spectrum features, which are used as the input features of the lightweight neural network model.

[0063] Then, in device A, λ number of samples are randomly taken, one for each reference device, and a certain number of new samples with the feature distribution of devices B to S3 are generated by subtracting the correction feature.

[0064] Unlike the operation in spectrum correction where the amplitude spectrum of each sample is first averaged along the time axis and the dimension is first converted to 1025×1, this method does not perform a separate operation on the time axis and retains the information in the time domain as much as possible.

[0065] Visualizing the two-dimensional tensor generated by the Mel-spectrogram calculation reveals that most information is distributed in the lower-dimensional regions of the frequency domain, throughout the time domain, with data values ranging from (0, 800). For data values less than 1, all values are set to 0 to reduce computational effort. For data with larger values, the addition or removal of correction features does not significantly change the values. Figure 2 (c) and (d) are the spectrograms output by using spectrum correction and paired feature fusion, respectively. For some samples with few sound signals in the audio, such as Figure 2 In the first row, the spectrum correction is almost the same as the method in this paper, but for some complex sound samples, such as Figure 2 In the second row, the high-frequency information of the spectrum correction is almost completely lost, while the method proposed in this paper retains the high-frequency part of the audio.

[0066] S2: Establish a lightweight neural network model based on lightweight neural network architecture search;

[0067] Furthermore, the network structure of this paper is as follows Figure 3As shown in the figure, the proposed method consists of a multi-channel fusion module, a lightweight quantization module, and a lightweight attention module. The input is a log-mel spectrogram and its first-order and second-order spectrograms, concatenated in the channel dimension. This is because the first-order spectrogram can well reflect the dynamic characteristics of the audio. The first-order difference mel spectrogram is further differentiated to obtain the second-order difference mel spectrogram, further capturing more dynamic features. The multi-channel fusion module performs a 3×3 convolution with a stride of 1, a 3×3 convolution with a stride of 2, and 2×2 max pooling and average pooling on the input features. This operation extracts and fuses the original features into different dimensions, enhancing the model's ability to perceive the original information and facilitating downstream classification tasks. The concatenated input features have a size of 64×1059×32. After downsampling and dimension upscaling by the three lightweight quantization modules, the output dimensions become 8×133×64. The features output from the lightweight attention module are then downsampled and dimension upscaling, and element-wise added to the output of the lightweight quantization module to serve as the input for the next stage. Finally, the number of channels is reduced to 10 through 1×1 convolution, and the classification probability of the sample is obtained after global average pooling and Softmax.

[0068] Furthermore, the multi-channel fusion module performs 3×3 convolution with a stride of 1, 3×3 convolution with a stride of 2, 2×2 maximum pooling and average pooling on the input features respectively; the input feature size after splicing is 64×1059×32, and after downsampling and dimensionality upscaling by three lightweight modules, the output dimension becomes 8×133×64; the features output from the lightweight attention module are downsampled and upscaled, and then element-by-element added to the output of the lightweight module as the input of the next stage; finally, the number of channels is reduced to 10 through 1×1 convolution, and the classification probability of the sample is obtained after global average pooling and Softmax.

[0069] Furthermore, the lightweight module includes three inverted bottleneck modules; the three inverted bottleneck modules are connected in a stacked manner to form a lightweight module. The first bottleneck module of each lightweight module downsamples the input spectrum map so that the time domain and frequency domain dimensions are halved. The following two bottleneck modules supplement the original sample information of the input through residual connections. The specific structure of each inverted bottleneck module is as follows: Figure 4 shown.

[0070] It should be noted that throughout the entire network structure, downsampling of the spectrogram dimensions is achieved using depthwise separable convolutions. During downsampling, the kernel size is 3, the stride is 2, and the padding is 1. In other modules, the kernel size is 3, the stride is 1, and the padding is 1. Finally, 1×1 convolutions are used for dimensionality reduction, ensuring that the number of output channels matches the number of input channels. Because nonlinear activation functions can significantly affect low-dimensional features, causing them to lose important information, the ReLU activation function is removed and replaced with a linear transformation before the final output. Figure 4 The residual connection in is only valid when the convolution stride is 1 and the input and output dimensions are equal.

[0071] Furthermore, the lightweight attention module compresses the frequency domain of each channel to obtain the average frequency domain features, and then obtains the global information of part of the frequency domain to the entire time domain through the calculation of the self-attention mechanism; then expands the frequency domain to the original dimension to obtain the complete global features; finally, the residual connection is used at the tail to supplement part of the input features.

[0072] S3: Input the paired features into the lightweight neural network model to obtain the classification probability of the sample.

[0073] Furthermore, the features after paired feature fusion are used as input, and at the same time pass through the multi-channel fusion module and the lightweight attention module, and then pass through three lightweight modules, and finally pass through global average pooling and Softmax to obtain the classification probability of the sample.

[0074] Example 2

[0075] Reference Figure 1 —4 is the second embodiment of the present invention, which provides an acoustic scene classification method based on paired feature fusion, and scientifically demonstrates the present invention through specific implementation steps.

[0076] The dataset used in this example is the development dataset from TUT2020 (TAU Urban Acoustic Scenes 2020 Mobile). The data contains 10 types of sound scenes: airports, buses, subways, subway stations, parks, squares, shopping malls, sidewalks, traffic streets, and trams, with a total of 23,040 audio data items over 64 hours, from 12 major European cities: Amsterdam, Barcelona, Helsinki, Lisbon, London, Lyon, Madrid, Milan, Prague, Paris, Stockholm, and Vienna. These data were recorded by 9 devices, of which device A is a ZoomF8 recorder with a 48kHz sampling rate and 24-bit resolution, device B is a Samsung Galaxy S7, and device C is an iPhone SE, all of which are real devices. The remaining S1 to S6 are virtual devices, which are obtained by convolving the recording of device A with a selected Si impulse response and then performing dynamic range compression using a set of selected parameters. The segments of the virtual devices are composed of randomly selected segments of device A, so they all overlap with the data of device A, but they do not necessarily overlap with each other. The audio is provided in a single-channel, 44.1kHz, 24-bit format and is ultimately clipped into 10-second audio segments. 70% of the data is used for training and 30% for testing. The specific distribution of the data across devices is shown in Table 1.

[0077] Table 1 Data distribution of different devices

[0078]

[0079] In order to balance the number of devices, 3855 segments in device A are not used in the training set and test set. S4, S5, and S6 only appear in the test set and are called invisible devices. The task to be solved in this embodiment is the low-complexity sound scene classification of multiple devices. Multiple devices refer to the problem of low classification accuracy caused by large differences in the number of multiple devices. Low complexity mainly limits the size of the model, and the size of the entire model must not exceed 128Kb. The teams generally compare the number of parameters of each model.

[0080] The entire model is trained on Tensorflow 2.0 and Keras 2.3 frameworks. The optimizer uses stochastic gradient descent (SGD), with momentum set to 0.9 and decay set to 10^ (-6), and uses the categorical cross-entropy loss function. The number of iterations for each model training is 256, and the batch size is 32. The initial learning rate is set to 0.1, and the learning rate is reset at the 3rd, 7th, 15th, 31st, 127th, and 255th rounds to obtain the retraining effect. The learning rate decay uses the cosine annealing algorithm. In the model, two data augmentation methods that do not increase the number of samples are also used. The α value of mixup is 0.4, and new features are generated by crossing adjacent data to enrich the data and obtain better robustness. Spec Augment is used in each mini-batch, and the time axis and frequency axis are randomly masked to alleviate overfitting.

[0081] Example 3

[0082] Reference Figure 5 —6 is an embodiment of the present invention, which provides an acoustic scene classification method based on paired feature fusion. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through comparative experiments.

[0083] like Figure 5 The figure shows the correctly classified sample data for each category using different methods. The three different lines represent the classification results for the three networks using paired feature fusion, the baseline, and paired feature fusion plus a lightweight attention module. For the airport category, our algorithm achieved a 9.2% improvement over the baseline, correctly classifying 30 more samples. For the bus, subway, metro station, park, and tram categories, our method outperformed the baseline by 0.9%, 2.7%, 0.3%, 8.4%, and 2.3%, respectively, and outperformed the paired feature fusion algorithm alone by 0.4%, 0.7%, 1.6%, 3.9%, and 1.3%. However, for the shopping mall, sidewalk, and traffic street categories, accuracy decreased somewhat after adding the lightweight attention module.

[0084] For the invisible devices S4, S5, and S6 that only appear in the test set, our method achieves 1.3%, 2.0%, and 4.3% higher accuracy than the baseline, respectively, as shown in Table 2.

[0085] Table 2 Comparison of paired feature fusion and spectral correction on invisible devices

[0086]

[0087] The second row in Table 2 only adds the spectrum correction method. It can be seen that the algorithm proposed in this paper can train a richer feature distribution through the neural network, and can accurately identify devices that do not appear in the training set, thereby enhancing the generalization ability of the model.

[0088] The classification results of the method of this embodiment and other methods are shown in Table 3. In the table, the dark gray shading indicates the highest accuracy in this category, and the light gray shading indicates the second highest classification accuracy. The classification accuracy of the method in the airport and square categories is ahead of other tasks. Among them, in the airport category, the accuracy is 8.4% higher than the second. The difference between the method with the highest accuracy in the subway, subway station, and park categories is not much, especially in the park category, the accuracy is only 0.4% lower. The accuracy is lower in the bus and sidewalk categories, and there is a big gap with other methods. Compared with the method with the highest total classification accuracy, the method in this article has a greater advantage in the number of parameters, and obtains higher accuracy with a smaller number of parameters.

[0089] Table 3 Comparison of paired feature fusion with other methods

[0090]

[0091] This example demonstrates the influence of sample size on the classification method of the present invention by setting the number of conversion samples, specifically:

[0092] In the paired feature fusion algorithm, the number of device A samples converted to other samples is controlled by λ. One is the number of samples of the device to be converted, which is 330 in the TUT2020 dataset. This paper defaults to 1, such as Figure 6 The figure below shows the classification accuracy for different λ values. When λ is 1, meaning the number of converted samples equals the number of samples from the other devices, the highest accuracy is achieved. Furthermore, as the λ value increases, the training time per round also increases. Overall, a λ value of 1 yields the best model performance, achieving an accuracy of 72%.

[0093] This embodiment also demonstrates the effectiveness of each module of the neural network of the present invention by setting up an ablation experiment, specifically:

[0094] Several sets of ablation experiments were conducted on TUT2020. Taking the complete model as the benchmark, the various modules proposed in this article were continuously removed, and the obtained accuracy is shown in Table 4, where Full represents the complete model; PFF represents the paired feature fusion algorithm; SC represents the spectrum correction algorithm; MF represents the multi-channel fusion module; LA represents the lightweight attention module. "+" and "-" respectively indicate the restoration and removal of the module.

[0095] Table 4 Accuracy of ablation experiments of our algorithm

[0096]

[0097] The second and third rows, respectively, show a 1.9% improvement in accuracy when only the paired feature fusion algorithm and the spectral correction algorithm are added. Removing the multi-channel fusion module significantly reduces model accuracy. This is due to the lack of features from different dimensions. The network captures limited information, making it less sensitive to sound signals outside its receptive field. Removing the lightweight attention module also reduces model accuracy. This is because the neural network focuses too much on local information during training, lacking the overall continuity of the sound signal.

[0098] From the above demonstration, it can be seen that the classification method of the present invention has good practicality and high classification accuracy in real application scenarios.

[0099] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An acoustic scene classification method based on paired feature fusion, characterized in that: include: A paired feature fusion algorithm is established based on spectrum correction and spectrum subtraction, and the paired features of the samples are obtained by calculating the original information of the samples; Establish a lightweight neural network model based on lightweight neural network architecture search; Inputting the paired features into the lightweight neural network model to obtain the classification probability of the sample; The feature fusion algorithm includes: The original information of the sample is converted into a 1025×431 two-dimensional tensor through fast Fourier transform. The two vectors of the two-dimensional tensor are subtracted to obtain the feature difference of different devices on the same sample after filtering the background noise. The absolute value of the feature difference after subtraction is taken, and the difference is expressed as: X m =||f(s a )|-|f(s m )|| Among them, s a A sample representing the original device, s m represents the sample of the reference device, and the f function is the fast Fourier transform; The feature fusion algorithm includes: The sample differences of all reference devices are calculated and averaged to obtain the calibration characteristics between the original device and the reference device. The calibration characteristics are expressed as: Where N represents the number of reference devices, and i represents the i-th sample; The feature fusion algorithm further includes: The correction feature is subtracted from the sample of the original device to obtain a new sample with the reference device feature. The new sample with the reference device feature is expressed as: in, It can be considered as a new sample with the characteristics of the reference device; Performing a modulo square and logarithm operation on the new sample having the reference device characteristics to obtain a logarithmic Melton spectrum feature as an input feature of the lightweight neural network model; Neural network models, including: The features after paired feature fusion are used as input, and pass through the multi-channel fusion module and the lightweight attention module, then through three lightweight modules, and finally through global average pooling and Softmax to obtain the classification probability of the sample.

2. The acoustic scene classification method based on paired feature fusion according to claim 1, characterized in that The multi-channel fusion module includes: A maximum pooling layer, an average pooling layer and two convolutional layers are spliced in the time domain to form the multi-channel fusion module.

3. The acoustic scene classification method based on paired feature fusion according to claim 1, characterized in that The multi-channel fusion module includes: The multi-channel fusion module performs 3×3 convolution with a stride of 1, 3×3 convolution with a stride of 2, 2×2 maximum pooling and average pooling on the input features respectively; the input feature size after splicing is 64×1059×32, and after downsampling and dimensionality upscaling by three lightweight modules, the output dimension becomes 8×133×64; the features output from the lightweight attention module are downsampled and upscaled, and element-by-element added to the output of the lightweight module as the input of the next stage; finally, the number of channels is reduced to 10 through 1×1 convolution, and the classification probability of the sample is obtained after global average pooling and Softmax.

4. The acoustic scene classification method based on paired feature fusion according to claim 1, characterized in that The lightweight module includes three inverted bottleneck modules; The three inverted bottleneck modules are connected in a stacked manner to form a lightweight module. The first bottleneck module of each lightweight module downsamples the input spectrum graph so that the time domain and frequency domain dimensions are halved. The following two bottleneck modules supplement the original sample information of the input through residual connections.

5. The acoustic scene classification method based on paired feature fusion according to claim 1 or 4, characterized in that: In the lightweight neural network model, downsampling of the spectrum graph dimension is achieved by depthwise separable convolution; when downsampling, the depthwise separable convolution kernel size is 3, the stride is 2, and the padding is 1; in other modules, the depthwise separable convolution kernel size is 3, the stride is 1, and the padding is 1; finally, 1×1 convolution is used for dimensionality reduction so that the number of output channels is consistent with the number of input channels.

6. The acoustic scene classification method based on paired feature fusion according to claim 1, characterized in that The lightweight attention module includes: The lightweight attention module compresses the frequency domain of each channel to obtain the average frequency domain features, and then obtains the global information of part of the frequency domain to the entire time domain through the calculation of the self-attention mechanism; then expands the frequency domain to the original dimension to obtain the complete global features; finally, the residual connection is used at the tail to supplement part of the input features.

Images