Speaker audio classification method under multiple heterogeneous devices

Abstract

The embodiment of the present disclosure provides a loudspeaker audio classification method under a plurality of heterogeneous devices, and belongs to the technical field of data identification, and specifically comprises the following steps: step 1, calling a built-in acceleration sensor of a mobile phone to collect an initial vibration signal played by a same on-board loudspeaker; step 2, performing a pretreatment operation on the initial vibration signal; step 3, removing interference signals in the pretreated initial vibration signal to obtain a target vibration signal; step 4, screening signal statistical features highly related to audio content and independent of devices; and step 5, classifying the audio content of the target vibration signal by using an adversarial neural network according to the signal statistical features. Through the scheme of the present disclosure, the efficiency and accuracy of speech recognition are improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of data recognition, and in particular, to a loudspeaker audio classification method under multi-heterogeneous devices. BACKGROUND

[0002] Currently, modern smartphones equipped with various embedded sensors (such as loudspeakers, microphones, and accelerometers) to achieve a wide range of applications have become an indispensable part of people's daily life. Among the sensors of smartphones, the loudspeaker is an important sensor for telephone, voice message, and video playback. The audio information played by the loudspeaker can involve sensitive and personal information of the user, such as user identity, financial information, password, and preference planning information, etc. Due to its importance, the system permission of the microphone (i.e., the audio recording component) is set to the highest level in commercial smartphone operating systems (such as Android and iOS), which greatly reduces the voice privacy leakage of malicious applications.

[0003] However, there is empirical evidence revealing a key limitation caused by device diversity, and existing work in this field is suffering from the following problem: the loudspeaker-induced vibration signals (referred to as LIV) reaching the built-in accelerometer carry a large amount of information highly related to the physical structure of the device, which was previously suspected by recent research. Therefore, this inherent device diversity can seriously hinder the universality of LIV-based audio content attacks, which involve a large number of heterogeneous devices. Due to the involvement of a large number of heterogeneous devices, the new attack scenario brings unprecedented device diversity problems. Therefore, this will make the existing methods ineffective, because the efforts to analyze and train LIV-based audio signals on some devices may not be applicable to LIV signals carrying the same audio information but collected by another device that is not exactly the same as these training devices.

[0004] It can be seen that there is an urgent need for an efficient and accurate loudspeaker audio classification method under multi-heterogeneous devices. SUMMARY

[0005] In view of this, embodiments of the present disclosure provide a loudspeaker audio classification method under multi-heterogeneous devices, which at least partially solves the problem of poor efficiency and accuracy of audio classification in the prior art.

[0006] In a first aspect, embodiments of the present disclosure provide a loudspeaker audio classification method under multi-heterogeneous devices, comprising:

[0007] Step 1, calling a built-in acceleration sensor of a mobile phone to collect an initial vibration signal played by a same on-board loudspeaker;

[0008] Step 2, performing a preprocessing operation on the initial vibration signal;

[0009] Step 3, removing interference signals in the pre-processed initial vibration signal to obtain a target vibration signal;

[0010] Step 4, screening signal statistical features highly related to audio content and independent of devices;

[0011] Step 5, classifying audio content of the target vibration signal using an adversarial neural network according to the signal statistical features.

[0012] According to a specific implementation manner of the embodiment of the present disclosure, the step 2 specifically comprises:

[0013] interpolating the initial vibration signal;

[0014] filtering the initial vibration signal using a Kalman filter with a preset cutoff frequency;

[0015] segmenting the initial vibration signal based on amplitude thresholds of signal time domain and frequency domain signals using coarse-grained and fine-grained segmentation manners;

[0016] normalizing the initial vibration signal using a preset formula.

[0017] According to a specific implementation manner of the embodiment of the present disclosure, the preset formula is wherein μ is an average value of original signal amplitudes, max(abs(Signal)) is a maximum value of absolute values of original signal amplitudes, Signal(i) is an i-th amplitude of the original signal, and Signal(i) Normalization is an i-th amplitude of the original signal after normalization.

[0018] According to a specific implementation manner of the embodiment of the present disclosure, the step 3 specifically comprises:

[0019] Step 3.1, calculating signal features in each time window of the pre-processed initial vibration signal and putting the signal features into a one-class support vector machine classifier to obtain an output normalization score, wherein the signal features include Sharpness, Smoothness, SpecKurt and SpecCrest;

[0020] Step 3.2, regarding a signal corresponding to a normalization score lower than a threshold value as a target vibration signal.

[0021] According to a specific implementation manner of the embodiment of the present disclosure, the step 4 specifically comprises:

[0022] screening key features highly related to audio content from a LibXtract feature library using a random forest method;

[0023] Based on the screened key features, a preset number of features with small device influence are selected as signal statistical features by correlation analysis.

[0024] According to a specific implementation of an embodiment of the present disclosure, the adversarial neural network comprises a feature extractor, a word recognizer and a device discriminator, wherein the feature extractor comprises a feedforward neural network and a convolutional neural network.

[0025] According to a specific implementation of an embodiment of the present disclosure, the step 5 specifically comprises:

[0026] The feature extractor inputs the signal statistical features and the frequency spectrum of the target vibration signal into the feedforward neural network and the convolutional neural network for compression to obtain one-dimensional features and connect them in the same feature vector;

[0027] The multiple feature vectors are input into the word recognizer to output the confidence of each audio content, and when the value of the confidence is greater than an empirical threshold, the content of the audio signal is determined as the category with the greatest confidence to form a word label;

[0028] The output of the feature extractor and the word recognizer is connected into the output device label vector by the device discriminator;

[0029] The min-max game between the feature extractor, the word recognizer and the device discriminator is iteratively updated by using the total loss function to obtain the classification result corresponding to the audio content of the target vibration signal.

[0030] The loudspeaker audio classification scheme under multiple heterogeneous devices in the embodiment of the present disclosure comprises: step 1, calling a built-in acceleration sensor of a mobile phone to collect an initial vibration signal generated by a same on-board loudspeaker; step 2, performing a preprocessing operation on the initial vibration signal; step 3, removing interference signals in the preprocessed initial vibration signal to obtain a target vibration signal; step 4, screening signal statistical features highly related to audio content and independent of devices; and step 5, classifying the audio content of the target vibration signal using an adversarial neural network according to the signal statistical features.

[0031] The embodiment of the present disclosure has the following beneficial effects: through the scheme of the present disclosure, the built-in acceleration sensor is called to collect the vibration signal generated by the loudspeaker playing, and the vibration signal is sent to a remote server; then the vibration signal is preprocessed by using the interpolation, filtering, segmentation and normalization methods; the single-class support vector machine method is used to identify the target vibration signal from the candidate signals, thereby reducing the system energy consumption; a key feature set highly related to the audio content and less affected by the device diversity is screened from numerous statistical features; finally, the adversarial neural network in the multi-task learning mode is used to solve the limitation caused by the device diversity and realize high-precision speech recognition. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only represent some of the embodiments of the present disclosure, and all other drawings obtained by those of ordinary skill in the art based on the drawings without creative labor should also fall within the scope of the present disclosure.

[0033] Figure 1 A flowchart of a loudspeaker audio classification method under a plurality of heterogeneous devices provided by the embodiments of the present disclosure;

[0034] Figure 2 A signal segmentation diagram provided by the embodiments of the present disclosure;

[0035] Figure 3 A time-frequency distribution diagram of collecting different types of high-frequency signals provided by the embodiments of the present disclosure, wherein (a) is a ringing vibration, (b) is a sliding signal, and (c) is a target vibration signal;

[0036] Figure 4 Importance score of a single feature in an audio signal classification provided by the embodiments of the present disclosure;

[0037] Figure 5 An experimental setup diagram for screening out device-independent features provided by the embodiments of the present disclosure;

[0038] Figure 6 An adversarial neural network structure diagram provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0040] The embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0041] It is important to note that the various aspects described herein are illustrative only and are provided to impart additional understanding of the aspects described herein. Accordingly, those of ordinary skill in the art will readily recognize a variety of ways to implement the aspects described herein without departing from the scope of the aspects. Additionally, well-known methods, procedures, components and circuits have not been described in detail since it would be appreciated by one of ordinary skill in the art that such description would be unnecessary and would only unnecessarily obfuscate the application. While aspects described herein are described in the general context of methods and systems, it should be understood that these aspects can also be implemented in combination with other aspects unless specifically stated otherwise.

[0042] It is also important to note that the concertinaed drawings included in the following detailed description are included for illustrative purposes only and are not intended to limit the scope of the disclosure. In the drawings, like reference numerals refer to like elements throughout the several views, unless otherwise indicated. The drawings are intended to illustrate various aspects of the disclosure and should not be construed as limiting the scope of the disclosure.

[0043] In addition, in the following description, specific details are set forth to provide a thorough understanding of examples. However, persons having ordinary skill in the art will understand that the aspects described herein can be practiced without such specific details.

[0044] The embodiments of the present disclosure provide a loudspeaker audio classification method under a plurality of heterogeneous devices, which can be applied to the loudspeaker audio analysis process in the mobile Internet scene.

[0045] Referring to Figure 1 A flowchart of a loudspeaker audio classification method under a plurality of heterogeneous devices provided by the embodiments of the present disclosure is shown. As shown in Figure 1 The method mainly includes the following steps:

[0046] Step 1, calling a built-in acceleration sensor of a mobile phone to collect an initial vibration signal caused by playing of a same on-board loudspeaker;

[0047] For example, an independent application running on an Android / iOS platform is implemented, which can be installed on different heterogeneous smart phones such as Samsung S8, S9, S10+, Note8, Xiaomi 9, MIX3, MIX4, etc., and is used to call the built-in acceleration sensor of the mobile phone to collect the LIV vibration signal caused by playing of the same on-board loudspeaker when the smart phone is running, and send the signal to a background server to identify the audio content of the victim.

[0048] Step 2, performing a preprocessing operation on the initial vibration signal;

[0049] On the basis of the above-mentioned embodiments, the step 2 specifically includes:

[0050] interpolating the initial vibration signal;

[0051] filtering the initial vibration signal using a Kalman filter with a preset cutoff frequency;

[0052] segmenting the initial vibration signal using coarse-grained and fine-grained segmentation methods based on amplitude thresholds of the signal in time domain and frequency domain;

[0053] normalizing the initial vibration signal using a preset formula.

[0054] Further, the preset formula is wherein μ is the average value of the amplitudes of the original signal, max(abs(Signal)) is the maximum value of the absolute values of the amplitudes of the original signal, Signal(i) is the i-th amplitude of the original signal, and Signal(i) Normalization is the i-th amplitude of the normalized original signal.

[0055] In specific implementation, the vibration signal collected by the accelerometer is preprocessed, specifically in the following steps:

[0056] Interpolation and filtering. Since the sampling rates of the accelerometers of different smartphones are not unified, which will affect the audio content recognition under multi-heterogeneous devices, we use the sine interpolation method to unify the sampling rates of the accelerometers of different devices to 1000 Hz, and then use the Kalman filter with a cutoff frequency of 80 Hz to eliminate the influence of human motion, environmental noise and device noise.

[0057] Segmentation. Since the vibration signal is continuous in the time domain, we first need to extract the potential LIV segments generated on the loudspeaker from the collected signal. Therefore, the signal can be segmented in coarse-grained and fine-grained manners. Specifically, for coarse-grained segmentation, the absolute value of the filtered signal is first processed, and a sliding window with a length of 300 and 100 is used for two rounds of smoothing. On this basis, the maximum value M max and the minimum value M min of the smoothed sequence can be obtained. The region with an amplitude greater than 0.9M min +0.15M max is defined as a region where the audio signal exists, and the start and end points of each region are located, and 100 and 200 samples are moved forward and backward respectively to ensure that the entire speech is covered. For fine-grained segmentation, the threshold segmentation method of spectral energy is adopted here. Specifically, the short-time Fourier transform is first performed on the input signal to obtain a frequency spectrum matrix containing frequency and corresponding power changing over time. According to the calculated short-time Fourier transform, the total cumulative power T pow is calculated. According to experience, T powthe average of the first 50 sample points of the signal as a threshold Thres pow , find the first sample point that exceeds 3Thres pow and 2Thres pow as the front and back segmentation points of the fine-grained segmentation respectively, as shown in Figure 2

[0058] Normalization. Due to the different playback volume or internal volume characteristics of different mobile phones, the amplitude of the collected LIV signal is not uniform, which is not conducive to cross-device LIV classification. Therefore, the average value of the signal can be normalized to 0, and the amplitude can be normalized to between-1 and 1, so as to eliminate the interference of hardware differences on the signal amplitude.

[0059]

[0060] Furthermore, the present application constructs an LIV detection mechanism to improve the attack efficiency and concealment. Specifically, the present application finds that although the previous process standardizes the data by filtering the interference caused by body movement and mobile phone self-noise, it still retains the interference caused by using touch input and the like, that is, sliding the screen and the vibration of the mobile phone ring, and the main frequency component thereof is between 80Hz and 200Hz.

[0061] Step 3, removing the interference signal in the pre-processed initial vibration signal to obtain the target vibration signal;

[0062] On the basis of the above embodiment, the step 3 specifically comprises:

[0063] Step 3.1, calculating the signal features in each time window of the pre-processed initial vibration signal and putting them into a one-class support vector machine classifier to obtain an output normalized score, wherein the signal features include Sharpness, Smoothness, SpecKurt and SpecCrest;

[0064] Step 3.2, taking the signal corresponding to the normalized score lower than the threshold value as the target vibration signal.

[0065] In specific implementation, as Figure 3 ​As shown, the time and frequency profiles of the pre-processed initial vibration signals are different for the three signals (ring, swipe, and LIV). For example, in the time-frequency domain, the swipe signal is sharper. In the frequency domain, the distribution of the ring vibration is more uniform. Although the frequency ranges of the swipe and LIV signals are similar, the distribution of the swipe signal is smoother. In view of this, in order to save operation time, we only selected four features in each time window to accurately and timely identify the LIV segment: Sharpness, Smoothness, SpecKurt, and SpecCrest. In order to further address the problem of continuous extraction and calculation of features to identify LIV signals, the present application will only respond to signals above 100 Hz.

[0066] When a high-frequency signal is detected, the detection model enters the confirmation stage, i.e., distinguishing LIV signals from other signals, which requires constructing a single-class support vector machine classifier. In order to more accurately identify, the threshold will be manually set. Specifically, under the given unknown signal cutting condition, the trained model will output a normalized score, and only samples below the threshold are considered to be target vibration signals.

[0067] Step 4, screening signal statistical features highly related to audio content and device-independent;

[0068] Further, the step 4 specifically comprises:

[0069] Using the random forest method to screen out key features highly related to audio content from the LibXtract feature library;

[0070] On the basis of the screened key features, a correlation analysis is used to select a preset number of features less affected by the device as signal statistical features.

[0071] In specific implementation, when the vibration signal is pre-processed and the detected LIV signal is extracted, it enters the feature selection stage. Considering the influence of device diversity on LIV-based speech content, the present application explores the use of statistical features of LIV signals to distinguish speech content. Specifically, the present application uses a commonly used lightweight feature analysis library LibXtract to study 40 scalar features in the time and frequency domains to mine key features that suppress device diversity.

[0072] In order to screen out key features highly related to audio content, we use random forests to analyze the contribution of each feature to audio content classification. Figure 4The feature importance of the ranking when classifying the speech content is shown. Note that the importance of the remaining features is much lower than the features listed in the figure, so we first exclude them from the set of potential key features. Thus, there are 15 features selected as key features that are highly relevant to the speech content.

[0073] Based on the determined key features, we further explore the statistical features that reveal less device-specific information to suppress device diversity. The present invention finds that the main reason for the heterogeneous characteristics of different devices is the diversity of loudspeakers, signal propagation media, and accelerometers. Therefore, it is possible to strictly limit the use of Figure 5 The experimental setup shown in the figure, a mobile phone is selected as the speech player (referred to as VP), and another mobile phone is selected as the LIV signal recorder (referred to as LSR). For each experiment, the LSR is placed above the VP with a specific medium in between. By changing the VP, LSR, and medium between the two smartphones respectively, specifically, the sources of different VP and LSR are any of the Samsung S8, S9, and Xiaomi 9 mobile phones, and the medium is any of a book, cloth, and mouse pad. Each of the mobile phones as LSR can collect the audio signals of 10 hot words (Password, Code, Key, Secret, Account, Salary, Encoder, Bank, Number, Word) pronounced by 10 volunteers, each of which is pronounced 50 times, played by each of the mobile phones as VP. Therefore, 5000 LIV signal sets are obtained in the same order under each single variable. By calculating the key features of the LIV signal set, a feature matrix of (5000x15) is formed, and each feature corresponds to a (5000x1) vector. Then, the average Pearson correlation coefficient (PCC) of the specific single feature vector between the multiple variables is calculated to explore the correlation between each feature of the LIV signal and the inherent properties of the signal in the case of serious device diversity. It is worth noting that, considering the simplicity and intuitiveness of PCC, the present invention uses PCC to calculate the correlation between different vector pairs. Other methods, such as feature filtering methods based on support vector machines or KNN, can also work well. By calculating the Pearson correlation coefficient of the specific single feature vector between multiple variables, the 12 features that are least related to the device can be finally screened out, and these 12 features are key features that are highly relevant to the audio content retained in the LIV signal and independent of the feature device, as shown in Table 1.

[0074]

[0075] Table 1

[0076] Step 5, according to the signal statistical features, using an adversarial neural network to classify the audio content of the target vibration signal.

[0077] Optionally, the adversarial neural network comprises a feature extractor, a word recognizer and a device discriminator, wherein the feature extractor comprises a feedforward neural network and a convolutional neural network.

[0078] Further, the step 5 specifically comprises:

[0079] The feature extractor inputs the signal statistical features and the frequency spectrum of the target vibration signal into the feedforward neural network and the convolutional neural network for compression to obtain one-dimensional features and connect them in the same feature vector;

[0080] The multiple feature vectors are input into the word recognizer to output the confidence of each audio content, and when the value of the confidence is greater than an empirical threshold, the content of the audio signal is determined as the category with the maximum confidence to form a word label;

[0081] The output of the feature extractor and the word recognizer is connected into a device label vector by the device discriminator;

[0082] The min-max game between the feature extractor, the word recognizer and the device discriminator is iteratively updated by using a total loss function to obtain the classification result corresponding to the audio content of the target vibration signal.

[0083] In specific implementation, it is considered that both the deep learning model irrelevant to features (such as DenseNet) and the machine learning model considering only features (such as support vector machine) have poor generalization ability. Therefore, the adversarial neural network with feature enhancement is used in the multi-task learning method to suppress the influence of device diversity on the model universality. The specific task refers to vibration recognition under a specific device, and the multi-task learning specifically refers to learning common information under multiple devices (by using the word recognizer) and suppressing the influence of the specific device (by using the device discriminator). It mainly includes the following steps:

[0084] LIV signal conversion. When the segmented LIV signal is detected, the model should convert the LIV signal into a standardized frequency spectrum. In the case of a given input LIV signal, the short-time Fourier transform (STFT) is calculated. In order to calculate the short-time Fourier transform, the given LIV signal is divided into equal-length blocks (denoted as wlen=256); then the Fourier transform is calculated on each block. When the period hamming window length is 256 and the overlap length is 250, the resulting signal spectrum matrix contains the frequency and the corresponding power change over time. Since most of the original spectrum matrix is close to zero, directly using the matrix may cause data overflow. In order to make full use of the useful information retained in the spectrum, the square root of the elements in the spectrum matrix (spectrum matrix standardization) is taken, and the image size is further unified to 200*200*1 to adapt to the neural network.

[0085] Artificial feature extraction. The z-axis signal of the acceleration sensor can be first pre-processed to let M z (k) and T z (k) is the kth value of the z-axis acceleration sensor and the timestamp, Z f and Z m is the frequency and magnitude of the z-axis in the frequency domain. To obtain the key features, first use M z (k) to calculate the features of the z-axis signal in the time domain, including Smoothness, Kurtosis, etc., and then use {Z f , Z m} to calculate the features of the signal in the frequency domain, including SpecKurt, SpecCrest, etc. Thus, the matrix representation of the feature set can be obtained: {StdDev, Smoothness, Kurtosis, …, SpecKurt, SpecCrest, MaxFreq, MaxMagnitude}. As Figure 6 shown, the present application uses this feature set as the input of the adversarial neural network to improve the generalization performance of the system to eavesdrop on heterogeneous devices.

[0086] Adversarial neural network. Although the signal spectrum matrix and the artificially screened features contain information highly related to the inherent properties of the signal, they also include the influence of device diversity. Therefore, the present application proposes an adversarial neural network to eliminate the uniqueness of each field (defined as different devices) and can extract the commonality shared in different fields (defined as the label of a single word). Therefore, the network can identify the voice vibration signal collected under an unknown device. Figure 6 The structure diagram of the adversarial neural network is shown.

[0087] Let x1∈X1 be the input tensor of the spectrogram, x2∈X2 be the input tensor of the artificial feature vector, y∈Y be the word label, and s∈S represent the auxiliary label referring to the specific input tensors x1 and x2, i.e. the device label. Note that y and s are the benchmark truth assigned to each input x1 and x2. The network consists of three parts: a feature extractor F, a word recognizer W, and a device discriminator D. When it works, F extracts elements from the inputs x1 and x2, labeled as f1 and f2, and then concatenates them into a whole f all . Then W uses f all to calculate its result w, which is the predicted word label. Then w and f all are concatenated as the input of D, which will output the predicted device label d. The relationship between the three follows the min-max game, i.e. the feature extractor tries to improve the performance of the word recognizer while cheating the device discriminator. In this way, regardless of the influence of device diversity, the adversarial neural network can ultimately learn the common features of the audio content itself.

[0088] The adversarial neural network is composed of a feature extractor F, a word recognizer W and a device discriminator D. Details are shown as follows:

[0089] Feature extractor. First, we convert the input into a tensor form, since the dimensions of the spectrum matrix (batchsize x 1 x 64 x 64) and the artificial feature vector (batchsize x 1 x 12) are different, we put them into CNN and FNN respectively to extract compressed feature vectors. Specifically, the CNN network has two convolutional layers and one fully connected layer. After each convolutional layer, there is a ReLU activation function and a pooling layer. The convolution kernel, step, padding size and pooling kernel are set to 3, 1, 1 and 2 respectively. In addition, the FNN first performs a batch normalization on the artificial feature vector, then uses two fully connected layers and a ReLU activation function, and adds a dropout layer to prevent overfitting. Let the parameters of CNN and FNN be and X1 and X2 are the input spectrum matrix and artificial feature vector, which can be calculated according to The output f of the feature extractor can be calculated as all where is a concatenation operation.

[0090] Word recognizer. The present embodiment sets three fully connected layers and a Softmax activation function, and specifies θ W as the parameters of the word recognizer. The output f all of the feature extractor is brought into the output w, which is w = f all x θ W .

[0091] Device discriminator. Since f all contains both word-related features and device-specific features, it needs to be considered in order to identify the commonality of different devices. At the same time, although some features are specific to the device, they still help to recognize the word, so these features still need to be retained, therefore, the present embodiment sets as the input of the device discriminator. Then use two fully connected layers with the activation function Softmax to project to output d. Therefore, when the parameters θ D of D are given, the output d can be calculated as

[0092] It should be noted that the semantic information of the vibration signal to be identified can come from outside the target vocabulary, i.e. the null class. Therefore, an acceptance threshold can be set. When the maximum value of the elements in the output w is greater than the threshold, the vibration signal is considered by the embodiment of the present disclosure to have the semantic information with the maximum confidence, otherwise the vibration signal is considered as the NULL class.

[0093] The present application realizes the min-max game with iterative update by establishing a total loss function L = L w -λL d (λ>0), where L w and L d are the cross-entropy losses of W and D respectively. After calculating the total loss, θ F , θ W and θ D can be updated by backpropagation network. Here, the embodiment of the present disclosure uses the Adam optimization function. The learning rate of all components is set to 5e-5, and the number of samples trained under each batch is set to 100. In addition, in order to prevent the model from overfitting, the embodiment of the present disclosure sets the network training to end in time when the loss no longer decreases.

[0094] The loudspeaker audio classification method under multiple heterogeneous devices provided by the embodiment comprises the following steps: collecting the vibration signal generated by the loudspeaker playing through the built-in acceleration sensor, and sending it to the remote server; then using the interpolation, filtering, segmentation and normalization method to preprocess the vibration signal; using the one-class support vector machine method to identify the target vibration signal from the candidate signal, thereby reducing the system energy consumption; screening out a key feature set which is highly related to the audio content and less affected by device diversity from numerous statistical features; finally, using the multi-task learning method of the adversarial neural network to solve the restriction caused by device diversity, and realizing high-precision speech recognition.

[0095] The units involved in the embodiments of the present disclosure can be implemented in software or hardware.

[0096] It should be understood that parts of the present disclosure can be implemented in hardware, software, firmware or a combination thereof.

[0097] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any changes or replacements within the technical range disclosed by the present disclosure can be easily thought of by those skilled in the art, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method for classifying loudspeaker audio in a multi-heterogeneous device environment, characterized in that, include: Step 1: Use the phone's built-in accelerometer to collect the initial vibration signal played by the same onboard speaker; Step 2: Perform preprocessing on the initial vibration signal; Step 3: Remove interference signals from the preprocessed initial vibration signal to obtain the target vibration signal; Step 3 specifically includes: Step 3.1: Calculate the signal features in each time window of the preprocessed initial vibration signal and put them into a single-class support vector machine classifier to obtain the output normalized score. The signal features include Sharpness, Smoothness, SpecKurt and SpecCrest. Step 3.2: The signal corresponding to the normalized score being lower than the threshold is taken as the target vibration signal; Step 4: Filter signal statistical features that are highly relevant to the audio content and independent of the device; Step 4 specifically includes: The random forest method was used to filter out key features that were highly relevant to the audio content from the LibXtract feature library; Based on the selected key features, correlation analysis was used to select a preset number of features that are less affected by the equipment as signal statistical features; Step 5: Based on the statistical characteristics of the signal, an adversarial neural network is used to classify the audio content of the target vibration signal. The adversarial neural network includes a feature extractor, a word recognizer, and a device discriminator. The feature extractor includes a feedforward neural network and a convolutional neural network. Step 5 specifically includes: The feature extractor inputs the signal statistical features and the spectrum of the target vibration signal into the feedforward neural network and the convolutional neural network for compression, and obtains one-dimensional features and concatenates them in the same feature vector; Multiple feature vectors are input into the word recognizer, which outputs the confidence score of each audio content. When the confidence score is greater than the empirical threshold, the audio content is identified as the category with the highest confidence score, thus forming a word label. The outputs of the device discriminator, feature extractor, and word recognizer are concatenated into the output device label vector; By iteratively updating the min-max game between the feature extractor, word recognizer, and device discriminator using the total loss function, the classification result corresponding to the audio content of the target vibration signal is obtained.

2. The method according to claim 1, characterized in that... Step 2 specifically includes: Interpolate the initial vibration signal; The initial vibration signal is filtered using a Kalman filter with a preset cutoff frequency; The initial vibration signal is segmented using coarse-grained and fine-grained segmentation methods based on amplitude thresholds in the time and frequency domains of the signal. The initial vibration signal is normalized using a preset formula.

3. The method according to claim 2, characterized in that... The preset formula is: ,in, The average value of the original signal amplitude. The maximum value of the absolute value of the original signal amplitude. For the i-th amplitude of the original signal, This represents the i-th amplitude value after normalization of the original signal.

Images