Head tracking-based TWS earphone audio scene identification method

Abstract

The invention relates to the technical field of audio scene recognition methods, in particular to a TWS earphone audio scene recognition method based on head tracking, and the method comprises the following specific steps: S1, collecting audio data; s2, performing feature extraction on the audio data; s3, carrying out weight calculation on the head information; s4, constructing an audio scene classification model; s5, identifying the audio scene: according to the obtained audio scene classification model, inputting real-time data, identifying the audio scene, and combining a multi-head attention mechanism with a long-short term memory network, thereby not only realizing effective capture of short-term and long-term time sequence characteristics, but also greatly improving the sensitivity of the model to environmental change; the dynamic change of a scene can be quickly adapted while the high recognition accuracy is kept; high accuracy and quick response are mutually promoted, and smoother and more accurate user experience is provided; and secondly, the improvement of the calculation efficiency and the reduction of the energy consumption form a virtuous cycle.

Description

Technical Field

[0001] The present invention relates to the technical field of audio scene recognition methods, and in particular to a TWS headset audio scene recognition method based on head tracking. Background Art

[0002] With the rapid development of wireless technology, true wireless stereo (TWS) earphones have become one of the most popular products in today's consumer electronics market. Users' requirements for TWS earphones are no longer limited to high-quality audio experience, but also expect these devices to be able to intelligently adapt to different usage scenarios. In this context, audio scene recognition technology came into being and became the key to the intelligence of TWS earphones.

[0003] Traditional audio scene recognition methods mainly rely on the analysis of audio signals. These methods usually use features such as Mel-frequency cepstral coefficients (MFCC) or linear prediction coefficients (LPC) combined with classifiers such as support vector machines (SVM) or hidden Markov models (HMM) to achieve scene recognition. Although these methods have achieved certain results in some applications, they often face many challenges in resource-constrained and complex environments such as TWS headphones.

[0004] First, methods that rely solely on audio features are prone to misjudgment in complex noisy environments. For example, in a noisy coffee shop and a crowded subway station, pure audio features may have difficulty accurately distinguishing between the two scenes. Second, traditional methods have high computational complexity and it is difficult to achieve real-time and continuous scene recognition under the limited processing power and battery capacity of TWS headphones. In addition, these methods lack awareness of user behavior and cannot capture important contextual information such as whether the user is moving and how the head posture changes.

[0005] More importantly, existing technologies perform poorly when dealing with dynamically changing audio environments. When users quickly switch from one scene to another, traditional methods often take a long time to adapt and make correct judgments. This not only affects the user experience, but may also cause the audio settings of the headphones to be inappropriate, such as maintaining a high-intensity noise reduction effect after the user has left the noisy environment. Summary of the invention

[0006] In response to these problems, the present invention proposes a TWS headset audio scene recognition method based on head tracking. This method cleverly combines audio data and head motion data, and achieves more accurate and efficient scene recognition through innovative multimodal fusion technology. The present invention not only solves the problem of low recognition accuracy in complex environments in the prior art, but also overcomes the defects of low computational efficiency and high energy consumption of traditional methods.

[0007] The present invention proposes a TWS headset audio scene recognition method based on head tracking, and the specific steps are as follows:

[0008] S1. Collect audio data: insert a pre-processing module into the head tracking function of the TWS headset, and use the head tracking module to collect audio data and motion data of the left and right earphones;

[0009] S2. Extract features from audio data: extract the fundamental frequency and spectrogram of the sound signal using the wavelet transform method, resample the audio data and extract the fundamental frequency, use the Hilbert-Huang transform to estimate the instantaneous amplitude spectrum, instantaneous frequency spectrum and Hilbert spectrum, and use the Mel frequency filter bank with logarithmic weighted average to extract the Mel spectrum to obtain the feature map of the audio signal;

[0010] S3. Calculate the weight of the head information: Perform spatial attention weighting on the head motion data, and use multi-head attention operation to obtain the weight of the head information. The calculation formula of spatial attention weighting is:

[0011] W s =softmax(f s (H i , H j ))

[0012] Among them, W s is the spatial attention weight, f s is the spatial attention mapping function, H i and H j are the i-th and j-th head motion data respectively;

[0013] S4. Construct an audio scene classification model: Use a combined model architecture of a convolutional neural network and a long short-term memory network, adopt a contrastive learning algorithm, take the feature map obtained in S2 and the weight value obtained in S3 as input, and output the recognition result of the audio scene. The loss function of contrastive learning is:

[0014]

[0015] Among them, z i and z j is the feature representation of the positive sample pair, z k is the feature representation of negative samples, s i m is the cosine similarity function, τ is the temperature parameter;

[0016] S5. Identify the audio scene: According to the obtained audio scene classification model, input the real-time data and identify the audio scene.

[0017] Preferably, in S1, collecting audio data and motion data of the left and right earphones includes: inserting a preprocessing module into the head tracking function, using the head tracking module to obtain head motion data in real time, and combining the data acquisition module to obtain multi-microphone audio data of the left and right earphones, wherein the acquisition formula of the head motion data is:

[0018] H=[a x , a y , a z ,ω x ,ω y ,ω z , m x , m y , m z ],

[0019] Among them, a x , a u , a z is the accelerometer data, ω x ,ω y ,ω z is the gyroscope data, m x , m y , m z The magnetometer data.

[0020] Preferably, in S2, extracting features from audio data includes: extracting the fundamental frequency and spectrogram of the sound signal using a wavelet transform method, and the specific steps are as follows:

[0021] For a given audio sample, the wavelet basis function is constructed according to the basic formula of wavelet transform:

[0022]

[0023] In the formula, a is the multi-sampling frequency, b is the sampling frequency, t is the time, f is the frequency, t k is the kth moment in the sample, N is the number of sampling points, ψ a , b(t) represents the kth wavelet basis function

[0024] Spectrum of samples;

[0025] Next, the audio signal is demodulated using wavelet transform, and the filter coefficients are expressed as:

[0026]

[0027] Among them, x(t) is the audio signal, ψ a , b * (t) is the conjugate of the wavelet basis function;

[0028] Finally, the fundamental frequency is extracted using the following formula:

[0029] f0=argmax f |W f (a, b)|,

[0030] Where f0 represents the fundamental frequency of the audio signal, |W f (a, b)| represents the modulus of the wavelet coefficients.

[0031] Preferably, in S2, extracting features from the audio data further comprises: using Hilbert-Huang transform to estimate the instantaneous amplitude spectrum, the instantaneous frequency spectrum and the Hilbert spectrum;

[0032] For a given audio signal, the empirical mode decomposition method is used to decompose the original signal into a finite number of intrinsic mode functions, namely:

[0033] x(t)=∑ i c i (t)+r n (t),

[0034] Among them, c i (t) is the ith eigenmode function, r n (t) is the residual term;

[0035] Then the intrinsic mode function is converted into an analytical signal through Hilbert transform, and the instantaneous frequency is obtained:

[0036] Where: f i (t) represents the instantaneous frequency of the signal in the i-th sample, θ i (t) represents the phase angle of the signal, i represents the analytical signal of the i-th eigenmode function;

[0037] Finally, a Mel-frequency filter bank with log-weighted averaging is used to extract the Mel-spectrum:

[0038] M(m)=∑ k H m (k)·|X(k)| 2 ,

[0039] Where: M(m) is the Mel spectrum, X(k) represents the spectrum of the input signal, H m (k) represents the Mel frequency filter of logarithmic weighted average, k represents the dimension of DFT, and m represents the index of Mel filter.

[0040] Preferably, in S3, weight calculation of head information includes: preprocessing the head motion data to unify its units; for the preprocessed head motion data, firstly using spatial attention operation to obtain weight coefficients of each orientation, and then using multi-head attention to perform weight integration to obtain spatial weighted features. The specific steps are as follows:

[0041] For head motion data, spatial attention is used to obtain the weight coefficients of each orientation:

[0042] α i =softmax(W s tanh(W h ·H i +b h )),

[0043] Where: H i represents the original head motion data, W S is the weight matrix of spatial attention, W h is the hidden layer weight matrix, b h is the bias term, α i Represents the attention weight of the i-th position; then use multi-head attention to integrate the head information by weight:

[0044] MultiHead(Q,K,V)=Concat(head1,...,head h )·W O ,

[0045] Where: Q, K, V are query, key and value matrices respectively, is the learnable parameter matrix.

[0046] Preferably, in S4, constructing the audio scene classification model includes: using a combined model architecture of a convolutional neural network and a long short-term memory network to construct the audio scene classification model, including:

[0047] S41. Perform convolutional neural network feature extraction on Mel cepstral coefficients and spectrograms;

[0048] S42. Input the spatial weighted features and perform serialization in combination with the long short-term memory network;

[0049] S43. Use context attention to assign different weights to different positions to obtain a global context feature vector;

[0050] First, the i-th context feature is obtained through a linear layer:

[0051] h i =W h ·xi +b h ,

[0052] Where: x i represents the spatial weighted feature obtained by the i-th multi-head attention after serialization, W h is the learnable projection matrix, b h is the bias term, and then, multi-head attention is used to linearly transform the context features: i =MultiHead(Q i , K i , V i ), where: Q i , K i , V i are the query, key, and value matrices of the i-th context feature respectively; then, the context attention weight is used to calculate the weight value of the context feature: β i =softmax(v T tanh(W c ·z i +b c )), where: v, W c , b c is a learnable parameter; finally, the global context vector is input into the multi-layer neural network and the result is obtained: y = MLP (∑ i β i ·z i ), where y is the global context vector, MLP is a multi-layer perceptron;

[0053] S44. Add a multi-layer perceptron and a global average pooling layer after the combined model architecture to output the predicted category of the audio sample.

[0054] Preferably, in S4, constructing the audio scene classification model further comprises: using a contrastive learning algorithm: using an objective function to learn to enhance the association between similar data samples and weaken the association between data samples of different categories, and the objective function is as follows:

[0055]

[0056] Where: x i , x j represents a positive sample pair, x k represents negative samples, f(·) represents the encoder function, sin(·,·) represents the cosine similarity function, T is the temperature parameter.

[0057] Preferably, in S1, the data acquisition module acquires multi-microphone audio data of the left and right earphones, including:

[0058] For a microphone array, the audio data from each microphone is recorded and expressed as:

[0059] A i (t) = s(t)·h i (t)+n i (t), where i is the serial number of the microphone, A i (t) represents the audio data collected by the i-th microphone, t is the time axis, s(t) is the source signal, h i (t) is the impulse response of the ith microphone n i (t) is the noise; for the head tracking module, a gyroscope and a magnetometer are used to record the motion data of the head in all directions, which can be expressed as: H j =[a x , a y , a z ,ω x ,ω y ,ω z , m x , m y , m z ] j , where: j is the sequence number of the head motion data, H j represents the jth sample data, a, ω, m represent the three-dimensional components of acceleration, angular velocity and magnetic field intensity respectively; then, the audio data and head motion data are combined: D = {A i (t), M j |i=1,...,M;j=1,...,N}, where: D represents the data set, M is the number of microphones, and N is the number of head motion data sampling points.

[0060] Preferably, in S5, identifying the audio scene includes: identifying the real-time collected data, first obtaining audio features through a convolutional neural network feature extractor, and then obtaining a spatial weight coefficient based on the features of the head motion data; then, using the context attention function described in S43 to assign a weight value to the head information, and inputting the global context vector into the multi-layer neural network, and using the global average pooling layer described in S44 to obtain a prediction result, wherein the calculation formula of the prediction result is:

[0061] y pred =softmax(W·GAP(F CNN (A)+F LSTM (H))+b),

[0062] Where: y pred is the prediction result, F CNN and F LSTMare the feature extraction functions of convolutional neural networks and long short-term memory networks, A is the audio feature, H is the head motion feature, GAP is the global average pooling operation, W and b are the weight and bias of the fully connected layer, respectively.

[0063] Preferably, in the objective function, a contrastive learning algorithm is used, a negative sample enhancement strategy is adopted, negative samples are resampled and enhanced, and positive samples are noise enhanced, and the enhancement ratio is as follows:

[0064]

[0065] Where: R is the enhancement ratio, N p is the number of positive samples after enhancement, N p is the number of original positive samples, N n is the number of negative samples, α is a hyperparameter;

[0066] Next, obtain the trained network parameters according to the loss function and update the training parameters:

[0067] Among them, θ′ is the updated parameter, θ is the current parameter, ηη is the learning rate, is the gradient about the current parameter, fθ is the neural network function, x is the input data, and y is the true label;

[0068] Input the dataset into the convolutional neural network feature extractor, multi-head attention and self-attention, learn the weight matrix and parameters, use the objective function to learn the joint representation, and maximize the objective function:

[0069]

[0070] Among them, p pos represents the distribution of positive sample pairs, sim is the similarity function, and f θ is the encoder function and τ is the temperature parameter.

[0071] The beneficial effects of the present invention are mainly reflected in the following aspects:

[0072] The core of the present invention lies in its unique algorithm design, which cleverly solves multiple seemingly contradictory technical problems. First, by introducing head motion data, the present invention increases the dimension of input information, which theoretically increases the computational complexity. However, we innovatively designed a lightweight feature extraction and fusion algorithm, which not only does not increase the computational burden, but also improves the overall computational efficiency by reducing the reliance on complex audio processing.

[0073] Secondly, the contrastive learning algorithm of the present invention not only improves the generalization ability of the model, but also enhances the model's ability to distinguish different scenarios. These two goals are often in a trade-off relationship in traditional machine learning, but our method successfully strikes a balance between the two goals through a carefully designed negative sample enhancement strategy.

[0074] In addition, the combination of the multi-head attention mechanism and the long short-term memory network of the present invention not only effectively captures short-term and long-term temporal features, but also greatly improves the model's sensitivity to environmental changes. This design enables our method to quickly adapt to dynamic changes in the scene while maintaining high recognition accuracy.

[0075] The beneficial effects of the present invention are multifaceted, and there are significant synergies and superpositions between them. First, high accuracy and fast response promote each other, providing a smoother and more accurate user experience. Secondly, the improvement of computing efficiency and the reduction of energy consumption form a virtuous circle, making continuous and real-time scene recognition possible. Furthermore, the fusion of multimodal data not only improves the accuracy of recognition, but also lays the foundation for future functional expansion (such as user behavior analysis).

[0076] In general, this invention successfully solves a series of key technical problems in TWS headset audio scene recognition through innovative algorithm design and multimodal data fusion. It not only greatly improves the accuracy and efficiency of recognition, but also opens up a new direction for the intelligence of TWS headsets. The application of this method will significantly improve the user's listening experience, promote the development of TWS headsets towards a higher level of intelligence, and set a new technical standard for future smart wearable devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 It is a logic block diagram of the overall method of the present invention.

[0078] Figure 2 It is the signal acquisition logic block diagram of the present invention.

[0079] Figure 3 This is a logic block diagram of data preprocessing of the present invention.

[0080] Figure 4 This is a logic block diagram of the deep learning module processing of the present invention.

[0081] Figure 5 It is the signal separation logic block diagram of the present invention.

[0082] Figure 6 This is a module fusion logic block diagram of the present invention. DETAILED DESCRIPTION

[0083] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects thereof are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0084] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0085] Example 1

[0086] See also Figure 1-6 The present invention relates to a TWS headset audio scene recognition method based on head tracking. The method first inserts a preprocessing module into the head tracking function of the TWS headset, and uses the head tracking module to collect audio data and motion data of the left and right headsets. This step makes full use of the characteristics of the TWS headset and provides rich input information for subsequent scene recognition.

[0087] Next, this method extracts features from the collected audio data. Specifically, the wavelet transform method is first used to extract the fundamental frequency and spectrogram of the sound signal. Then, the audio data is resampled and the fundamental frequency is extracted. After that, the Hilbert-Huang transform is used to estimate the instantaneous amplitude spectrum, instantaneous frequency spectrum, and Hilbert spectrum. Finally, the Mel frequency filter group with logarithmic weighted average is used to extract the Mel spectrum to obtain the feature map of the audio signal. This series of feature extraction steps can fully capture the various characteristics of the audio signal and provide rich input information for subsequent scene classification.

[0088] Preferably, when calculating the weight of the head information, the present invention first performs spatial attention weighting on the head motion data, and uses a multi-head attention operation to obtain the weight of the head information. The calculation formula for spatial attention weighting is:

[0089] W s =softmax(f s (H i , H j ))

[0090] Among them, W s is the spatial attention weight, f s is the spatial attention mapping function, H i and H j are the i-th and j-th head motion data respectively. This innovative weight calculation method can effectively extract key information from head motion data and provide valuable context clues for scene recognition.

[0091] A core step of the present invention is to construct an audio scene classification model. The model uses a combined model architecture of a convolutional neural network and a long short-term memory network, and adopts a contrastive learning algorithm. The model takes the feature map and weight values ​​obtained in the previous steps as input and outputs the recognition result of the audio scene. It is worth noting that the present invention adopts a contrastive learning loss function, which is:

[0092]

[0093] In this formula, z i and z j is the feature representation of the positive sample pair, z k is the feature representation of negative samples, sim is the cosine similarity function, and τ is the temperature parameter. The application of this contrastive learning can significantly improve the classification performance of the model, especially when the sample size is small or the categories are unbalanced.

[0094] Finally, this method uses the obtained audio scene classification model to input real-time data and identify the audio scene. This enables the method to realize real-time audio scene recognition on resource-constrained devices such as TWS headphones, which is very practical.

[0095] In one embodiment of the present invention, the specific steps of collecting audio data and motion data of the left and right earphones include: inserting a preprocessing module into the head tracking function, using the head tracking module to obtain head motion data in real time, and combining the data collection module to obtain multi-microphone audio data of the left and right earphones. The collection formula of the head motion data is:

[0096] H=[a x , a y , a z ,ω x ,ω y ,ω z , m x , m y , m z ]

[0097] In this formula, a x , a u , a z is the accelerometer data, ω x ,ω y ,ω z is the gyroscope data, m x , m y , m z This comprehensive data collection method can provide rich input information for subsequent scene recognition.

[0098] Furthermore, the step of extracting features from audio data in the present invention includes extracting the fundamental frequency and spectrogram of the sound signal using a wavelet transform method. Specifically, firstly, a wavelet basis function is constructed according to the basic formula of wavelet transform:

[0099]

[0100] Among them, a is the multi-sampling frequency, b is the sampling frequency, t is time, f is frequency, t k is the kth moment in the sample, N is the number of sampling points, ψ a , b(t) represents the spectrum of the kth sample of the wavelet basis function.

[0101] Then, the audio signal is demodulated using wavelet transform and the filter coefficients are expressed as:

[0102]

[0103] Among them, x(t) is the audio signal, ψ a , b * (t) is the conjugate of the wavelet basis function;

[0104] Finally, the fundamental frequency is extracted using the following formula:

[0105] f0=argmax f |W f (a, b)|,

[0106] Where f0 represents the fundamental frequency of the audio signal, |W f (a, b)| represents the modulus of the wavelet coefficients.

[0107] This feature extraction method based on wavelet transform can effectively capture the time-frequency characteristics of audio signals and provide important input information for subsequent scene classification.

[0108] Another innovation of the present invention is to use the Hilbert-Huang transform to estimate the instantaneous amplitude spectrum, instantaneous frequency spectrum and Hilbert spectrum. The specific steps are as follows: First, the original signal is decomposed into a finite number of intrinsic mode functions using the empirical mode decomposition method, namely:

[0109] x(t)=∑ i c i (t)+r n (t),

[0110] Among them, c i (t) is the ith eigenmode function, r n (t) is the residual term;

[0111] Then the intrinsic mode function is converted into an analytical signal through Hilbert transform, and the instantaneous frequency is obtained:

[0112] Where: f i (t) represents the instantaneous frequency of the signal in the i-th sample, θ i (t) represents the phase angle of the signal, i represents the analytical signal of the i-th eigenmode function;

[0113] Finally, the Mel frequency filter bank with logarithmic weighted averaging is used to extract the Mel spectrum: M(m) = ∑ k H m (k)·|X(k)| 2 ,

[0114] Where: M(m) is the Mel spectrum, X(k) represents the spectrum of the input signal, H m (k) represents the Mel frequency filter of logarithmic weighted average, k represents the dimension of DFT, and m represents the index of Mel filter.

[0115] This multi-dimensional feature extraction method can comprehensively capture the various characteristics of audio signals and provide rich input information for subsequent scene classification.

[0116] When calculating the weight of head information, the present invention adopts an innovative two-step attention mechanism. First, the spatial attention operation is used to obtain the weight coefficients of each direction. Then, multi-head attention is used to integrate the weights to obtain spatial weighted features. The specific steps are as follows:

[0117] For head motion data, spatial attention is used to obtain the weight coefficients of each orientation:

[0118] α i =softmax(W s ·tan h (W h ·H i +b h )),

[0119] Where: H i represents the original head motion data, W S is the weight matrix of spatial attention, W h is the hidden layer weight matrix, b h is the bias term, α i Represents the attention weight of the i-th position; then use multi-head attention to integrate the head information by weight:

[0120] MultiHead(Q,K,V)=Concat(head1,...,head h )·W O ,

[0121] Where: Q, K, V are query, key and value matrices respectively,

[0122] is the learnable parameter matrix.

[0123] The design of this two-step attention mechanism can effectively extract key information from head motion data and provide valuable contextual clues for scene recognition.

[0124] When constructing an audio scene classification model, the present invention adopts a hybrid model architecture that combines a convolutional neural network and a long short-term memory network. The specific steps include:

[0125] First, convolutional neural network features are extracted from the Mel-frequency cepstral coefficients and spectrograms.

[0126] Then, the spatial weighted features are input and serialized in combination with the long short-term memory network.

[0127] Next, we use contextual attention to assign different weights to different locations and obtain the global context feature vector. The specific implementation of this step is as follows:

[0128] First, the i-th context feature is obtained through a linear layer:

[0129] h i =W h ·x i +b h ,

[0130] Where: x i represents the spatial weighted feature obtained by the i-th multi-head attention after serialization, W h is the learnable projection matrix, b h is the bias term,

[0131] Then, multi-head attention is used to linearly transform the context features: z i =MultiHead(Q i , K i , V i ), where: Q i , K i , V i are the query, key, and value matrices of the i-th context feature, respectively;

[0132] Next, the contextual attention weight is used to calculate the weight value of the contextual feature: βi = softmax(v T tanh(W c ·z i +b c )), where: v, W c , b cis a learnable parameter;

[0133] Finally, the global context vector is input into the multi-layer neural network and the result is obtained: y = MLP (∑ i β i ·z i ), where y is the global context vector and MLP is a multi-layer perceptron.

[0134] After combining the model architecture, the present invention also adds a multi-layer perceptron and a global average pooling layer to output the predicted category of the audio sample. This hybrid architecture can simultaneously utilize the advantages of CNN in spatial feature extraction and LSTM in temporal modeling, achieving effective fusion of audio and head motion data.

[0135] Another innovative point of the present invention is the use of a contrastive learning algorithm. The algorithm uses an objective function to learn to enhance the association between similar data samples and weaken the association between data samples of different categories. The objective function is as follows:

[0136]

[0137] Where: x i , x j represents a positive sample pair, x k represents negative samples, f(·) represents the encoder function, sin(·,·) represents the cosine similarity function, and τ is the temperature parameter. The application of this contrastive learning can significantly improve the classification performance of the model, especially when the sample size is small or the categories are unbalanced.

[0138] In the data collection stage, the present invention adopts an innovative multi-microphone audio data collection method. For the microphone array, the audio data from each microphone is recorded and expressed as:

[0139] A i (t) = s(t)·h i (t)+n i (t),

[0140] Where: i is the serial number of the microphone, A i (t) represents the audio data collected by the i-th microphone, t is the time axis, s(t) is the source signal, h i (t) is the impulse response of the ith microphone n i (t) is the noise;

[0141] For the head tracking module, a gyroscope and a magnetometer are used to record the motion data of the head in all directions, which can be expressed as: H j =[a x , a y , a z ,ωx ,ω y ,ω z , m x , m y , m z ] j ,

[0142] Where: j is the sequence number of the head motion data, H j represents the jth sampling data, a, ω, and m represent the three-dimensional components of acceleration, angular velocity, and magnetic field intensity, respectively;

[0143] Next, combine the audio data with the head motion data: D = {A i (t), H j |i=1,...,M;j=1,...,N}, where: D represents the data set, M is the number of microphones, and N is the number of head motion data sampling points. This comprehensive data collection method provides rich input information for subsequent scene recognition.

[0144] In practical applications, the audio scene recognition method of the present invention processes the data collected in real time. First, the audio features are obtained through the convolutional neural network feature extractor, and then the spatial weight coefficient is obtained based on the features of the head motion data. Next, the context attention function is used to assign weight values ​​to the head information, and the global context vector is input into the multi-layer neural network, and finally the global average pooling layer is used to obtain the prediction result. The calculation formula of the prediction result is:

[0145] y pred =softmax(W·GAP(F CNN (A)+F LSTM (H))+b),

[0146] Where: y pred is the prediction result, F CNN and F LSTM are the feature extraction functions of the convolutional neural network and the long short-term memory network, A is the audio feature, H is the head motion feature, GAP is the global average pooling operation, and W and b are the weight and bias of the fully connected layer, respectively. This prediction method can effectively integrate audio features and head motion features to improve the accuracy of scene recognition.

[0147] In one embodiment of the present invention, in order to further improve the performance of the model, a negative sample enhancement strategy is adopted. Specifically, the negative samples are resampled and enhanced, and the positive samples are noise enhanced. The calculation formula of the enhancement ratio is as follows:

[0148]

[0149] Where: R is the enhancement ratio, N pis the number of positive samples after enhancement, N p is the number of original positive samples, N n is the number of negative samples, and α is a hyperparameter. Preferably, the value range of α is between 0.1 and 0.5, because it is found in practice that the value of α in this range can achieve a good balance between enhancing sample diversity and maintaining the original data distribution.

[0150] Next, the trained network parameters are obtained according to the loss function and the training parameters are updated. The parameter update formula is:

[0151] Among them, θ′ is the updated parameter, θ is the current parameter, η is the learning rate, is the gradient of the current parameter, f θ is the neural network function, x is the input data, and y is the true label. It is worth noting that the choice of learning rate ηη has an important impact on the convergence speed and performance of the model. In the present invention, we adopt an adaptive learning rate strategy, with the initial value set to 0.001, and then dynamically adjusted according to the loss changes during training.

[0152] The data set is input into the convolutional neural network feature extractor, multi-head attention and self-attention to learn the weight matrix and parameters. This step is the core of model training, and the parameters of each network component are continuously optimized through the back-propagation algorithm.

[0153] Finally, the joint representation is learned using the objective function and the objective function is maximized. The specific form of the objective function is:

[0154]

[0155] Among them, p pos represents the distribution of positive sample pairs, sim is the similarity function, and f θ is the encoder function, and τ is the temperature parameter. In practice, we found that setting the temperature parameter τ to 0.07 can achieve better results. This is because a smaller τ value can make the model pay more attention to difficult samples, thereby improving classification performance.

[0156] The present invention provides a TWS headset audio scene recognition method based on head tracking, which has several significant advantages. First, the method makes full use of the characteristics of TWS headsets, collects audio data and head motion data at the same time, and provides rich input information for scene recognition. For example, when a user walks, the head motion data can reflect the regular swinging pattern, and combined with the ambient audio, the "outdoor walking" scene can be more accurately identified.

[0157] Secondly, this method uses advanced audio feature extraction technologies, including wavelet transform, Hilbert-Huang transform and Mel spectrum analysis. These technologies can fully capture the various characteristics of audio signals. For example, when identifying the "cafe" scene, Mel spectrum analysis can effectively capture the characteristics of human voice and background music, while wavelet transform can identify instantaneous sounds such as cups and plates colliding, thereby improving the accuracy of recognition.

[0158] Thirdly, the present invention innovatively introduces a two-step attention mechanism, including spatial attention and multi-head attention. This mechanism can effectively extract key information from head motion data. For example, when identifying the "gym" scene, spatial attention can capture the violent movement of the user's head, while multi-head attention can integrate motion features of different time scales, thereby more accurately distinguishing the "gym" from other sports scenes.

[0159] In addition, this method adopts a hybrid model architecture that combines convolutional neural networks with long short-term memory networks, and introduces a contrastive learning algorithm. This design can simultaneously utilize the advantages of CNN in spatial feature extraction and LSTM in temporal modeling, and achieve effective fusion of audio and head motion data. For example, when identifying the "watching a movie" scene, CNN can extract the features of movie audio, while LSTM can capture the user's relatively still head posture for a long time. The combination of the two can more accurately identify this scene.

[0160] Finally, the present invention adopts the strategy of negative sample enhancement and positive sample noise enhancement, which can improve the generalization ability of the model. For example, when training to recognize the "subway" scene, by adding different degrees of background noise to the positive samples, the model can better adapt to subway environments with different noise levels.

[0161] In general, the TWS headset audio scene recognition method based on head tracking of the present invention realizes efficient and accurate audio scene recognition through many innovations. This method can not only improve the intelligence level of TWS headsets, but also provide users with a more personalized audio experience. For example, when the "office" scene is identified, the headset can automatically switch to noise reduction mode; when the "outdoor running" scene is identified, the audio equalizer can be adjusted to enhance the sense of rhythm. The realization of these functions will greatly enhance the user experience and open up new application prospects for TWS headsets. To verify the superiority of the present invention, we conducted a series of experiments to compare the embodiments of the present invention with the comparative examples of the traditional method. The following is a detailed experimental design, result analysis and discussion.

[0162] Example 1: TWS headset audio scene recognition method based on head tracking using the present invention

[0163] In this embodiment, we fully implemented the method of the present invention, including the collection of audio data and head motion data, multi-dimensional audio feature extraction, innovative head information weight calculation, combined neural network model architecture and contrastive learning algorithm. We used a common TWS headset on the market, in which we integrated the software module developed. The test data set contains 10 common audio scenes, such as office, cafe, subway, outdoor walking, gym, etc., and 1000 samples were collected for each scene.

[0164] Comparative Example 1: Traditional scene recognition method based only on audio features

[0165] In the comparative example, we implemented a traditional scene recognition method based only on audio features. This method uses Mel-Frequency Cepstral Coefficients (MFCC) as audio features and adopts Support Vector Machine (SVM) as classifier. This method has been widely used in past studies and can serve as a good benchmark to evaluate our new method.

[0166] Testing standards and methods:

[0167] 1. Recognition accuracy: The ten-fold cross-validation method is used to evaluate the average recognition accuracy of the model on the test set.

[0168] 2. Computational efficiency: record the average time required to process a 10-second audio clip.

[0169] 3. Battery consumption: Measures the battery consumption percentage of the TWS headset per hour when scene recognition is continuously performed.

[0170] 4. User experience rating: Invite 50 volunteers to use the two methods and rate the accuracy and timeliness of their scene recognition on a scale of 1-10.

[0171] The following is a detailed table of test results:

[0172] Test indicators Example 1 Scale 1 Improvement ratio Recognition accuracy 94.5％ 82.3％ 12.20％ Computational efficiency 0.8 sec / sample 1.5 seconds / sample -0.70 sec / sample Battery consumption 5.2% / hour 7.8% / hour N / A User experience rating 8.7 / 10 6.5 / 10 2.20 / 10

[0173] It is clear from the test results that the method of the present invention is significantly superior to the traditional method in all aspects. First, in terms of recognition accuracy, the method of the present invention reached 94.5%, 12.2 percentage points higher than the traditional method. This significant improvement is mainly due to our innovative multimodal data fusion method, especially the introduction of head motion data, which provides additional contextual information for scene recognition.

[0174] In terms of computational efficiency, although the method of the present invention involves more complex calculations, through optimized algorithm design, we have successfully controlled the processing time to 0.8 seconds per sample, which is nearly twice as fast as the traditional method. This means that our method can respond to environmental changes more quickly and provide more timely scene recognition results.

[0175] Battery consumption is a very important indicator for TWS headset users. Surprisingly, although our method introduces additional data collection and processing, through efficient algorithm design and hardware optimization, we have successfully controlled the battery consumption per hour to 5.2%, which is 2.6 percentage points less than the traditional method. This means that users can enjoy smarter scene recognition functions without significantly affecting battery life.

[0176] Finally, the user experience score directly reflects the performance of the method in actual use. The method of the present invention received a high score of 8.7, far exceeding the 6.5 of the traditional method. Many users said that the scene recognition of the new method is more accurate and timely, and can better adapt to their daily use scenarios.

[0177] These test results fully demonstrate the superiority of the method of the present invention. Through innovative multimodal data fusion, advanced feature extraction technology and optimized neural network architecture, we have successfully improved the recognition accuracy while reducing computational complexity and energy consumption. This not only improves the intelligence level of TWS headphones, but also provides users with a more personalized and smooth experience.

[0178] It is worth noting that in actual applications, we found that the method of the present invention performs particularly well in some specific scenarios. For example, when distinguishing between the two easily confused scenes of "subway" and "bus", the accuracy of the traditional method is only about 75%, while the method of the present invention achieves a high accuracy of 92%. This is mainly because our method can use head motion data to capture the different motion patterns of subways and buses, thereby better distinguishing these two scenes.

[0179] In general, these test results not only verify the technical superiority of the method of the present invention, but also demonstrate its great potential in practical applications. With the continuous development of the TWS headset market, we believe that this head tracking-based audio scene recognition method will bring users a smarter and more personalized listening experience, and promote the development of TWS headsets towards a higher level of intelligence.

[0180] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A TWS headset audio scene recognition method based on head tracking, characterized in that: The specific steps are as follows: S1. Collect audio data: insert a pre-processing module into the head tracking function of the TWS headset, and use the head tracking module to collect audio data and motion data of the left and right earphones; S2. Extract features from audio data: extract the fundamental frequency and spectrogram of the sound signal using the wavelet transform method, resample the audio data and extract the fundamental frequency, use the Hilbert-Huang transform to estimate the instantaneous amplitude spectrum, instantaneous frequency spectrum and Hilbert spectrum, and use the Mel frequency filter bank with logarithmic weighted average to extract the Mel spectrum to obtain the feature map of the audio signal; S3. Calculate the weight of the head information: Perform spatial attention weighting on the head motion data, and use multi-head attention operation to obtain the weight of the head information. The calculation formula of spatial attention weighting is: W s =softmax(f s (H i ,H j )) Among them, W s is the spatial attention weight, f s is the spatial attention mapping function, H i and H j are the i-th and j-th head motion data respectively; S4. Construct an audio scene classification model: Use a combined model architecture of a convolutional neural network and a long short-term memory network, adopt a contrastive learning algorithm, take the feature map obtained in S2 and the weight value obtained in S3 as input, and output the recognition result of the audio scene. The loss function of contrastive learning is: Among them, z i and z j is the feature representation of the positive sample pair, z k is the feature representation of negative samples, sim is the cosine similarity function, and τ is the temperature parameter; S5. Identify the audio scene: According to the obtained audio scene classification model, input the real-time data and identify the audio scene.

2. According to a TWS headset audio scene recognition method based on head tracking according to claim 1, it is characterized in that: In S1, collecting audio data and motion data of the left and right earphones includes: inserting a preprocessing module into the head tracking function, using the head tracking module to obtain head motion data in real time, and combining the data acquisition module to obtain multi-microphone audio data of the left and right earphones, wherein the acquisition formula of the head motion data is: H=[a x ,a y ,a z ,oh x ,oh y ,oh z ,m x ,m y ,m z ], Among them, a x ,a u ,a z is the accelerometer data, ω x ,ω y ,ω z is the gyroscope data, m x ,m y ,m z The magnetometer data.

3. According to a TWS headset audio scene recognition method based on head tracking according to claim 1, it is characterized in that: In S2, extracting features from audio data includes: extracting the fundamental frequency and spectrogram of the sound signal using a wavelet transform method, and the specific steps are as follows: For a given audio sample, the wavelet basis function is constructed according to the basic formula of wavelet transform: In the formula, a is the multi-sampling frequency, b is the sampling frequency, t is the time, f is the frequency, t k is the kth moment in the sample, N is the number of sampling points, ψ a , b(t) represents the kth wavelet basis function Spectrum of samples; Next, the audio signal is demodulated using wavelet transform, and the filter coefficients are expressed as: Among them, x(t) is the audio signal, ψ a ,b * (t) is the conjugate of the wavelet basis function; Finally, the fundamental frequency is extracted using the following formula: f0=argmax f |W f (a,b)|, Where f0 represents the fundamental frequency of the audio signal, |W f (a,b)| represents the modulus of the wavelet coefficients.

4. According to a TWS headset audio scene recognition method based on head tracking according to claim 1, it is characterized in that: In S2, extracting features from the audio data further includes: using Hilbert-Huang transform to estimate the instantaneous amplitude spectrum, the instantaneous frequency spectrum and the Hilbert spectrum; For a given audio signal, the empirical mode decomposition method is used to decompose the original signal into a finite number of intrinsic mode functions, namely: x(t)=∑ i c i (t)+r n (t), Among them, c i (t) is the i-th eigenmode function, r n (t) is the residual term; Then the intrinsic mode function is converted into an analytical signal through Hilbert transform, and the instantaneous frequency is obtained: Where: f i (t) represents the instantaneous frequency of the signal in the i-th sample, θ i (t) represents the phase angle of the signal, i represents the analytical signal of the i-th eigenmode function; Finally, a Mel-frequency filter bank with log-weighted averaging is used to extract the Mel-spectrum: M(m)=∑ k H m (k)·|X(k)| 2 , Where: M(m) is the Mel spectrum, X(k) represents the spectrum of the input signal, H m (k) represents the Mel frequency filter of logarithmic weighted average, k represents the dimension of DFT, and m represents the index of Mel filter.

5. A TWS headset audio scene recognition method based on head tracking according to claim 4, characterized in that: In S3, weight calculation of head information includes: preprocessing the head motion data to unify its units; for the preprocessed head motion data, firstly using spatial attention operation to obtain weight coefficients of each orientation, and then using multi-head attention to perform weight integration to obtain spatial weighted features. The specific steps are as follows: For head motion data, spatial attention is used to obtain the weight coefficients of each orientation: α i =softmax(W s ·tanh(W h ·H i +b h )), Where: H i represents the original head motion data, W s is the weight matrix of spatial attention, W h is the hidden layer weight matrix, b h is the bias term, α i Represents the attention weight of the i-th position; then use multi-head attention to integrate the head information by weight: MultiHead(Q,K,V)=Concat(head1,...,head h )·W O , Where: Q, K, V are query, key and value matrices respectively, is the learnable parameter matrix.

6. A TWS headset audio scene recognition method based on head tracking according to claim 5, characterized in that: In S4, constructing the audio scene classification model includes: using a combined model architecture of a convolutional neural network and a long short-term memory network to construct the audio scene classification model, including: S41. Perform convolutional neural network feature extraction on Mel cepstral coefficients and spectrograms; S42. Input the spatial weighted features and perform serialization in combination with the long short-term memory network; S43. Use context attention to assign different weights to different positions to obtain a global context feature vector; First, the i-th context feature is obtained through a linear layer: h i =W h ·x i +b h , Where: x i represents the spatial weighted feature obtained by the i-th multi-head attention after serialization, W h is the learnable projection matrix, b h is the bias term, and then, multi-head attention is used to linearly transform the context features: i =MultiHead(Q i ,K i ,V i ), where: Q i ,K i ,V i are the query, key, and value matrices of the i-th context feature respectively; then, the context attention weight is used to calculate the weight value of the context feature: β i =softmax(v T tanh(W c ·z i +b c )), where: v, W c ,b c is a learnable parameter; finally, the global context vector is input into the multi-layer neural network and the result is obtained: y = MLP (Σ i β i ·z i ), where: y is the global context vector, MLP is a multi-layer perceptron; S44. Add a multi-layer perceptron and a global average pooling layer after the combined model architecture to output the predicted category of the audio sample.

7. A TWS headset audio scene recognition method based on head tracking according to claim 1, characterized in that: In S4, constructing the audio scene classification model further includes: using a contrastive learning algorithm: using an objective function to learn to enhance the association between similar data samples and weaken the association between data samples of different categories, and the objective function is as follows: Where: x i ,x j represents a positive sample pair, x k represents negative samples, f(·) represents the encoder function, sin(·,·) represents the cosine similarity function, and τ is the temperature parameter.

8. The method for TWS headset audio scene recognition based on head tracking according to claim 2, characterized in that: In S1, the data acquisition module acquires multi-microphone audio data of the left and right earphones, including: For a microphone array, the audio data from each microphone is recorded and expressed as: A i (t) = s(t)·h i (t)+n i (t), where i is the serial number of the microphone, A i (t) represents the audio data collected by the i-th microphone, t is the time axis, s(t) is the source signal, h i (t) is the impulse response of the ith microphone n i (t) is the noise; for the head tracking module, a gyroscope and a magnetometer are used to record the motion data of the head in all directions, which can be expressed as: H j =[a x ,a y ,a z ,ω x ,ω y ,ω z ,m x ,m y ,m z ] j , where: j is the sequence number of the head motion data, H j represents the jth sample data, a, ω, m represent the three-dimensional components of acceleration, angular velocity and magnetic field intensity respectively; then, combine the audio data with the head motion data: D = {A i (t),H j |i=1,...,M;j=1,...,N}, where: D represents the data set, M is the number of microphones, and N is the number of sampling points of head motion data.

9. The method for TWS headset audio scene recognition based on head tracking according to claim 1, characterized in that: In S5, the audio scene recognition includes: recognizing the real-time collected data, first obtaining the audio features through the convolutional neural network feature extractor, and then obtaining the spatial weight coefficient based on the features of the head motion data; then, using the context attention function described in S43 to assign weight values ​​to the head information, and inputting the global context vector into the multi-layer neural network, and using the global average pooling layer described in S44 to obtain the prediction result, wherein the calculation formula of the prediction result is: y pred =softmax(W·GAP(F CNN (A)+F LSTM (H))+b), Where:y pred is the prediction result, F CNN and F LSTM are the feature extraction functions of convolutional neural networks and long short-term memory networks, A is the audio feature, H is the head motion feature, GAP is the global average pooling operation, W and b are the weight and bias of the fully connected layer, respectively.

10. A TWS headset audio scene recognition method based on head tracking according to claim 7 or 8, characterized in that: In the objective function, a contrastive learning algorithm is used, a negative sample enhancement strategy is adopted, negative samples are resampled and enhanced, and positive samples are noise enhanced. The enhancement ratio is as follows: Where: R is the enhancement ratio, N' p is the number of positive samples after enhancement, N p is the number of original positive samples, N n is the number of negative samples, α is a hyperparameter; Next, obtain the trained network parameters according to the loss function and update the training parameters: Among them, θ′ is the updated parameter, θ is the current parameter, η is the learning rate, is the gradient with respect to the current parameter, f θ is the neural network function, x is the input data, and y is the true label; Input the dataset into the convolutional neural network feature extractor, multi-head attention and self-attention, learn the weight matrix and parameters, use the objective function to learn the joint representation, and maximize the objective function: Among them, p pos represents the distribution of positive sample pairs, sim is the similarity function, and f θ is the encoder function and τ is the temperature parameter.

Images