A Speaker Recognition and Emotion Perception Method Based on AR Glasses

Abstract

This invention discloses a speaker recognition and emotion perception method based on AR glasses. The system combines microphone arrays, cloud computing, and multi-model fusion technology to achieve real-time recognition and display of speaker identity information, speech content, and emotional state. The method first acquires multi-channel speech signals through a microphone array and improves speech quality using preprocessing techniques such as beamforming, noise reduction, speech enhancement, and voice endpoint detection (VAD). Subsequently, the system runs an ECAPA-TDNN voiceprint recognition model in the cloud to obtain speaker embedding vectors. Through acoustic features such as MFCC, Fbank, and LPCC, and an AAM-Softmax training mechanism, it identifies gender, age, and speaker identity. A CTC end-to-end speech recognition model is used to complete real-time speech transcription. Finally, an emotion recognition model based on Wav2Vec 2.0 and Transformer, combined with MFCC features, emotion embedding, and attention-weighted fusion strategies, achieves high-precision recognition of multiple emotions such as "happy, angry, sad, and calm."

Description

Technical Field

[0001] This invention relates to the field of augmented reality (AR) technology, specifically to a method for use in AR glasses to display a speaker's basic information (such as gender, age, etc.) and emotional state in real time, and to display them together with the content of the speech. Background Technology

[0002] With the rapid development of augmented reality (AR) technology, AR glasses are gradually becoming an important medium for human-computer interaction. Voice interaction, as a natural and convenient interaction method, has broad application prospects in AR glasses. Currently, some AR glasses support speech-to-text functionality, which can convert the user's speech into text in real time and display it on the glasses for easy viewing and recording. However, existing AR glasses speech-to-text technology still has some limitations:

[0003] Currently, there are still many technical shortcomings, such as: The inability to display speaker identity and emotions: Traditional AR glasses mainly display visual information, lacking real-time recognition and display of speaker identity and emotional state, and the accuracy of recognition needs improvement: In noisy environments, the accuracy of speech recognition drops significantly, leading to text display errors. Lack of integration of speaker identity and emotions: Existing technology cannot combine emotional state with the speaker's basic information for display.

[0004] To address the aforementioned issues, it is necessary to explore the application of identity recognition and emotion recognition technologies to AR glasses speech-to-text systems. This is a technique that identifies the speaker by analyzing feature parameters in the speech signal. Compared to traditional speech recognition technology, this technology offers the following advantages: it provides a method capable of identifying the speaker's basic information (such as gender and age) and emotional state in real time, and displaying this information along with the spoken content in the AR glasses, thereby enhancing the user experience. Summary of the Invention

[0005] A method for use in AR glasses is provided, which can identify the speaker's basic information (such as gender and age) and emotional state in real time and display them together with the content of the speech.

[0006] To address the aforementioned technical problems, this invention provides a method for speaker information and emotion recognition applied to AR glasses, comprising the following steps:

[0007] This system uses AR glasses as the core terminal and combines multiple advanced software and hardware technologies to build an intelligent voice interaction system capable of identity recognition, voice recognition, and emotion recognition.

[0008] In terms of hardware, the AR glasses incorporate several key modules to support the system's functionality. First, a microphone array accurately captures the user's voice input, ensuring accurate acquisition even in multi-source sound environments. Second, a cloud server provides powerful computing resources to run complex deep learning models for tasks such as speech recognition and emotion recognition. Simultaneously, the AR glasses are equipped with ample memory and storage to support model operation and data caching. A display module projects real-time speech recognition and translation results into the user's field of vision. A battery module provides power for the entire device's normal operation. Furthermore, a wireless communication module supports convenient interaction and remote communication between the user and the device, enhancing the overall user experience.

[0009] In terms of software, the system consists of three core modules: identity information recognition module, voice content recognition module, and emotion recognition module.

[0010] In the identity information recognition module, the speech signal is first preprocessed, including noise reduction using a deep learning model, endpoint detection using a dual-threshold method combining energy and zero-crossing rate or a speech activity detection (VAD) model, pre-emphasis to boost high-frequency components, and frame segmentation and windowing (commonly Hamming or Heining windows) using a 25ms frame length and 10ms frame shift. In the feature extraction stage, voiceprint features such as MFCC (Mel frequency cepstral coefficients), Fbank (Mel filter bank energy), and LPCC (linear predictive cepstral coefficients) are extracted. For the model, the advanced ECAPA-TDNN voiceprint recognition model is used, whose structure includes a TDNN layer, an SE attention mechanism, multi-layer feature aggregation, and a global context module, outputting a fixed-length voiceprint embedding vector. This model is trained using the AAM-Softmax loss function and is primarily based on the VoxCeleb1 and VoxCeleb2 datasets. In practical applications, speaker verification can be performed by calculating the cosine similarity between two embedding vectors, or speaker recognition can be performed by comparing with a known voiceprint database.

[0011] The speech content recognition module and the speaker recognition module share the same preprocessing procedure for the speech signal. For feature extraction, MFCC and Fbank are still used as the main input features. The recognition model employs an end-to-end CTC-based speech recognition model, directly converting the input acoustic features into text. This model is trained using a cross-entropy loss function, with training data primarily derived from large speech datasets such as LibriSpeech and Common Voice, achieving high-accuracy speech-to-text transcription.

[0012] In the emotion recognition module, speech preprocessing still includes noise reduction and enhancement (such as Wiener filtering, adaptive noise suppression (ANC), or Log-Mel spectral enhancement), removal of silence segments through speech endpoint detection, and standardization of the audio format (16kHz sampling rate, single-channel PCM) to adapt to mainstream pre-trained models such as Wav2Vec 2.0 or HuBERT. Wav2Vec 2.0, through self-supervised learning, is pre-trained on unlabeled speech data and can be fine-tuned for emotion classification. This model extracts low-level waveform features using CNNs and then utilizes Transformers to learn more complex temporal and contextual relationships in speech. Finally, the extracted contextual sequence features are input into an emotion classification model combining a Transformer encoder and a multilayer perceptron (MLP). The Transformer's self-attention mechanism can more accurately focus on key emotion-related information, thereby achieving efficient and accurate emotion recognition.

[0013] This invention utilizes Wav2Vec2.0 + Transformer for high-precision speech emotion recognition, and combines cloud computing with local caching to optimize processing speed, achieving efficient and accurate display of speaker information and emotions. This method improves the accuracy of speech emotion recognition, enhances the intelligent interaction capabilities of AR glasses, and allows users to perceive the emotional state of the speaker in real time, thus improving the communication experience. It is applicable to scenarios such as intelligent meetings, social assistance, and medical communication. A specific illustration is shown below. Figure 3 As shown. Attached Figure Description

[0014] Figure 1 This is an abstract with attached figures.

[0015] Figure 2 It is a system flowchart.

[0016] Figure 3 It is a schematic diagram. Detailed Implementation

[0017] The present invention provides a more detailed description of a speaker recognition and emotion perception method based on AR glasses, in conjunction with the accompanying drawings and specific examples. The advantages and features of the present invention will become clearer from the following description.

[0018] This system integrates a microphone array, a voiceprint recognition module (based on the ECAPA-TDNN model), a language identity recognition module (based on an attention-based model), an emotion recognition module (based on Wav2Vec 2.0), and an AR display module. It can identify the voiceprint of a specific speaker in real time, enrich speaker information and emotions, and present this information to the user through the AR display module. The system is suitable for various scenarios such as meetings, education, and communication, aiming to provide efficient and accurate speech-to-text services.

[0019] The system's hardware comprises a microphone array, a cloud server, an AR display unit, a wireless communication module, and a power management system. The microphone array consists of multiple miniature microphones evenly distributed across the frame of the AR glasses to collect ambient sound signals. Utilizing beamforming technology, the system enhances speech from specific directions while reducing environmental noise interference. Unlike traditional local computing solutions, this system offloads voiceprint recognition and speech processing tasks to the cloud, with the AR glasses primarily handling sound acquisition, data transmission, and result presentation, thereby enhancing real-time processing capabilities. The AR display module, composed of a miniature display and an optical lens system, seamlessly overlays text information onto the user's field of vision, providing an intuitive augmented reality experience. The wireless communication module supports Bluetooth and Wi-Fi for data interaction with external devices or the cloud server. The power management system employs a low-power optimized design to extend the device's battery life and improve overall efficiency.

[0020] AR glasses have a built-in microphone array to collect sound signals. The microphone array consists of 4 to 8 microphones, with a sampling rate of 16 to 48 kHz and a bit depth of 16 bits.

[0021] Preprocessing process

[0022] For noise reduction, traditional Wiener filtering or deep learning noise reduction models (DNN, LSTM, CNN, etc.) are used. The number of parameters in the noise reduction model is 0.5 to 5 M, which is used to improve the audio signal-to-noise ratio in noisy environments.

[0023] Endpoint detection (VAD) uses a dual-threshold method based on short-time energy and zero-crossing rate, or a deep learning VAD model, with the following parameters: frame length: 20–30 ms; frame shift: 8–12 ms.

[0024] Beamforming and speech enhancement utilize microphone arrays for beamforming (delay summation or MVDR algorithm) to enhance sound in the target direction and suppress background noise.

[0025] In the software component, the entire system comprises several key modules. First, to improve the quality of voice data, the system employs noise reduction and Voice Activity Detection (VAD) techniques to reduce the impact of background noise and ensure that only valid speech segments are captured, thereby improving the accuracy of speech recognition. The system first uses a microphone array to collect ambient sound and employs beamforming technology to enhance the speech signal in the target direction while suppressing noise from other directions. By calculating the time difference of different microphone signals and adjusting the weights, the system effectively reduces background interference and improves speech clarity. Furthermore, the system combines deep learning noise reduction algorithms (such as CNN or LSTM) to model and suppress noise, further improving the speech signal-to-noise ratio (SNR). Endpoint detection is used to identify the start and end points of speech to avoid recording invalid silences or background noise. The system uses short-time energy (STE) and zero-crossing rate (ZCR) to calculate speech activity and further optimizes it using Mel-frequency cepstral coefficients (MFCC). By combining noise reduction and endpoint detection, the system can accurately extract speech in noisy environments, improve the accuracy of speaker recognition and emotion analysis, and optimize the real-time interactive experience of AR glasses.

[0026] The cloud server runs the ECAPA-TDNN model for voiceprint recognition. First, the speech signal is preprocessed to extract Mel-frequency cepstral coefficients (MFCCs) and filter bank features (FBank). Specifically, the ECAPA-TDNN model then extracts voiceprint features. The ECAPA-TDNN model combines a multi-layer time-delay neural network (TDNN), channel attention mechanism, and feature aggregation layer, enabling efficient extraction of high-dimensional voiceprint feature vectors. In the voiceprint recognition task, the ECAPA-TDNN model uses AAM-Softmax (Additive Angular Margin Softmax) as the loss function.

[0027] Specifically, in the acoustic feature extraction stage, the system extracts various feature parameters from the preprocessed speech signal to represent the speech spectrum and timbre characteristics, including MFCC, Fbank, and LPCC features. The MFCC feature uses 13-dimensional static coefficients, and its first and second differences are further calculated, forming a total 39-dimensional MFCC feature vector. To ensure the expressive power of speech spectrum details, 26 or 40 Mel filters are used in the MFCC extraction process, with the FFT point count set to 512–1024 points. In addition to MFCC, the system also extracts 40–80-dimensional Fbank features to obtain energy distribution information of speech at the Mel scale, thus supplementing the shortcomings of MFCC in terms of speech dynamic changes. Furthermore, the system can optionally extract 12–20-dimensional LPCC (Linear Predictive Cepstral Coefficients) features to enhance the system's ability to model the speaker's oral cavity structure and vocal tract features. These multiple sets of features can be used individually or in combination in different model modules to improve the overall accuracy and robustness of voiceprint recognition, speech recognition, and emotion recognition tasks.

[0028] The speech signal collected by the microphone is preprocessed by a module that includes noise reduction, endpoint detection, and speech enhancement, and then input into a CTC-based speech recognition model for identity determination.

[0029] The CTC-based speech recognition model employs end-to-end modeling, directly mapping the input speech waveform to corresponding text information and extracting speaker identity features. Compared to traditional HMM-DNN speech recognition methods, CTC is trained using alignment-free training, not relying on precise frame-level labels, and can better adapt to real-time speech input. Specifically, given an input speech feature sequence... The CTC model calculates the output probability distribution P(Y|X), where Y is the target speaker's identity label. Its loss function is as follows:

[0030]

[0031] Here, P(Y|X) is obtained by summing all possible alignment paths in the time dimension, and dynamic programming is used for efficient computation. To improve recognition accuracy, this scheme combines the CTC model with a speaker embedding extraction model (such as ECAPA-TDNN) to further optimize identity discrimination capabilities. Specifically, based on the text features extracted by the CTC speech recognition model, a speaker feature extraction network is additionally used to calculate the speaker embedding E:

[0032]

[0033] By adopting a joint decision-making mechanism, we can improve accuracy, enhance robustness, reduce misjudgments, adapt to noisy environments, and optimize the real-time interactive experience.

[0034] The system uses the Wav2Vec 2.0 + Transformer model to analyze the emotional state of speech. Wav2Vec 2.0 directly extracts high-dimensional speech representations from the original waveform and learns global speech features through the Transformer network to classify different emotion categories, such as "happy, angry, sad, calm," etc. To improve the accuracy of emotion classification, the model combines features such as speech rate, pitch, and energy, and employs feature enhancement techniques and multimodal feature fusion to improve the model's generalization ability.

[0035] In speech emotion recognition systems, the fusion of audio and emotional features is a key step in improving model performance. While Wav2Vec 2.0, as a powerful speech representation model, can capture deep semantic features, it still has limitations in handling certain acoustic details. Therefore, the system introduces MFCC (Mel-frequency cepstral coefficients) features as an important supplement. This traditional acoustic feature can more accurately characterize the spectral properties of speech signals. MFCC transforms the speech signal to the Mel-frequency scale by simulating the nonlinear characteristics of the human auditory system, and then obtains cepstral coefficients through discrete cosine transform. This process is particularly suitable for extracting features closely related to emotional expression, such as timbre and prosody. Specifically, static MFCC coefficients reflect the spectral envelope of the speech signal, while first- and second-order difference coefficients capture the dynamic changes in fundamental frequency and energy. These features have unique advantages in recognizing the sharp tone of anger or the low tone of sadness.

[0036] In the specific implementation of feature fusion, the system adopts a weighted fusion strategy based on an attention mechanism. First, the 768-dimensional feature vector extracted by Wav2Vec 2.0 is dimensionality-reduced to better match the 39-dimensional MFCC features (including 13 static coefficients and their first and second-order differences). Considering the different contributions of various features to the final emotion recognition, the system designs learnable weight parameters and normalizes the weight allocation through a softmax function. This dynamic weighting mechanism allows the model to automatically adjust the degree of attention given to different features. For example, when recognizing surprise, it may rely more on the high-frequency energy surge features captured by MFCC, while when analyzing neutral tone, it focuses more on the semantic context information extracted by Wav2Vec. The feature fusion layer also introduces a residual connection structure to ensure that the original feature information is not lost during the fusion process, while mitigating the gradient vanishing problem.

[0037] In the emotion recognition process, the system employs the Wav2Vec 2.0 model to extract high-dimensional contextual representation features from the original waveform of the input speech. The Base version contains a 12-layer Transformer encoder with a hidden unit dimension of 768, while the Large version contains a 24-layer Transformer encoder with a hidden unit dimension of 1024, allowing for flexible selection under different computational resource conditions. To enhance the ability to capture acoustic details related to speech emotion, the system further extracts 39-dimensional MFCC features and pre-trained emotion embedding features, and uses a weighted fusion strategy based on an attention mechanism to uniformly model the multi-source features. Weighted Fusion: A weighted fusion strategy is used to combine features from Wav2Vec 2.0, MFCC features, and emotion tag embeddings. By assigning different weights to each feature, the model can focus more on the features most useful for emotion recognition. The calculation method is shown in the following formula.

[0038]

[0039] in, , , ∈[0,1] and satisfy The weights are trainable parameters that can be automatically adjusted based on the importance of different emotional features to the target classification task, enabling the model to dynamically focus on the most effective feature sources under different speech conditions. The fused features are input into an emotion classification network for inference. This network consists of 2-6 Transformer encoder layers and 1-3 MLP classification layers, outputting recognition results for 4-8 emotion categories, including happy, angry, sad, calm, neutral, and surprised. Experiments on standard emotional speech data show that the model achieves an emotion recognition accuracy of 85-92% in clean speech environments and maintains 75-85% recognition performance under noisy conditions, significantly improving the robustness and reliability of the system in complex real-world scenarios.

[0040] By adjusting the loss function and optimization strategy, the cross-entropy loss function is a commonly used optimization method in emotion recognition. If class imbalance exists in the data, weighted cross-entropy loss can be used, which assigns different weights to different emotion categories to address the class imbalance problem. Focal loss is used to address class imbalance. This method helps the model focus more on samples that are difficult to classify, thereby improving overall accuracy. The formula for focal loss is shown below.

[0041]

[0042] Where pt is the probability of correctly predicting the class, αt is the class weight coefficient, and γ is the weight parameter for adjusting the hard-to-classify samples.

[0043] The final model is able to better capture emotion-related multimodal information, thereby improving the accuracy of emotion recognition.

Claims

1. A speaker recognition and emotion perception method based on AR glasses, characterized in that, include: Voice acquisition steps: A microphone array consisting of 4 to 8 microphones arranged on the AR glasses frame is used to acquire ambient voice signals at a sampling rate of 16 to 48 kHz; and a beamforming algorithm based on array time difference estimation is used to enhance the voice in a specific direction to improve the target voice signal-to-noise ratio.

2. Speech preprocessing steps: A multi-level preprocessing workflow, including noise reduction, endpoint detection (VAD), and speech enhancement, is performed on the acquired speech, where: —The noise reduction adopts a combined noise reduction structure of Wiener filtering and deep learning noise reduction network; —Endpoint detection employs a multi-parameter dual-threshold decision model or deep VAD that fuses three features: short-time energy, zero-crossing rate, and spectral entropy; —Speech enhancement employs a joint strategy of MVDR beamforming and noise adaptive modeling; thereby obtaining effective speech with high signal-to-noise ratio and preservation of emotional details.

3. Multi-source acoustic feature extraction steps: Simultaneously extract MFCC (including first-order and second-order differences, totaling 39 dimensions), 40-80 dimensional Fbank, and 12-20 dimensional LPCC features from the preprocessed speech; among which, 26 or 40 Mel filters and 512-1024-point FFT are used for MFCC extraction to form a multimodal acoustic feature set for speaker recognition, speech recognition, and emotion recognition.

4. Speaker identification steps: Input the features into the ECAPA-TDNN voiceprint recognition model, which includes a TDNN layer, an SE channel attention layer, and a multi-layer feature aggregation structure, to generate a fixed-length voiceprint embedding; train the model using the AAM-Softmax loss function; and complete speaker identification by comparing the embedding vector with the user's voiceprint database or by using cosine similarity.

5. Speech content recognition steps: Input MFCC and Fbank features into the CTC-based speech recognition model and output the text content in an unaligned manner; combine CTC language features with speaker embedding to help improve the robustness of speaker discrimination.

6. Emotion Recognition Steps: The raw speech is input into the Wav2Vec 2.0 model to extract high-dimensional contextual speech representations, and simultaneously MFCC features and emotion label embeddings are obtained; emotion fusion features are generated through an attention-weighted fusion module with trainable weights, where: ,and ; The fusion feature input consists of a classification network composed of 2-6 layers of Transformer encoder and 1-3 layers of MLP, and the output is the emotion category.

7. AR Display Steps: The identified speaker's identity information (including gender, age, identity, or user-defined information), speech-to-text, and emotional state are overlaid on the AR glasses display interface in real time, and low-latency feedback is provided based on the cloud model output.

8. The innovation of this method lies in the fact that it integrates beamforming enhancement, high-redundancy acoustic feature set, ECAPA-TDNN voiceprint recognition, CTC speech recognition and Wav2Vec2.0-Transformer emotion recognition into a unified processing chain through a trainable fusion mechanism, so as to realize the synchronous real-time recognition of speaker identity, content and emotion, and optimize the AR display quality.

9. The method according to claim 1, characterized in that, The noise reduction step employs a parallel structure of traditional Wiener filtering and deep neural networks (DNN, LSTM, or CNN), dynamically selecting weighted outputs based on noise characteristics to improve adaptability to noisy environments.

10. The method according to claim 1, characterized in that, Endpoint detection employs a multi-feature joint threshold strategy using short-time energy (STE), zero-crossing rate (ZCR), and spectral entropy (SE), which reduces the false detection rate of silent segments by at least 20%.

11. The method according to claim 1, characterized in that, The multi-layer feature aggregation structure in the ECAPA-TDNN voiceprint model achieves the superposition of voiceprint information at different semantic levels through residual connections, thereby improving the generalization ability across speaking scenarios.

12. The method according to claim 1, characterized in that, The hidden features output by CTC are combined with the voiceprint embedding vector through concatenation or element-wise weighting to form joint features, which are used to enhance the robustness of speaker recognition in fast speech or weak voice scenarios.

13. The method according to claim 1, characterized in that, The Wav2Vec 2.0 model is either the Base or Large version, which includes 12 and 24 layers of Transformer encoders, respectively.

14. The method according to claim 1, characterized in that, The attention-weighted fusion module uses a multi-head attention mechanism to dynamically adjust the importance of different acoustic features.

15. The method according to claim 1, characterized in that, Emotion recognition and classification networks employ weighted cross-entropy or focus loss to mitigate recognition bias caused by imbalances in emotion categories.

16. The method according to claim 1, characterized in that, The AR display module supports dynamic visual cues based on speech rate and the degree of emotional shifts, enhancing users' ability to perceive changes in the speaker's state.

17. The method according to claim 1, characterized in that, By performing voiceprint, speech recognition, and emotion inference on cloud servers and caching high-frequency user features locally, a real-time recognition experience with an average latency of less than 50 ms can be achieved.

18. The method according to claim 1, characterized in that, The method improves the accuracy of emotion recognition by 10% to 20% in noisy environments through a feature fusion strategy.

Images