AR glasses supporting emotion monitoring

Abstract

The invention provides AR glasses supporting emotion monitoring, and relates to the technical field of augmented reality, the AR glasses supporting emotion monitoring comprise a glasses body, a two-way eye camera, an emotion recognition module and an image processing unit, and the image processing unit comprises an emotion recognition algorithm and an image processing unit. The emotion recognition accuracy is 95.8% (angry) and 92.3% (anxiety), and compared with a traditional single-mode scheme (only facial expression), the emotion recognition accuracy is improved by 22% (angry) and 41% (anxiety). The blink frequency detection precision reaches + / -0.5 Hz through dynamic frame rate adjustment of the two cameras, and the feature point detection rate in a rapid blink scene is 98% (only 85% of a traditional monocular). By combining HRV physiological features extracted by PPG signals, the system can recognize the coupling relation between pupil diameter changes and blinking frequency, and physiological blinking and emotional blinking are successfully distinguished.

Description

Technical Field

[0001] The present invention belongs to the technical field of augmented reality, and more specifically, particularly relates to an AR glasses supporting emotion monitoring. Background Art

[0002] In recent years, augmented reality (AR) technology has achieved rapid development. As an important hardware device of it, AR glasses are gradually moving from concept to practical application. Currently, the AR glasses on the market are mainly applied to scenarios such as information overlay, gaming and entertainment, and industrial maintenance. Their core functions focus on projecting virtual images into the user's field of vision through an optical display system, and realizing the precise overlay and interaction of images and reality by means of sensor integration and real-time data processing technology. However, most of these AR glasses ignore an important aspect, that is, the monitoring and analysis of the user's emotional state, and cannot provide personalized feedback or adjustment according to the user's emotional changes, resulting in certain limitations in the user experience.

[0003] In the field of emotion monitoring technology, traditional methods mainly rely on means such as facial expression analysis and speech analysis. The facial expression analysis method is greatly affected by the environment in practical applications. For example, in a scene with drastic light changes, such as suddenly entering a dim room from a bright outdoor environment, the facial images captured by the camera may be blurred due to insufficient light, making the extraction of facial features inaccurate, and thus affecting the accuracy of emotion recognition. There is research data showing that in a low-light (illuminance below 10 lux) environment, the accuracy rate of traditional facial expression analysis methods can drop suddenly from about 85% under normal light to about 30%. Moreover, this method is difficult to capture some subtle emotional changes, such as the emotional information conveyed by the tiny expressions in the eyes, and it is easy to cause misjudgment or missed judgment.

[0004] The speech analysis method also has obvious deficiencies. Ambient noise is one of the key factors affecting its accuracy. In noisy environments such as shopping malls and construction sites, the noise may mask the key features of the speech signal, resulting in deviations in speech recognition. Experiments show that when the ambient noise reaches above 70 dB, the misjudgment rate of the speech analysis method will increase significantly, up to 40% at most. In addition, differences in the speech habits and accents of different people will also interfere with the analysis results, challenging the generality and stability of this method.

[0005] Furthermore, traditional emotion monitoring methods often need to be carried out in specific scenarios, lacking real-time performance and flexibility. For example, facial expression analysis may require the user to face the camera directly and remain relatively stationary, which is difficult to fully meet in actual daily activities. And speech analysis requires the user to speak clearly. In some special situations, such as when the user is in a tense or excited mood, the speech may become rapid and blurred, further reducing the accuracy of monitoring.

[0006] In summary, it is of great practical significance and application value to develop an AR glasses that can monitor the user's eye expressions in real time and accurately, and then monitor the emotional state. The purpose of the present invention is to solve the problems existing in the prior art and provide a more personalized and intelligent AR experience for users. Summary of the Invention

[0007] To solve the above technical problems, the present invention provides an AR glasses supporting emotion monitoring to solve the above problems.

[0008] An AR glasses supporting emotion monitoring includes:

[0009] Glasses body: including waveguide holographic display lenses, frames and temple arms;

[0010] Dual-channel eye cameras: symmetrically installed inside the frames, using global shutter CMOS and built-in infrared fill light;

[0011] Emotion recognition module: integrated with a dedicated AI chip, including:

[0012] Image processing unit: realizing face detection, iris localization, and calculation of eyelid opening degree;

[0013] Emotion recognition algorithm: a spatio-temporal feature fusion model based on Transformer (training data > 1 million frames, emotion classification accuracy 95.2%);

[0014] Display module: superimposing AR images on the lenses through a MicroLED array, supporting the display of emotion icons and real-time emotion values (0 - 100 points).

[0015] Preferably, the dual-channel eye cameras adopt synchronous triggering technology:

[0016] The phase difference between the left and right cameras ≤ 0.5 ms, ensuring the binocular parallax measurement accuracy of ±0.2 mm;

[0017] Dynamic frame rate adjustment: when the blinking frequency > 20 times / minute, automatically increase the frame rate to ensure complete capture of eyelid movement.

[0018] Preferably, the emotion recognition module includes multi-dimensional feature extraction:

[0019] Geometric features: pupil diameter change, iris texture complexity;

[0020] Motion features: blinking frequency, gazing direction;

[0021] Physiological features: extracting heart rate variability (HRV) through PPG signals and constructing an emotion stress index (ESI) in combination with the blinking frequency.

[0022] Preferably, the emotion recognition algorithm adopts cross-modal transfer learning:

[0023] The pre-trained model is iterated 800,000 times on the dataset (loss function: cross-entropy + contrastive loss);

[0024] Real-time data augmentation: Synthetic eye images in low-light environments are generated through GAN (PSNR ≥ 30dB) to improve the robustness of the algorithm in low-light scenarios (illuminance < 5 lux).

[0025] Preferably, the display module supports emotion-driven interaction:

[0026] When detecting that the user is angry, automatically reduce the brightness of the AR image and push a calm reminder;

[0027] When it is recognized that the user is focused, enhance the contrast of the target area and block the push of irrelevant information.

[0028] Preferably, the glasses body is built-in with multi-modal sensors:

[0029] Six-axis IMU, used to compensate for the influence of head movement on the camera;

[0030] Microphone array (beamforming technology), real-time collection of environmental noise, and automatically enhance the emotion recognition confidence threshold when the noise > 75dB.

[0031] Preferably, the emotion recognition module supports the privacy protection mode:

[0032] Local storage: The original eye images are encrypted and stored, and only the desensitized feature vectors are synchronized to the cloud;

[0033] Anonymization processing: Through differential privacy technology, ensure that group emotion analysis does not disclose individual information.

[0034] Preferably, the temple of the glasses is integrated with physiological signal electrodes:

[0035] Contact skin electrodes, real-time collection of skin conductance response;

[0036] Non-contact respiration sensor (millimeter wave radar), detecting respiration frequency, and fusing with eye features to improve the emotion recognition accuracy to 97.3%.

[0037] Preferably, the lens adopts an intelligent dimming film:

[0038] Electrochromic material, automatically adjusting the light transmittance according to the Emotion Stress Index (ESI).

[0039] Preferably, the system supports third-party application interfaces:

[0040] Open emotion API, allowing external APPs to call real-time emotion data;

[0041] Link with a mental health platform (such as a meditation app) to automatically push a deep breathing guidance animation when anxiety is detected.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] In the present invention, through a dual-channel synchronous camera and multi-dimensional feature fusion (geometric / motion / physiological features), the emotion recognition accuracy rates are 95.8% (anger) and 92.3% (anxiety), which are 22% (anger) and 41% (anxiety) higher than those of traditional single-modal solutions (only facial expressions). The dynamic frame rate adjustment of the dual-channel camera enables the detection accuracy of the blink frequency to reach ±0.5 Hz, and the feature point detection rate is 98% in the scenario of rapid blinking (25 times per minute) (only 85% for traditional single-eye cameras). Combining the HRV physiological features extracted from the PPG signal, the system can identify the coupling relationship between the pupil diameter change and the blink frequency, successfully distinguish "physiological blinking" from "emotional blinking", and reduce the false alarm rate from 35% to 8%.

[0044] In the present invention, the GAN data augmentation technology is used to generate low-light images (PSNR≥30dB), and the algorithm still maintains an accuracy rate of 89% when the illuminance is 5 lux (only 62% for traditional solutions). Experimental data shows that the present invention can successfully identify the user's surprised emotion in a dark meeting room (3 lux), while the traditional solution fails completely due to pupil positioning failure. After compensating for head movement with a six-axis IMU, the extraction error of the iris texture complexity is ≤0.5 pixels, ensuring the stability of emotion recognition in low-light environments.

[0045] In the present invention, the response time of the dynamic frame rate adjustment is ≤100 ms, ensuring that the motion blur is <1 pixel. When the user is watching a horror video, the system can identify and automatically reduce the lens brightness within 120 ms, and the user feedback is that "the brightness change is timely". The display module supports the overlay of emotion icons and real-time emotion values (0-100 points). When the user's concentration is detected, the contrast of the target area is increased by 30%, and the task completion efficiency is improved.

[0046] In the present invention, the dual-channel camera uses a global shutter CMOS and infrared fill light, and can work continuously for 500 hours without failure in the IP65 waterproof test. After the drop test (1.5 meters), the lens scratch is ≤0.1 mm and the function is normal. After large-scale production, the cost per set is reduced, and the battery life is 6.5 hours. It has successfully promoted the transformation of AR glasses from an entertainment tool to a mental health assistance device. Description of the Drawings

[0047] Figure 1 is a schematic diagram of the usage process of the present invention;

[0048] Figure 2 is a schematic diagram of the structure of the AR glasses in the present invention;

[0049] Figure 3 It is a schematic diagram of the components of the AR glasses in the present invention;

[0050] Figure 4 It is a schematic diagram of the content of the emotion recognition module in the present invention;

[0051] Figure 5 It is a schematic diagram of the dimension extraction of the emotion recognition module in the present invention;

[0052] Figure 6 It is a schematic diagram of the temple of the glasses in the present invention;

[0053] Figure 7 It is a schematic diagram of the dual-channel eye camera in the present invention;

[0054] Figure 8 It is a schematic diagram of cross-modal transfer learning of the emotion recognition algorithm in the present invention;

[0055] Figure 9 It is a schematic diagram of the lens in the present invention;

[0056] Figure 10 It is a schematic diagram of the main body of the glasses in the present invention.

[0057] In the figure, the corresponding relationship between the structure name and the reference numeral is: 1. Main body of the glasses; 2. Dual-channel eye camera; 3. Emotion recognition module; 4. Display module. Detailed implementation manner

[0058] The following further describes the implementation manner of the present invention in detail with reference to the drawings and embodiments. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0059] Please refer to Figures 1 - 10 , the present invention provides an AR glasses supporting emotion monitoring, including:

[0060] Main body of the glasses 1: including a waveguide holographic display lens, a frame and temples;

[0061] Dual-channel eye camera 2: symmetrically installed inside the frame, using a global shutter CMOS, with an in-built 940nm infrared fill light, and a field of view angle ≥ 120°;

[0062] Emotion recognition module 3: integrated with a dedicated AI chip, including:

[0063] Image processing unit: realizing face detection (MTCNN accuracy 99.8%), iris positioning (error ≤ 0.5 pixels), and calculation of eyelid opening degree (accuracy ± 0.1mm);

[0064] Emotion Recognition Algorithm: Spatiotemporal Feature Fusion Model Based on Transformer (Training Data > 1 million frames, Emotion Classification Accuracy 95.2%);

[0065] Display Module 4: Superimpose AR images on the lens through a MicroLED array (brightness > 500 nit), supporting the display of emotion icons and real-time emotion values (0 - 100 points).

[0066] Example 1: Dual-channel Synchronous Eye Monitoring System:

[0067] Hardware Configuration:

[0068]

[0069]

[0070] Synchronous Trigger Mechanism:

[0071] Hardware Synchronization: Generate a synchronous clock signal through FPGA, with the exposure time difference between the left and right cameras ≤ 0.5 ms;

[0072] Dynamic Frame Rate Adjustment: When the blink frequency > 20 times / minute, automatically increase to 120 fps to ensure complete capture of eyelid movement (e.g., 6 frames can be captured when the blink duration is 80 ms).

[0073] Measured Data (Compared with Traditional Monocular Scheme):

[0074]

[0075]

[0076] Example 2: Multi-dimensional Emotion Feature Extraction:

[0077] Sensor Integration:

[0078] PPG Sensor: Integrated on the inner side of the temple (sampling rate 200 Hz), extracting HRV through corneal reflected light;

[0079] Six-axis IMU: Bosch BMI 323 (angular velocity error 0.05° / s), compensating for the influence of head movement on eye features.

[0080] Feature Fusion Algorithm:

[0081] Geometric Feature: Pupil diameter change ΔD = 0.1 mm (e.g., pupil dilation by 15% when angry);

[0082] Motion Feature: Blink frequency 18 Hz (normal) → 25 Hz (anxious), gaze direction deviation ±1°;

[0083] Physiological characteristics: Standard deviation of HRV (SDNN) changes from 50 ms (calm) to 20 ms (stressed).

[0084] Experimental data (compared with traditional facial expression schemes):

[0085]

[0086] Method embodiments:

[0087] Embodiment 3: Cross-modal transfer learning:

[0088] Model training:

[0089] Datasets: FER-2013 (35,887 images) + AffectNet (450,000 images), 800,000 iterations; Loss function: Cross-entropy (class loss) + Contrastive loss (feature similarity);

[0090] Data augmentation: GAN generates low-light images (PSNR = 30 dB, SSIM = 0.92 with real images).

[0091] Low-light performance test:

[0092]

[0093] Experimental examples (full-scenario verification):

[0094] Experimental example 4: Effectiveness of dynamic frame rate adjustment:

[0095] Test conditions:

[0096] Scenario: Rapid blinking (25 times / minute) + Head movement (angular velocity 150° / s); Metrics: Feature point detection rate, Image blur (MTF50 ≥ 30 lp / mm).

[0097] Test results:

[0098]

[0099] Experimental example 5: Real-time interaction response test:

[0100] Test process:

[0101] Users watch horror videos (inducing angry emotions), and the system automatically adjusts the lens brightness; Metrics: Emotion recognition latency, Brightness adjustment response time.

[0102] Data table:

[0103]

[0104] Comparative example:

[0105] Comparative Example 1: Dual-channel Synchronization vs Monocular Acquisition:

[0106]

[0107] Comparative Example 2: Multimodal Fusion vs Single Modality:

[0108]

[0109] Mass Production Example (Cost and Reliability): Hardware Cost (per set):

[0110]

[0111]

[0112] Reliability Test (500-hour Aging):

[0113]

[0114] Through the above examples and experiments, the present invention has achieved a breakthrough in the following dimensions:

[0115] Emotion Recognition Accuracy: Multimodal fusion increases the anger recognition rate by 22%, and the performance in low-light environment (5 lux) increases from 62% to 89%;

[0116] Dynamic Response Speed: The synchronous triggering technology reduces the parallax error by 87%, and the real-time interaction response time ≤ 300 ms;

[0117] User Experience: The lens brightness is automatically adjusted (response time 280 ms), and 94% of users think that "emotional feedback is natural";

[0118] Cost and Reliability: After large-scale production, the cost per set is ￥2800 (traditional ￥4700), with IP65 waterproof and no failure after a 1.5 m drop.

[0119] Through the technical closed-loop of "biometric perception - multimodal fusion - real-time response", the present invention has achieved a leapfrog upgrade of AR glasses from "information carrier" to "emotional interaction terminal". The core breakthroughs are reflected in the following dimensions:

[0120] 1. The accuracy of capturing micro-expressions of the eyes is increased by 87%:

[0121] Dual-channel Synchronous Triggering Technology: Dual cameras are synchronously acquired with a phase difference ≤ 0.5 ms, solving the parallax error of traditional monocular cameras (±1.5 mm → ±0.2 mm), and still maintaining a 98% feature point detection rate when the blinking frequency > 20 times / minute (Example 1).

[0122] Dynamic frame rate adjustment: High-speed acquisition at 120fps is achieved through FPGA hardware triggering, improving the eyelid movement capture accuracy to ±0.1mm (the traditional solution can only capture 50% of complete blinking actions), and successfully identifying the differences between "rapid eyelid closure in anger" and "normal blinking" (Example 3).

[0123] 2. Multimodal feature fusion breaks through environmental limitations:

[0124] Geometric - motion - physiological feature fusion: Combining pupil diameter change (ΔD≤0.1mm), blink frequency (±0.5Hz), and HRV (SDNN±5ms), the emotion recognition accuracy is improved by 15% - 41% compared with traditional single modality (Comparative Example 2).

[0125] Low - light enhancement algorithm: The GAN - generated low - light images (PSNR≥30dB) increase the accuracy rate from 62% to 89% when the illuminance is 5lux (Experimental Example 3), solving the problem of the failure of CN117224370A in low - light environments.

[0126] 3. Real - time interaction response is shortened by 76%:

[0127] Hardware acceleration engine: A dedicated AI chip (with a computing power of 2TOPS) controls the emotion recognition delay within 120ms (the traditional solution >500ms). Combining with dynamic brightness adjustment (response 280ms), a closed - loop of "emotion perception - feedback regulation" is achieved (Experimental Example 2).

[0128] Cross - modal transfer learning: The generalization ability of the pre - trained model on the FER - 2013+AffectNet dataset is improved by 37% in low - light scenarios, ensuring the interaction stability in complex environments (Example 4).

[0129] Experimental verification: Data - driven technical reliability:

[0130] Through the strict tests of a third - party testing agency, the present invention reaches the leading level in the industry in the following key indicators:

[0131] 1. All - weather emotion recognition ability:

[0132]

[0133]

[0134] 2. Dynamic response performance:

[0135] Blink frequency detection: The detection rate is 99% at 25 blinks per minute (the traditional solution misses detections at 18 blinks per minute), ensuring real - time monitoring under high - intensity emotional fluctuations (Experimental Example 1).

[0136] Interaction latency: The full-link response time from emotion recognition to brightness adjustment ≤ 300 ms (industry standard > 800 ms), and the user satisfaction rate reaches 94% (user empirical evidence).

[0137] 3. Reliability and comfort:

[0138] Environmental adaptability: IP65 waterproof, no faults in the temperature cycle test from -20°C to 60°C, and the function is normal after dropping 1.5 meters (shell damage ≤ 0.1 mm).

[0139] Wearing experience: 0.8 kg lightweight design (traditional 1.2 kg), 6.5-hour battery life (typical working conditions), and 87% of users think that "there is no sense of oppression when wearing for a long time".

[0140] Application value: Opening a new era of emotional interaction for AR glasses:

[0141] 1. In the field of mental health:

[0142] Anxiety warning: When the detected anxiety emotion value > 75, an animated deep breathing guide (breathing frequency 12 times per minute) is automatically pushed, and the user's stress index drops by an average of 23% (clinical experiment data).

[0143] Depression monitoring: Through the coupled analysis of blink frequency (normal 15 times per minute → 8 times per minute during depression) and HRV (SDNN > 50 ms → < 30 ms), the accuracy rate of early depression recognition reaches 89% (compared with the traditional scale method of 72%).

[0144] 2. In the fields of education and training:

[0145] Attention management: When the detected focused emotion value > 90, the contrast of the target area is enhanced (increased by 30%), and the learning efficiency is improved by 18% (laboratory tests).

[0146] Stress regulation: In the simulated exam scenario, the system reduces the heart rate fluctuation of candidates by 15% and the error rate by 21% through lens dimming (transmittance 30% when ESI ≥ 70).

[0147] 3. In the fields of industry and services:

[0148] Operation safety: During factory inspections, when fatigue emotion (blink frequency < 10 times per minute) is detected, a rest reminder is automatically pushed, and the accident rate drops by 34% (enterprise actual measurements).

[0149] Service optimization: In the customer service scenario, combining emotion recognition (such as anger value > 80) with voice analysis, automatically transfer to a senior customer service, and the customer satisfaction rate is increased by 28%.

[0150] Special group care: Provide emotional visualization assistance for autistic patients (Experimental Example 5), with a 45% improvement in their emotional expression and understanding ability.

[0151] Public safety: Deploy an emotion monitoring system in public places, shortening the early warning response time for sudden violent incidents to 2.1 seconds and increasing the social security index by 17%.

[0152] Through continuous innovation, the present invention will promote the evolution of AR glasses from "tools" to "partners", ultimately realizing a new paradigm of human-machine symbiosis where "technology has warmth and interaction has emotion".

[0153] The embodiments of the present invention are given for purposes of illustration and description, and are not exhaustive or limit the invention to the disclosed form. Many modifications and variations are obvious to those of ordinary skill in the art. The embodiments are chosen and described in order to better illustrate the principles of the invention and its practical application, and to enable those of ordinary skill in the art to understand the invention and design various embodiments with various modifications suitable for a particular purpose.

Claims

1. An AR glasses supporting emotion monitoring, characterized in that: It includes: Glasses body (1): It contains waveguide holographic display lenses, frames and temple arms; Dual-channel eye cameras (2): Symmetrically installed on the inner side of the frame, using global shutter CMOS and built-in infrared fill light; Emotion recognition module (3): Integrated with a dedicated AI chip, including: Image processing unit: Realize face detection, iris localization, and calculation of eyelid opening and closing degree; Emotion recognition algorithm: A spatio-temporal feature fusion model based on Transformer; Display module (4): Superimpose AR images on the lenses through a MicroLED array, supporting the display of emotion icons and real-time emotion values (0 - 100 points).

2. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, The dual-channel eye cameras (2) adopt synchronous triggering technology: The phase difference between the left and right cameras ≤ 0.5ms, ensuring the binocular parallax measurement accuracy of ±0.2mm; Dynamic frame rate adjustment: When the blink frequency > 20 times per minute, automatically increase the frame rate.

3. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, The emotion recognition module (3) includes multi-dimensional feature extraction: Geometric features: Pupil diameter change, iris texture complexity; Motion features: Blink frequency, gaze direction; Physiological features: Extract heart rate variability through PPG signals and construct an emotion stress index in combination with the blink frequency.

4. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, The recognition algorithm used by the emotion recognition module adopts cross-modal transfer learning: The pre-trained model is iterated 800,000 times on the dataset; Real-time data augmentation: Generate synthetic eye images in low-light environments through GAN to improve the robustness of the algorithm in low-light scenarios.

5. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, The display module (4) supports emotion-driven interaction: When detecting that the user is angry, automatically reduce the brightness of the AR image and push a calm reminder; When recognizing that the user is focused, enhance the contrast of the target area and block the push of irrelevant information.

6. The AR glasses supporting emotion monitoring according to claim 1, wherein The glasses body (1) is built-in with multi-modal sensors: Six-axis IMU to compensate for the influence of head movement on the camera; Microphone array to collect ambient noise in real time. When the noise > 75dB, automatically increase the emotion recognition confidence threshold.

7. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, The emotion recognition module (3) supports the privacy protection mode: Local storage: The original eye images are encrypted and stored, and only the desensitized feature vectors are synchronized to the cloud; Anonymization processing: Through differential privacy technology, ensure that group emotion analysis does not disclose individual information.

8. The AR glasses for supporting emotion monitoring according to claim 1, characterized in that, The temple arms are integrated with physiological signal electrodes: Contact skin electrodes to collect skin conductance responses in real time; Non-contact respiration sensors to detect respiration frequency.

9. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, The lenses adopt intelligent dimming films: Electrochromic materials to automatically adjust the light transmittance according to the emotion stress index (ESI).

10. The AR glasses supporting emotion monitoring according to claim 1, characterized in that, This AR glasses that support emotion monitoring support third-party application interfaces: Open emotion API to allow external APPs to call real-time emotion data; Link with a mental health platform. When detecting anxiety, automatically push a deep breathing guidance animation.

Images