A method for real-time facial expression tracking and intent-assisted parsing based on smart glasses

Abstract

This invention discloses a method for real-time facial expression tracking and intent-assisted parsing based on smart glasses. The system utilizes smart glasses terminals integrating multiple sensors and equipped with a mobile AI model for real-time assistance to wearers in natural interaction scenarios. Its core innovations include: iris authentication based on the cosine similarity formula; sound source localization based on the TDOA formula; automatic dialogue target recognition based on a weighted fusion of semantic, spatiotemporal, and responsive dimensions; emotion trend prediction based on exponential smoothing; and feedback priority calculation and tiered AR display based on multi-factor weighting. This invention enables automatic dialogue target recognition in multi-person scenarios based on a weighted fusion of semantic, spatiotemporal, and responsive dimensions. The weights can be adaptively adjusted according to the scenario, significantly improving recognition accuracy in multi-person scenarios. It is primarily applied to scenarios such as family education assistance, business communication assistance, psychological counseling assistance, and social training assistance.

Description

Technical Field

[0001] This invention relates to the technical field of wearable devices, and in particular to a method for real-time facial expression tracking and intent-assisted analysis based on smart glasses. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, smart glasses, as wearable devices, are increasingly being used in fields such as augmented reality (AR) and assisted communication. However, existing smart glasses and related technologies have the following shortcomings: Existing speech signal processing systems lack the ability to fuse and recognize multimodal information, and their accuracy in recognizing visual and auditory information is limited. While existing smart glasses devices possess image acquisition capabilities, they lack the ability to automatically identify the dialogue target, failing to accurately determine the current dialogue partner in multi-person interaction scenarios. Some existing AR / VR glasses devices, although possessing image acquisition capabilities, lack the ability to interpret the intent of the dialogue partner, failing to provide targeted auxiliary information to the wearer. Current technologies are insufficient in predicting emotional trends and intelligently scheduling feedback priorities, lacking exponential smoothing-based emotional temporal modeling and multi-factor weighted feedback priority calculation mechanisms, resulting in insufficiently intelligent timing and intensity of AR auxiliary information display.

[0003] In real-world social interaction scenarios where multiple people are present simultaneously, accurately identifying who a user is currently conversing with is a critical technical challenge. Therefore, there is an urgent need for a technical solution that can adapt to natural interaction, be portable in a small size, automatically identify the conversation target, integrate multimodal information, generate real-time facial expression tracking maps, and simultaneously possess video recording capabilities while ensuring controllable data privacy and security. Summary of the Invention

[0004] The purpose of this invention is to solve the problems in the prior art by proposing a method for real-time facial expression tracking and intent-assisted analysis based on smart glasses. This method solves the problem of how to accurately identify who the user is currently talking to, perform targeted facial expression tracking and intent analysis based on the identification results, and provide timely and effective auxiliary information to the wearer through intelligent feedback priority calculation.

[0005] To achieve the above objectives, this invention proposes a method for real-time facial expression tracking and intent-assisted parsing based on smart glasses, comprising the following steps: S1. User Authentication and Authorization Steps: When the smart glasses are activated or a change in wearing status is detected, the iris image of the wearer is captured by the iris scanning camera integrated inside the smart glasses. After preprocessing the iris image by denoising and contrast enhancement, the iris feature vector V_capture is extracted using a convolutional neural network. This vector is then compared with the iris template feature vector V_template pre-stored in a secure encryption chip to calculate the cosine similarity score M_retina, which ranges from [0,1]. The closer the score is to 1, the higher the matching degree. M_retina = (V_capture · V_template) / (‖V_capture‖ × ‖V_template‖) When M_retina ≥ θ_auth, authentication is successful and the system activates all sensor modules; when M_retina < θ_auth, the system remains locked. S2. Multimodal data synchronous acquisition steps: Visual data is acquired through the front-facing camera; multiple audio signals are acquired through a microphone array, and the azimuth angle of the sound source is calculated based on the time difference of arrival principle. θ_source = arcsin(c × Δt / d_mic), this formula applies to a horizontal angle of ±90° directly in front of the wearer. In-plane sound source localization The system collects the wearer's voice through a bone conduction microphone and converts it into text via speech recognition; it also collects heart rate, body temperature, and skin conductance through physiological sensors; and it synchronizes and aligns all data using a unified timestamp. S3. Steps for target recognition in multi-person scenarios: Obtain R_sem(i) through semantic dimension analysis, R_st(i) through spatiotemporal dimension analysis, and R_rsp(i) through response dimension analysis, and then perform weighted fusion: S_target(i) = α × R_sem(i) + β × R_st(i) + γ × R_rsp(i), α + β + γ = 1, where R_sem(i), R_st(i), and R_rsp(i) are all normalized to the interval [0,1], therefore S_target(i) The value range is [0,1]. The weighting coefficients are adaptively adjusted according to the scenario; when a candidate sound source has the highest comprehensive score and exceeds the threshold, it is identified as the dialogue target. S4. Facial Expression Tracking and Intent Parsing Steps: Perform AU detection and facial expression label matching on the confirmed dialogue target, perform cross-modal intent parsing, and predict sentiment trends based on exponential smoothing. E_trend(t) = λ × E_current(t) + (1 - λ) × E_trend(t-1), where λ is the exponent. Smoothing coefficient, with a value range of (0,1). And feedback priority calculation based on multi-factor weighting: P_prompt = w 1 × C_emotion + w 2 × C_intent + w 3 × |dE_trend / dt| Based on feedback priority, graded assistance information is presented to the wearer in real time through a dual-channel system of retinal projection AR display and bone conduction audio.

[0006] Preferably, step S4 further includes a multimodal consensus cross-validation algorithm, specifically: Emotion confidence vectors C_v, C_a, and C_s are obtained independently from the visual, audio, and semantic modalities, respectively. Multimodal consistency scores are then calculated, with values ​​ranging from [0,1]. A score closer to 1 indicates greater consistency between modalities. Consistency = 1 - √[((C_v - C_a)² + (C_v - C_s)² + (C_a - C_s)²) / 3] Where C_v is the sentiment confidence of the visual modality (expression AU + eye movement + posture), C_a is the sentiment confidence of the audio modality (tone + speech rate + pauses), and C_s is the sentiment confidence of the semantic modality (sentence analysis of dialogue text). When Consistency ≥ θ_cons, the multimodal analysis results are considered consistent and reliable, and the fusion result is directly output. When Consistency < θ_cons, the inconsistency correction mechanism is triggered, the weight of the deviating modality is reduced and the weight of the consistent modality is increased before re-fusion.

[0007] Preferably, the inconsistency correction mechanism also includes a semantic-expression contradiction detection and real-time correction algorithm, specifically: Sentiment polarity analysis is performed on the speech recognition text of the dialogue target to obtain the semantic sentiment polarity score Pol_s; simultaneously, the visual sentiment polarity score Pol_v is extracted from the facial expression recognition results of the visual modality; the semantic-facial expression contradiction degree is calculated. Contradiction = |Pol_s - Pol_v| / (|Pol_s| + |Pol_v| + ε) Where ε is a very small positive number to prevent division by zero, and its default value is 1 × 10⁻ 6 When Contradiction > θ_contra, a contradiction is determined between the semantics and the expression. A "speech-behavior inconsistency" label is added to the current intent parsing result, and the intent confidence is multiplied by a decay factor. C_intent_corrected = C_intent × (1 - Contradiction) This reduces the certainty of intention judgment when there is a semantic-expression contradiction, thus avoiding misleading the wearer.

[0008] Preferably, in step S3, the weighting coefficients are determined using a dynamic adaptive algorithm based on signal quality, specifically as follows: The semantic channel signal quality Q_sem, the spatiotemporal channel signal quality Q_st, and the response channel signal quality Q_rsp are calculated separately; where Q_sem is determined based on speech recognition confidence and environmental signal-to-noise ratio, Q_st is determined based on visual detection confidence and sound source localization stability, and Q_rsp is based on the number of response samples; the weights of each dimension are calculated through normalization. α = Q_sem / (Q_sem + Q_st + Q_rsp) β = Q_st / (Q_sem + Q_st + Q_rsp) γ = Q_rsp / (Q_sem + Q_st + Q_rsp) This weighted dynamic adaptive algorithm enables α to automatically decrease when the speech recognition quality Q_sem decreases in a high-noise environment, and β to automatically decrease when the visual detection quality Q_st decreases in a low-light environment, thus achieving real-time adaptive adjustment of weights according to environmental conditions.

[0009] Preferably, step S4 also includes a real-time emotion judgment correction algorithm based on the chat content, specifically: The speech recognition text sequence of the dialogue target is subjected to topic transition point detection and sentiment semantic analysis. When the dialogue content is detected to shift from a negative topic to a positive topic or when a clear humorous / ironic context marker appears, a semantic correction term is applied to the current sentiment recognition result. E_corrected(t) = E_raw(t) + δ × ΔSentiment(t) Where E_raw(t) is the raw emotion score obtained based on visual and audio modalities, ΔSentiment(t) is the semantic emotion change at the current time relative to the previous time, and δ is the semantic correction strength coefficient; this algorithm enables the system to correct misjudgments based solely on facial expressions in a timely manner when the dialogue content clearly expresses an emotional tendency different from that of facial expressions. Simultaneously, the corrected E_corrected(t) is substituted into the sentiment trend formula E_trend(t) = λ × E_corrected(t) + (1 - λ) × E_trend(t-1) in step S4(c) to realize the transmission from semantic correction to sentiment trend prediction.

[0010] Preferably, in the cross-modal intent parsing of step S4(b), a confidence-weighted multimodal fusion algorithm is used, specifically as follows: The facial expression intent feature vector F_v and its confidence c_v are obtained from the visual modality, the speech intent feature vector F_a and its confidence c_a are obtained from the audio modality, and the text intent feature vector F_s and its confidence c_s are obtained from the semantic modality. The formula for calculating the fused intent feature vector is as follows: F_fused = (c_v × F_v + c_a × F_a + c_s × F_s) / (c_v + c_a + c_s) This algorithm allows modalities with higher confidence to account for a larger proportion of the fusion results, while the influence of modalities with lower confidence is automatically weakened, thereby improving the robustness and accuracy of intent parsing in complex scenarios.

[0011] As a preferred option, it also includes a self-checking algorithm for misjudgments and a historical backtracking correction algorithm, specifically: Within a sliding time window [tW, t], the stability of the emotion judgment results is continuously monitored; the frequency of emotion label changes within the window is calculated. FlipRate = N_flip / W Where N_flip represents the number of times the sentiment label changes within window W; when FlipRate > θ_flip, it is determined that the current sentiment judgment is unstable and triggers historical backtracking correction: all sentiment confidence scores within the window are weighted and voted on, and the dominant sentiment within the window is determined by a confidence-weighted majority vote. E_dominant = argmax_e ∑_{t∈W} C_e(t) × ı{E(t) = e} Where C_e(t) is the sentiment confidence at time t, and ı{·} is the indicator function; E_dominant is used as the corrected sentiment output for this window, replacing the unstable frame-by-frame results.

[0012] Preferably, step S4 further includes a visual-audio temporal correlation anomaly detection algorithm, specifically: Cross-correlation analysis was performed on the timestamp sequences of facial expression changes detected by the visual modality and the timestamp sequences of tone changes detected by the audio modality to calculate the visual-audio temporal correlation. Corr_va = Cov(V_seq, A_seq) / (σ_V × σ_A) Where V_seq represents the temporal sequence of visual emotion changes, A_seq represents the temporal sequence of audio emotion changes, Cov represents the covariance, and σ_V and σ_A represent their respective standard deviations. When Corr_va is significantly lower than the threshold θ_corr, it is determined that the visual and audio signals may come from different subjects, triggering a re-verification of the dialogue target. When Corr_va remains high, it is confirmed that the visual and audio signals consistently point to the same dialogue target, enhancing the credibility of the intent parsing results.

[0013] As a preferred option, it also includes a dialogue context memory pool and a Bayesian posterior correction algorithm, specifically: A dialogue context memory pool is established to store the identified dialogue topic types, sentiment baseline, and historical intent label sequences in the current session. When a new intent judgment result is generated, it is corrected using a Bayesian posterior probability formula combined with contextual priors. P(Intent_k | Obs) = P(Obs | Intent_k) × P(Intent_k | Context) / P (Obs) Where P(Obs | Intent_k) is the likelihood of the current multimodal observation result under the intent Intent_k, P(Intent_k | Context) is the prior probability of the intent calculated based on historical information in the dialogue context memory pool, and P(Obs) is the marginal probability of the observation result. This algorithm makes intent judgment not only dependent on the multimodal observation of the current frame, but also incorporates the constraints of the dialogue history context, effectively reducing the misjudgment of isolated frames.

[0014] As an alternative, data security protection measures and a dynamic switching mechanism for dialogue targets are also included: User iris template data is matched only within the trusted execution environment of the secure encryption chip; data is encrypted locally before transmission; the inertial measurement module automatically locks and clears cached sensitive data when it detects abnormal retrieval. The dialogue target identification is a continuously dynamic update process. The system recalculates the score at a preset period. When the top score changes and remains unchanged for multiple consecutive periods, the target is automatically switched. When all scores are below the threshold, S4 is paused, and only S2 and S3 monitoring are maintained.

[0015] Beneficial effects of the present invention: The present invention 1. It realizes automatic recognition of dialogue targets in multi-person scenarios based on the weighted fusion of semantic, spatiotemporal and response dimensions. The weights can be adaptively adjusted according to the scenario, which significantly improves the recognition accuracy in multi-person scenarios.

[0016] 2. A sentiment trend prediction method based on exponential smoothing and a feedback priority calculation mechanism based on multi-factor weighting are proposed, which enable the timing, intensity and method of displaying AR auxiliary information to be intelligently adjusted according to the rate of change of sentiment.

[0017] 3. It provides a complete mathematical modeling process from iris cosine similarity verification to TDOA sound source localization, from multi-dimensional fusion scoring to emotion temporal prediction, with clear formulas and adjustable parameters for each step.

[0018] 4. It has a comprehensive data security protection mechanism. The iris template is only processed in a trusted execution environment to ensure the privacy and security of biometric information and interactive data. Attached Figure Description

[0019] The above and other features, properties and advantages of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings and embodiments, in which the same reference numerals always denote the same features, wherein: Figure 1 A flowchart of the method provided in an embodiment of the present invention; Figure 2 A system structure block diagram provided for embodiments of the present invention; Figure 3 This is a schematic diagram of the multi-dimensional fusion calculation for dialogue target recognition in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the facial expression tracking and intent parsing process in an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of the smart glasses terminal in an embodiment of the present invention; Figure 6 This is a schematic diagram of the multimodal data acquisition and processing flow in an embodiment of the present invention; Figure 7 This is a flowchart of iris authentication and authorization in an embodiment of the present invention.

[0020] In the diagram: 1-Smart glasses terminal; 2-Iris scanning camera; 3-Front-facing camera; 4-Microphone array; 5-Bone conduction microphone; 6-Physiological sensor; 7-Retina projection AR display module; 8-Inertial measurement module; 9-Security encryption chip; 10-Mobile AI model processing module; 11-Multimodal behavior analysis database; 12-Data storage and security control module; 13-Cloud server; 14-User terminal.

[0021] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. Example

[0023] See Figure 1 This invention provides a method for real-time facial expression tracking and intent-assisted parsing based on smart glasses. It includes four main steps executed sequentially: S1 user authentication and authorization, S2 multimodal data synchronous acquisition, S3 dialogue target recognition in multi-person scenarios, and S4 facial expression tracking and intent parsing. It also includes auxiliary algorithms such as multimodal cross-validation, semantic correction, and misjudgment self-checking that run through S3 and S4. The specific implementation methods for each step are described in detail below.

[0024] See Figure 7 When the smart glasses 1 are activated, the inertial measurement module 8 first detects whether the device is being worn using acceleration and gyroscope data. When a wearing action is detected (characterized by a change in acceleration pattern from rest to fit against the head), the system automatically wakes up the iris scanning camera 2.

[0025] The iris scanning camera 2 is positioned inside the eyeglass frame, facing the wearer's eyes, and uses near-infrared illumination to reduce ambient light interference. The acquired iris images first undergo the following preprocessing steps: First, Gaussian filtering is used to remove sensor noise and stray light spots caused by the near-infrared light source; second, adaptive histogram equalization enhances contrast, making the iris texture features clearer; third, pupil-iris boundary localization based on Hough transform accurately segments the iris region; fourth, iris polar coordinate unfolding maps the annular iris region into a rectangular image.

[0026] The preprocessed iris image is input into a pre-trained convolutional neural network (CNN). This network uses the first few layers of the ResNet-18 architecture as a feature extractor, outputting a high-dimensional iris feature vector V_capture ranging from 128 to 512 dimensions. In this embodiment, a 256-dimensional feature vector is preferred, achieving a balance between recognition accuracy and computational efficiency.

[0027] The system performs cosine similarity matching calculations within the Trusted Execution Environment (TEE) of the secure encryption chip 9: M_retina = (V_capture · V_template) / (‖V_capture‖ × ‖V_template‖) V_template is a pre-stored iris template feature vector stored in the secure encryption chip 9 during the user's initial registration. The entire matching operation is completed within the TEE, and V_template is not exported outside the TEE, thus ensuring the security of biometric data at the hardware level.

[0028] In this embodiment, the verification threshold θ_auth is set to 0.95 by default. When M_retina ≥ 0.95, identity verification is successful, the system plays a soft prompt to confirm identity, and then activates all sensor modules, including the front-facing camera 3, microphone array 4, bone conduction microphone 5, and physiological sensor 6, entering working mode. When M_retina < 0.95, the system remains locked, all sensors are not activated, no data is collected, and a 'identity verification failed' message is displayed to the wearer on the AR display module 7.

[0029] If three consecutive verification attempts fail, the system enters a security cooldown period (30 seconds by default), during which new verification requests are rejected to prevent brute-force attacks. Users can unlock the system using auxiliary verification methods on their associated user terminal 14 (such as a password or a verification code from their linked mobile phone).

[0030] See Figure 6 After successful authentication, the system enters the multimodal data synchronous acquisition phase. This step involves four parallel data acquisition channels, and the specific implementation methods for each channel are as follows: (1) Visual data acquisition channel The front-facing camera 3 is positioned slightly above the bridge of the nose on the front of the glasses frame, facing directly in front of the wearer. In this embodiment, the front-facing camera 3 uses a 2-megapixel CMOS sensor and supports real-time video stream output at 30fps. The acquired raw video stream undergoes the following processing: First, a lightweight face detection model (such as one based on the MobileNet-SSD architecture) is used to detect all face regions in each frame, outputting the face bounding box coordinates and detection confidence. Then, 68 facial key points are located for each detected face, extracting the geometric features of key areas such as the eyes, eyebrows, mouth, and nose. Simultaneously, the position information of each person in the image coordinate system is extracted, and the approximate distance and orientation relative to the wearer are estimated by combining the camera's focal length parameters.

[0031] (2) Audio data acquisition and sound source localization channel The microphone array 4 includes at least four MEMS microphone units distributed on both temples of the smart glasses. In this embodiment, it is configured with a symmetrical distribution of two microphones on the left temple and two on the right temple, with an adjacent microphone spacing of approximately 15mm. Each microphone synchronously acquires audio signals at a sampling rate of 16kHz.

[0032] The azimuth angle of the sound source is calculated based on the Time Difference of Arrival (TDOA) principle. For any pair of microphones (m1, m2), the time difference Δt of the received signal is calculated using the Generalized Cross-Correlation-Phase Transform (GCC-PHAT) algorithm, and then the azimuth angle of the sound source is calculated: θ_source = arcsin(c × Δt / d_mic), this formula applies to a horizontal angle of ±90° directly in front of the wearer. In-plane sound source localization Where c is the speed of sound (approximately 343 m / s in air at 20°C), and d_mic is the distance between the microphone pairs. In this embodiment, four microphones can form six microphone pairs. By performing least-squares fusion on the localization results of all microphone pairs, the angular resolution of sound source localization can be improved to approximately 5°, and the distance estimation error within a 3-meter range can be controlled within 0.5 meters.

[0033] The system also maintains a sound source tracking table, recording the spatial coordinates, activity status, and speech activity periods of each sound source. When a new sound source is detected, a tracking entry is automatically created; when a sound source has no speech activity for more than 10 consecutive seconds, it is marked as inactive.

[0034] (3) Voice and text data acquisition channel The bone conduction microphone 5 is positioned on the inner side of the left temple of the glasses, in direct contact with the wearer's temporal bone, and uses the principle of bone conduction to collect the wearer's own speech signal. Because bone conduction signals naturally isolate environmental noise, the signal-to-noise ratio of the wearer's speech is much higher than that of the mixed speech collected by the environmental microphone.

[0035] The wearer's bone conduction speech and multiple ambient speech streams collected by the microphone array are converted into text sequences in real time by a lightweight speech recognition model (such as a streaming ASR model based on the Conformer architecture) deployed on the edge. Each speech text includes three metadata fields: timestamp, source number, and speech recognition confidence score. In this embodiment, the speech recognition model has approximately 20M parameters and a single inference latency of less than 100ms, meeting the requirements for real-time processing.

[0036] (4) Physiological data acquisition channel The physiological sensor 6 is integrated on the inner side of the right temple, where it contacts the wearer's skin. It includes a photoplethysmography (PPG) heart rate sensor, a thermocouple body temperature sensor, and a ground-skin conductance (GSR) electrode. The heart rate sensor outputs instantaneous heart rate values ​​at 1 Hz, the body temperature sensor outputs skin surface temperature values ​​at 0.5 Hz, and the GSR electrode outputs skin conductance values ​​at 4 Hz. The physiological data is primarily used to help determine the wearer's stress state (such as tension or anxiety) and as a reference for adjusting feedback strategies.

[0037] (5) Multimodal data time synchronization All sensor data streams are timestamped using a hardware timer built into the mobile AI model processing module 10. In this embodiment, the timestamp accuracy is 1ms. After being timestamped, each modal data is stored in a circular buffer with a capacity of all multimodal data from the most recent 30 seconds, for real-time reading in subsequent steps S3 and S4.

[0038] See Figure 3 In interactive scenarios involving multiple people, the system needs to identify the wearer's current conversation target from N candidate sound sources. This step is achieved through three-dimensional analysis and weighted fusion.

[0039] (3a) Semantic dimension analysis The system calculates the semantic similarity between the wearer's speech recognition text (originating from bone conduction microphone 5) and the speech recognition text of each candidate sound source i. Specifically, the text sequence within the most recent T seconds (T=10 seconds in this embodiment) is encoded into semantic vectors by a pre-trained sentence vector encoder (such as a lightweight model based on DistilBERT), and the cosine similarity is calculated to obtain R_sem(i).

[0040] Simultaneously, the system performs named entity recognition and title extraction (e.g., Mr. Zhang, Xiao Wang) on ​​the wearer's voice text, and matches it with the historical self-identity statements of each candidate voice source (e.g., I am Zhang XX). If the match is successful, an additional term is added to R_sem(i) to increase the probability that the candidate voice source is identified as the dialogue target.

[0041] (3b) Spatiotemporal Dimension Analysis Spatiotemporal dimension analysis comprehensively examines three factors: speaker rotation pattern, gaze direction, and spatial distance.

[0042] Speaker turn-taking pattern: The frequency and regularity of turn-taking between the wearer and candidate sound source i are statistically analyzed. Natural dialogue typically exhibits an alternating speaking pattern, which is quantified by calculating the number of turn-taking interactions between the wearer and sound source i and the average turn-taking interval over the past 30 seconds.

[0043] Gaze direction: The front-facing camera 3 is used to detect the wearer's head orientation (as an approximation of the gaze direction), and the angle between the head orientation and the spatial location of each candidate sound source is calculated. The smaller the angle, the more likely the wearer is to be paying attention to that sound source.

[0044] Spatial distance: Based on sound source localization data, sound sources closer to the wearer are given priority. The distance factor is modeled using an inverse distance weighting function.

[0045] The three factors are combined in a weighted manner to obtain R_st(i).

[0046] (3c) Response Dimension Analysis The system analyzes the wearer's response characteristics to each candidate sound source's speech, including: response frequency (whether the wearer responds within 3 seconds after sound source i speaks), response delay (the time interval from when sound source i finishes speaking to when the wearer begins speaking), and response mode (whether it is a direct response, whether it includes a form of address, etc.). R_rsp(i) is calculated based on these characteristics.

[0047] (3D) Multi-dimensional weighted fusion The scores from the three dimensions are weighted and combined: S_target(i) = α × R_sem(i) + β × R_st(i) + γ × R_rsp(i) The weighting coefficients α, β, and γ satisfy α + β + γ = 1, where R_sem(i), R_st(i), and R_rsp(i) are all normalized to the interval [0,1]. Therefore, S_target(i) takes values ​​in the range [0,1]. The weighting coefficients are determined using a dynamic adaptive algorithm based on signal quality. α = Q_sem / (Q_sem + Q_st + Q_rsp) β = Q_st / (Q_sem + Q_st + Q_rsp) γ = Q_rsp / (Q_sem + Q_st + Q_rsp) Where Q_sem is the semantic channel signal quality (calculated based on ASR confidence and environmental signal-to-noise ratio), Q_st is the spatiotemporal channel signal quality (calculated based on face detection confidence and sound source localization stability), and Q_rsp is the response channel signal quality (calculated based on the accumulated number of response samples).

[0048] For example, in a noisy, multi-person gathering, the decrease in environmental signal-to-noise ratio leads to a decrease in Q_sem, and α automatically decreases to about 0.20, while Q_st remains high because visual cues (gaze direction) are still clear, and β automatically increases to about 0.50; in a quiet, one-on-one scene, Q_sem is high, and α automatically adjusts to about 0.40.

[0049] When the comprehensive score S_target(i) of a candidate sound source i is the highest among all candidate sound sources and exceeds the preset dialogue confirmation threshold θ_target (set to 0.6 by default in this embodiment), the system confirms the candidate sound source as the current dialogue target.

[0050] The dialogue target identification process is continuously and dynamically updated. The system continuously recalculates the comprehensive score of each candidate sound source every 500ms. When the top-ranked candidate sound source changes and the new top-ranked candidate sound source maintains the highest score for three consecutive cycles (i.e., 1.5 seconds), the system automatically switches the dialogue target to the new top-ranked candidate sound source. When the comprehensive score of all candidate sound sources is lower than θ_target, the system determines that the wearer is not currently in a dialogue state, and step S4 is paused, while only steps S2 and S3 continue monitoring.

[0051] See Figure 4 For the dialogue target confirmed by S3, the system performs targeted multimodal behavior analysis, which includes four sub-steps: facial expression recognition, cross-modal intent parsing, emotion trend prediction, and AR-assisted information display.

[0052] (4a) Facial expression recognition The system performs facial action unit (AU) detection on the facial image region of the dialogue target. In this embodiment, a multi-label AU detection model based on the EfficientNet-B0 backbone network is used, which can simultaneously detect the activation state and activation intensity of 17 common AUs. The model has approximately 5.3M parameters and a single-frame inference latency of approximately 15ms.

[0053] The detected AU combinations are fed into the AU-expression label mapping table in the multimodal behavior analysis database 11 for matching. This mapping table contains at least 200 labeled expression tags, covering 8 basic emotions (happiness, sadness, anger, surprise, fear, disgust, contempt, and neutral) and more than 30 complex emotional states (such as embarrassment, confusion, boredom, and interest). A specific mapping example is shown below: AU combination meaning emoji tags {AU6+AU12} Cheek lift + Lip corner lift Happy {AU1+AU4+AU15} Raise inner eyebrows + lower eyebrows + pull down corners of lips sad {AU4+AU5+AU7+AU23} Lowering eyebrows + raising upper eyelids + tightening eyelids + closing lips anger {AU1+AU2+AU5+AU26} Inner brow lift + outer brow lift + upper eyelid lift + chin drop surprise {AU12+AU6+AU25} Low intensity Slightly lift the corners of the lips + slightly raise the cheeks + slightly part the lips Polite smile {AU14} Dimple puller contempt / disdain The facial expression recognition module outputs the facial expression category label and its confidence score C_emotion for the current frame.

[0054] (4b) Cross-modal intent parsing The system performs cross-modal fusion of facial expression recognition results with the speech tone features (fundamental frequency F0 change rate, speech rate, pause patterns), eye movement features (gaze point shift, pupil diameter change), head posture features (nodding frequency, head shaking action), and gesture features (hand movement amplitude) of the dialogue target.

[0055] The fusion employs a confidence-weighted multimodal fusion algorithm. The facial expression intent feature vector F_v and its confidence c_v are obtained from the visual modality; the speech intent feature vector F_a and its confidence c_a are obtained from the audio modality; and the text intent feature vector F_s and its confidence c_s are obtained from the semantic modality. F_fused = (c_v × F_v + c_a × F_a + c_s × F_s) / (c_v + c_a + c_s) The fused vector is matched with the behavior-intent association rule set in the multimodal behavior parsing database 11, and the output intent category label and its confidence C_intent are determined. Intent categories include, but are not limited to: agreement, opposition, hesitation, deception, boredom, interest, confusion, avoidance, etc.

[0056] (4c) Sentiment Trend Prediction The system calculates an exponentially weighted moving average of the emotion score sequence output by facial expression recognition to predict emotion change trends: E_trend(t) = λ × E_current(t) + (1 - λ) × E_trend(t-1), where λ is the exponent. Smoothing coefficient, with a value range of (0,1). Where λ is the smoothing coefficient, which is set to 0.3 by default in this embodiment. The larger the value of λ, the more sensitive the model is to the current observation; the smaller the value of λ, the smoother the trend curve. Simultaneously, the rate of change of sentiment |dE_trend / dt| is calculated for subsequent feedback priority calculation.

[0057] When the system detects a shift in the topic of conversation (such as from complaining to joking, or the appearance of ironic context markers), it activates a real-time emotion judgment and correction algorithm based on the chat content: E_corrected(t) = E_raw(t) + δ × ΔSentiment(t) Where ΔSentiment(t) represents the semantic sentiment change from the previous moment to the current moment, and δ is the semantic correction strength coefficient (set to 0.4 by default in this embodiment). The corrected E_corrected(t) is substituted into the sentiment trend formula instead of E_current(t) to realize the transmission from semantic correction to sentiment trend prediction.

[0058] (4D) Feedback Priority Calculation and AR Display The system calculates a feedback priority score based on the analysis results: P_prompt = w1 × C_emotion + w2 × C_intent + w3 × |dE_trend / dt| Where w1=0.3, w2=0.4, and w3=0.3 are the default weight configurations in this embodiment, and w1+w2+w3=1. The higher the P_prompt value, the more priority the current analysis results need to be shown to the wearer.

[0059] Adopt a hierarchical display mechanism according to the P_prompt value: when P_prompt > 0.8, trigger a high-priority red visual warning and an audio prompt (such as a bone conduction voice 'Attention, the other party is emotionally fluctuating'); when 0.5 < P_prompt ≤ 0.8, trigger a medium-priority yellow label prompt (such as displaying a 'Hesitation' label near the face of the conversation target); when P_prompt ≤ 0.5, display it低调 in the form of a blue-green background icon (such as a smiling icon indicating 'The other party is in a positive state').

[0060] The selection of the display timing utilizes the results of voice activity detection, avoids the period when the conversation target is speaking, and uses the natural pause interval of the conversation (pause duration > 300ms) for information presentation to reduce interference with the wearer's attention.

[0061] Specific implementation of the multimodal cross-checking and correction algorithm During the execution of S4, the following auxiliary algorithms run in parallel to continuously monitor and correct the reliability of the analysis results.

[0062] (5a)Multimodal consistency cross-check After the system independently obtains the respective emotion confidence levels C_v, C_a, and C_s from the visual, audio, and semantic modalities, it calculates the multimodal consistency score, with a value range of [0,1]. The closer it is to 1, the more consistent the judgments between modalities are: Consistency = 1 - √[((C_v - C_a)² + (C_v - C_s)² + (C_a - C_s)²) / 3] In this embodiment, the consistency threshold θ_cons is set to 0.7. When Consistency ≥ 0.7, the judgment results of the three modalities are basically the same, and the system directly outputs the fusion result. When Consistency < 0.7, the system identifies the deviated modality with the largest difference from the judgments of the other two modalities, reduces its weight by 50%, and simultaneously proportionally increases the weights of the two consistent modalities, and then re-executes the fusion calculation.

[0063] (5b)Semantic-expression contradiction detection and correction The system performs emotional polarity analysis on the speech recognition text to obtain Pol_s (value range [-1, +1], negative value indicates negative emotion, positive value indicates positive emotion), and obtains Pol_v from visual expression recognition, and calculates the contradiction degree: Contradiction = |Pol_s - Pol_v| / (|Pol_s| + |Pol_v| + ε) When Contradiction > 0.6 (the default θ_contra value in this embodiment), a significant contradiction is determined between the semantics and the facial expression. A typical scenario is: the target of the dialogue says 'No problem, I'm fine' (Pol_s is positive), but the detected facial expression is a frown and a downturned mouth (Pol_v is negative). In this case, the system applies a decay to the confidence of the intent for that instance: C_intent_corrected = C_intent × (1 - Contradiction) Additionally, a 'inconsistency between words and actions' marker is added to the AR display to alert the wearer.

[0064] (5c) Self-checking for misjudgments and correction of historical data The system monitors the flipping frequency of sentiment tags within a time-sliding window (window size W = 5 seconds in this embodiment): FlipRate = N_flip / W When FlipRate > 0.6 times / second (θ_flip value), the emotion judgment is determined to be oscillating and unstable (e.g., frequently jumping between 'happy' and 'neutral'), triggering historical backtracking correction. The system performs weighted voting on the emotion confidence of all frames within the window: E_dominant = argmax_e ∑_{t∈W} C_e(t) × ı{E(t) = e} The E_dominant value is used as the corrected emotion output for this window. This mechanism effectively filters out emotion label jumps caused by transient noise, resulting in a more stable and reliable emotion assessment.

[0065] (5d) Visual-audio temporal correlation anomaly detection The system performs Pearson correlation coefficient analysis on the time series of facial expression changes (V_seq) in the visual modality and the time series of tone changes (A_seq) in the audio modality: Corr_va = Cov(V_seq, A_seq) / (σ_V × σ_A) In a normal one-on-one conversation, the changes in the target's facial expressions should be highly positively correlated with changes in their tone of voice (Corr_va > 0.5). When Corr_va remains below 0.3 (θ_corr value) for more than 3 seconds, the system determines that the visual and audio signals may come from different subjects (e.g., the person the wearer is looking at is not the speaker), and automatically triggers the S3 conversation target re-verification process.

[0066] (5e) Bayesian posterior correction The system maintains a dialogue context memory pool, storing the identified topic types (such as work discussions, casual conversation, and arguments), the sentiment baseline (the average sentiment level of the dialogue target in this conversation), and the historical intent label sequence. When a new intent judgment result is generated, it is corrected using Bayesian posterior probability. P(Intent_k | Obs) = P(Obs | Intent_k) × P(Intent_k | Context) / P(Obs) The prior probability P(Intent_k | Context) is calculated based on the historical intent sequences in the memory pool through frequency statistics. For example, if the current dialogue is always in the topic mode of 'friendly small talk', the prior probability of 'deceptive' intent is low. Even if the multimodal observation results of the current frame have certain deceptive features, the corrected posterior probability will be significantly reduced, thereby avoiding misjudgment of isolated frames.

[0067] The prior pool is initialized with a uniform prior at the start of each session and is updated incrementally as the conversation progresses. When a clear topic shift is detected (such as switching from a work discussion to a private topic), the system applies a decay factor to the prior distribution in the prior pool, gradually transitioning it to the prior distribution of the new topic.

[0068] System Architecture Implementation See Figure 2 and Figure 5 The present invention also provides a real-time facial expression tracking and intent-assisted analysis system based on smart glasses, the hardware and software architecture of which are as follows: The smart glasses terminal 1 integrates: an iris scanning camera 2 (inner side facing the eye, near-infrared light source), a front-facing camera 3 (front facing outward, 2-megapixel CMOS), a microphone array 4 (two MEMS microphones on each temple, spaced 15mm apart), a bone conduction microphone 5 (inner side of the left temple, in contact with the temporal bone), a physiological sensor 6 (inner side of the right temple, PPG + thermocouple + GSR), a retinal projection AR display module 7 (inner side of the lens, supporting text, icon and color encoding overlay display), an inertial measurement module 8 (9-axis IMU, including a three-axis accelerometer, a three-axis gyroscope and a three-axis magnetometer), and a security encryption chip 9 (supporting a TEE trusted execution environment, storing V_template and performing cosine similarity calculations).

[0069] The mobile AI model processing module 10 is equipped with an edge inference engine. In this embodiment, it uses a low-power NPU chip with a computing power of no less than 2 TOPS. It is responsible for performing all or part of the inference tasks locally, including: face detection and key point localization, AU detection, GCC-PHAT sound source localization, streaming speech recognition, semantic similarity calculation, three-dimensional fusion scoring, confidence-weighted multimodal fusion, consistency cross-validation, semantic correction, Bayesian posterior correction, sentiment trend prediction, feedback priority calculation, misjudgment self-check and historical backtracking, and visual-audio temporal correlation detection.

[0070] The multimodal behavior parsing database 11 is stored in the local flash memory of module 10 and includes an AU-emoji tag mapping table (200+ entries), a behavior-intent association rule set (500+ rules), emotion temporal prediction model parameters, and a dialogue context memory pool. The database can be remotely synchronized and updated via cloud server 13.

[0071] The data storage and security control module 12 is responsible for storing all raw data and analysis results collected by sensors using AES-256 encryption, implementing role-based access control (RBAC), and recording complete operation logs.

[0072] The system optionally includes a cloud server 13 and a user terminal 14. The cloud server 13 connects to the smart glasses terminal 1 via an end-to-end TLS 1.3 encrypted channel, providing cloud computing support for complex analysis tasks exceeding local computing power, and distributing parameter updates for the database and AI models. The user terminal 14 (such as a smartphone app) is used by the user to configure system parameters (such as adjusting parameter values ​​like θ_auth, λ, and δ) and view historical interaction analysis records.

[0073] Typical application scenarios implementation (1) Family education support scenarios When parents wear smart glasses to talk to their adolescent children, the system tracks the children's micro-expression changes and semantic content in real time. When it detects that the child's face shows {AU1+AU4} (frowning combination) but the semantic content is 'It's okay, I'm fine', the semantic-expression contradiction detection algorithm calculates that the Contradiction value exceeds the threshold. The system then displays a yellow warning label to the parents via AR, indicating 'There may be inconsistency between words and actions, please pay attention', helping parents to be more aware of any potential emotional distress in their children.

[0074] (2) Business negotiation support scenarios In business negotiations, when the wearer faces the other party's representative, the system uses multimodal fusion analysis to examine the representative's facial expressions, tone of voice, and content of speech. When the other party's expression rapidly changes from 'neutral' to 'tense' while discussing price terms (emotional change rate |dE_trend / dt|sudden increase), and the feedback priority P_prompt exceeds a high threshold, the system provides a low-pitched bone conduction audio alert that 'the other party's emotions are fluctuating, and the current issue is sensitive,' helping the wearer to control the pace of the negotiation.

[0075] (3) Psychological counseling auxiliary scenarios When a therapist wears smart glasses to converse with a client, the system continuously tracks the client's emotional curve using exponentially smoothed emotional trend prediction. A Bayesian posterior correction algorithm, combined with the client's emotional baseline recorded in the dialogue context memory pool, effectively distinguishes between the client's normal emotional fluctuations and abnormal emotional changes requiring attention, reducing false alarms and providing therapists with more accurate auxiliary information.

[0076] (4) Social training assistance scenarios For users with social cognitive impairments (such as autism spectrum disorder), the system helps them identify and understand others' emotional states in a more intuitive way (e.g., displaying large emoji icons and short text labels like 'He's smiling' or 'He looks confused' next to the target's face). A misjudgment self-checking algorithm ensures the stability of the displayed results, avoiding user confusion caused by rapidly changing labels.

[0077] Specific Implementation of Data Security Protection The data security protection of this invention extends throughout the entire system operation process: First, the iris template is secure. V_template is stored only within the TEE of the secure encryption chip 9. All cosine similarity matching operations are performed within the TEE, ensuring that the original iris feature data never leaves the TEE boundary, thus preventing extraction by malicious software.

[0078] Second, transmission security. All data uploaded from the smart glasses terminal 1 to the cloud server 13 is encrypted using AES-256, and the transmission process uses the TLS 1.3 end-to-end encryption protocol.

[0079] Third, anomaly detection and sensitive data clearing. The inertial measurement module 8 continuously monitors the wearing status of the glasses. When an abnormal removal event is detected (such as the acceleration characteristic of being suddenly pulled away from the head from the normal wearing state), the system automatically triggers a lock within 500ms, clears all unencrypted temporary data in the circular buffer, and requires iris verification to be performed again before it can be used again.

[0080] Fourth, access control and auditing. The data storage and security control module 12 implements role-based access control, distinguishing between three roles: wearer (full access), authorized viewer (view only analysis reports), and system administrator (configuration parameters). All data access operations are recorded in an immutable operation log, supporting post-event auditing.

[0081] This invention presents a real-time facial expression tracking and intent-assisted parsing method based on smart glasses. It achieves automatic target recognition in multi-person scenarios using a weighted fusion of semantic, spatiotemporal, and responsive dimensions. The weights can be adaptively adjusted according to the scenario, significantly improving recognition accuracy in multi-person scenarios. It proposes an exponential smoothing-based emotion trend prediction method and a multi-factor weighted feedback priority calculation mechanism, enabling the timing, intensity, and manner of AR-assisted information display to intelligently adjust according to the rate of emotion change. It provides a complete end-to-end mathematical modeling from iris cosine similarity verification to TDOA sound source localization, from multi-dimensional fusion scoring to emotion temporal prediction, with clearly defined formulas and adjustable parameters for each step.

[0082] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for real-time facial expression tracking and intent-assisted parsing based on smart glasses, characterized in that: Includes the following steps: S1. User Authentication and Authorization Steps: When the smart glasses are activated or a change in wearing status is detected, the iris image of the wearer is captured by the iris scanning camera integrated inside the smart glasses. After preprocessing the iris image by denoising and contrast enhancement, the iris feature vector V_capture is extracted using a convolutional neural network. This vector is then compared with the iris template feature vector V_template pre-stored in a secure encryption chip to calculate the cosine similarity score M_retina, which ranges from [0,1]. The closer the score is to 1, the higher the matching degree. M_retina = (V_capture · V_template) / (‖V_capture‖ × ‖V_template‖) When M_retina ≥ θ_auth, authentication is successful and the system activates all sensor modules; when M_retina < θ_auth, the system remains locked. S2. Multimodal data synchronous acquisition steps: Visual data is acquired through the front-facing camera; multiple audio signals are acquired through a microphone array, and the azimuth angle of the sound source is calculated based on the time difference of arrival principle. θ_source = arcsin(c × Δt / d_mic), this formula applies to the horizontal plane within ±90° directly in front of the wearer. Sound source localization The system collects the wearer's voice through a bone conduction microphone and converts it into text via speech recognition; it also collects heart rate, body temperature, and skin conductance through physiological sensors; and it synchronizes and aligns all data using a unified timestamp. S3. Steps for target recognition in multi-person scenarios: Obtain R_sem(i) through semantic dimension analysis, R_st(i) through spatiotemporal dimension analysis, and R_rsp(i) through response dimension analysis, and then perform weighted fusion: S_target(i) = α × R_sem(i) + β × R_st(i) + γ × R_rsp(i), α + β + γ = 1. Where R_sem(i), R_st(i), and R_rsp(i) are all normalized to the interval [0,1], therefore the value of S_target(i) is... The range is [0,1] The weighting coefficients are adaptively adjusted according to the scenario. When a candidate sound source has the highest overall score and exceeds the threshold, it is identified as the dialogue target. S4. Facial Expression Tracking and Intent Parsing Steps: Perform AU detection and facial expression label matching on the confirmed dialogue target, perform cross-modal intent parsing, and predict sentiment trends based on exponential smoothing. E_trend(t) = λ × E_current(t) + (1 - λ) × E_trend(t-1), where λ is the exponential smoothing factor. Coefficient, with a range of values ​​(0,1). And feedback priority calculation based on multi-factor weighting: P_prompt = w 1 × C_emotion + w 2 × C_intent + w 3 × |dE_trend / dt| Based on feedback priority, graded assistance information is presented to the wearer in real time through a dual-channel system of retinal projection AR display and bone conduction audio.

2. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: Step S4 also includes a multimodal consensus cross-validation algorithm, specifically: Emotion confidence vectors C_v, C_a, and C_s are obtained independently from the visual, audio, and semantic modalities, respectively. Multimodal consistency scores are then calculated, with values ​​ranging from [0,1]. A score closer to 1 indicates greater consistency between modalities. Consistency = 1 - √[((C_v - C_a)² + (C_v - C_s)² + (C_a - C_s)²) / 3] Where C_v is the sentiment confidence of the visual modality (expression AU + eye movement + posture), C_a is the sentiment confidence of the audio modality (tone + speech rate + pauses), and C_s is the sentiment confidence of the semantic modality (sentence analysis of dialogue text). When Consistency ≥ θ_cons, the multimodal analysis results are considered consistent and reliable, and the fusion result is directly output. When Consistency < θ_cons, the inconsistency correction mechanism is triggered, the weight of the deviating modality is reduced and the weight of the consistent modality is increased before re-fusion.

3. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: The inconsistency correction mechanism also includes a semantic-expression contradiction detection and real-time correction algorithm, specifically: Sentiment polarity analysis is performed on the speech recognition text of the dialogue target to obtain the semantic sentiment polarity score Pol_s; at the same time, the visual sentiment polarity score Pol_v is extracted from the facial expression recognition results of the visual modality. Calculate semantic-facial contradiction: Contradiction = |Pol_s - Pol_v| / (|Pol_s| + |Pol_v| + ε) Where ε is a very small positive number to prevent division by zero, and its default value is 1 × 10⁻ 6 When Contradiction > θ_contra, a contradiction is determined between the semantics and the expression. A "speech-behavior inconsistency" label is added to the current intent parsing result, and the intent confidence is multiplied by a decay factor. C_intent_corrected = C_intent × (1 - Contradiction) This reduces the certainty of intention judgment when there is a semantic-expression contradiction, thus avoiding misleading the wearer.

4. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: In step S3, the weighting coefficients are determined using a dynamic adaptive algorithm based on signal quality, specifically as follows: The semantic channel signal quality Q_sem, the spatiotemporal channel signal quality Q_st, and the response channel signal quality Q_rsp are calculated separately; where Q_sem is determined based on speech recognition confidence and environmental signal-to-noise ratio, Q_st is determined based on visual detection confidence and sound source localization stability, and Q_rsp is based on the number of response samples; the weights of each dimension are calculated through normalization. α = Q_sem / (Q_sem + Q_st + Q_rsp) β = Q_st / (Q_sem + Q_st + Q_rsp) γ = Q_rsp / (Q_sem + Q_st + Q_rsp) This dynamic adaptive weighting algorithm enables α to automatically decrease when the speech recognition quality Q_sem decreases in a high-noise environment, and β to automatically decrease when the visual detection quality Q_st decreases in a low-light environment, thus achieving real-time adaptive adjustment of weights according to environmental conditions.

5. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: Step S4 also includes a real-time emotion judgment correction algorithm based on chat content, specifically: The speech recognition text sequence of the dialogue target is subjected to topic transition point detection and sentiment semantic analysis. When the dialogue content is detected to shift from a negative topic to a positive topic or when a clear humorous / ironic context marker appears, a semantic correction term is applied to the current sentiment recognition result. E_corrected(t) = E_raw(t) + δ × ΔSentiment(t) Where E_raw(t) is the raw emotion score obtained based on visual and audio modalities, ΔSentiment(t) is the semantic emotion change at the current time relative to the previous time, and δ is the semantic correction strength coefficient; this algorithm enables the system to correct misjudgments based solely on facial expressions in a timely manner when the dialogue content clearly expresses an emotional tendency different from that of facial expressions. Simultaneously, the corrected E_corrected(t) is substituted into the sentiment trend formula E_trend(t) = λ × E_corrected(t) + (1 - λ) × E_trend(t-1) in step S4(c) to realize the transmission from semantic correction to sentiment trend prediction.

6. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: In the cross-modal intent parsing of step S4(b), a confidence-weighted multimodal fusion algorithm is used, specifically as follows: The facial expression intent feature vector F_v and its confidence c_v are obtained from the visual modality, the speech intent feature vector F_a and its confidence c_a are obtained from the audio modality, and the text intent feature vector F_s and its confidence c_s are obtained from the semantic modality. The formula for calculating the fused intent feature vector is as follows: F_fused = (c_v × F_v + c_a × F_a + c_s × F_s) / (c_v + c_a + c_s) This algorithm allows modalities with higher confidence to account for a larger proportion of the fusion results, while the influence of modalities with lower confidence is automatically weakened, thereby improving the robustness and accuracy of intent parsing in complex scenarios.

7. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: It also includes a self-checking algorithm for misjudgments and a historical backtracking correction algorithm, specifically: Within a sliding time window [tW, t], the stability of the emotion judgment results is continuously monitored; the frequency of emotion label changes within the window is calculated. FlipRate = N_flip / W Where N_flip represents the number of times the sentiment label changes within window W; when FlipRate > θ_flip, the current sentiment judgment is determined to be unstable and oscillating, triggering historical backtracking correction: all sentiment confidence scores within the window are weighted and voted on, and the dominant sentiment within the window is determined by a confidence-weighted majority vote. E_dominant = argmax_e ∑_{t∈W} C_e(t) × ı{E(t) = e} Where C_e(t) is the sentiment confidence at time t, and ı{·} is the indicator function; E_dominant is used as the corrected sentiment output for this window, replacing the unstable frame-by-frame results.

8. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: Step S4 also includes a visual-audio temporal correlation anomaly detection algorithm, specifically: Cross-correlation analysis was performed on the timestamp sequences of facial expression changes detected by the visual modality and the timestamp sequences of tone changes detected by the audio modality to calculate the visual-audio temporal correlation. Corr_va = Cov(V_seq, A_seq) / (σ_V × σ_A) Where V_seq represents the temporal sequence of visual emotion changes, A_seq represents the temporal sequence of audio emotion changes, Cov represents the covariance, and σ_V and σ_A represent their respective standard deviations. When Corr_va is significantly lower than the threshold θ_corr, it is determined that the visual and audio signals may come from different subjects, triggering a re-verification of the dialogue target. When Corr_va remains high, it is confirmed that the visual and audio signals consistently point to the same dialogue target, enhancing the credibility of the intent parsing results.

9. The method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: It also includes a dialogue context memory pool and a Bayesian posterior correction algorithm, specifically: A dialogue context memory pool is established to store the identified dialogue topic types, sentiment baseline, and historical intent label sequences in the current session. When a new intent judgment result is generated, it is corrected using a Bayesian posterior probability formula combined with contextual priors. P(Intent_k | Obs) = P(Obs | Intent_k) × P(Intent_k | Context) / P(Obs) Where P(Obs | Intent_k) is the likelihood of the current multimodal observation result under the intent Intent_k, P(Intent_k | Context) is the prior probability of the intent calculated based on historical information in the dialogue context memory pool, and P(Obs) is the marginal probability of the observation result. This algorithm makes intent judgment not only dependent on the multimodal observation of the current frame, but also incorporates the constraints of the dialogue history context, effectively reducing the misjudgment of isolated frames.

10. A method for real-time facial expression tracking and intent-assisted analysis based on smart glasses as described in claim 1, characterized in that: It also includes data security protection measures and a mechanism for dynamically switching dialogue targets: User iris template data is matched only within the trusted execution environment of the secure encryption chip; data is encrypted locally before transmission; the inertial measurement module automatically locks and clears cached sensitive data when it detects abnormal retrieval. The dialogue target identification is a continuously dynamic update process. The system recalculates the score at a preset period. When the top score changes and remains unchanged for multiple consecutive periods, the target is automatically switched. When all scores are below the threshold, S4 is paused, and only S2 and S3 monitoring are maintained.

11. A method for target recognition in multi-person dialogue scenarios based on wearable devices, characterized in that: Includes the following steps: A1. Multimodal data synchronous acquisition steps: Acquire visual data through the front-facing camera of the wearable device; Multiple audio signals are acquired using a microphone array, and the azimuth angle of the sound source is calculated based on the time difference of arrival principle. θ_source = arcsin(c × Δt / d_mic) The wearer's voice is collected via bone conduction microphone and converted into text through speech recognition; all data is synchronized and aligned using a unified timestamp. A2. Steps for target recognition in multi-person scenarios: Perform semantic dimension analysis, spatiotemporal dimension analysis, and response dimension analysis on N candidate sound sources, where: The semantic dimension analysis calculates the semantic similarity between the wearer's speech recognition text and the speech recognition text of each candidate sound source, and obtains the semantic dimension score R_sem(i). The spatiotemporal dimension analysis comprehensively examines the speaker alternation pattern, gaze direction, and spatial distance to obtain the spatiotemporal dimension score R_st(i); The response dimension analysis examines the wearer's response frequency, response delay, and response mode to each candidate sound source's speech, and obtains the response dimension score R_rsp(i); The scores from the three dimensions are weighted and fused to calculate the overall score for each candidate sound source: S_target(i) = α × R_sem(i) + β × R_st(i) + γ × R_rsp(i) Where α + β + γ = 1, R_sem(i), R_st(i), and R_rsp(i) are all normalized to the [0,1] interval; the weight coefficients α, β, and γ are determined by a dynamic adaptive algorithm based on signal quality: the semantic channel signal quality Q_sem, the spatiotemporal channel signal quality Q_st, and the response channel signal quality Q_rsp are calculated respectively, and the weights of each dimension are calculated by normalization. α = Q_sem / (Q_sem + Q_st + Q_rsp) β = Q_st / (Q_sem + Q_st + Q_rsp) γ = Q_rsp / (Q_sem + Q_st + Q_rsp) This allows the weights to be adaptively adjusted in real time according to environmental conditions; when the comprehensive score S_target(i) of a candidate sound source is the highest among all candidate sound sources and exceeds the preset dialogue confirmation threshold, the candidate sound source is confirmed as the current dialogue target.

Images