Emotional diary intelligent glasses system driven by multi-mode intelligent body

Abstract

The invention discloses an emotional diary intelligent glasses system driven by a multi-modal agent. The emotional diary intelligent glasses system comprises a multi-modal perception layer, a unified vector space processing layer and an agent collaborative decision-making layer. The multi-modal sensing layer constructs a full-dimensional sensing network by integrating a camera, a microphone array and an eye movement tracker, and collects image, sound and eye movement information; the unified vector space processing layer realizes cross-modal information unified representation by using a multi-modal large model; the agent collaborative decision-making layer comprises an emotion perception agent, a scene understanding agent, a diary generation agent and a personalized adaptive agent. According to the method, deep understanding of the emotional state and the environment scene of the user is realized through a multi-modal fusion perception and agent cooperation mechanism, the personalized emotional diary is generated in combination with chain thinking and reflection, and the content insight and the user experience are effectively improved.

Description

Technical Field

[0001] This invention relates to a smart glasses system, and more specifically to a multimodal intelligent agent-driven emotional diary smart glasses system. Background Technology

[0002] With the rapid development of modern technology, wearable devices have become an important part of smart living. Among them, smart glasses, as an emerging wearable technology, are widely favored by users because they can provide real-time image capture capabilities. In practical applications, the control method of these smart glasses is crucial to the user experience.

[0003] Existing technology includes an invention patent with publication number CN119556479A, entitled "Control Method and System for Smart Glasses with Camera Function," which discloses a method of collecting external image data by setting a camera module in smart glasses and then working in conjunction with other sensors to improve the portability and user experience of smart glasses. However, this method still mainly focuses on information display and simple interaction, with limited capabilities in emotional cognition and content generation. Traditional diary recording relies on manual input, which is inefficient and highly subjective. Current emotion recognition technology is limited to single-modal analysis and struggles to fully grasp true emotions. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a multimodal intelligent agent-driven emotional diary smart glasses system, which achieves real-time emotion perception, scene understanding, and personalized diary generation through an edge computing-cloud collaboration architecture.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal intelligent agent-driven emotional diary smart glasses system, characterized in that it includes: The multimodal perception layer is a full-dimensional perception network built by integrating cameras, microphone arrays, and eye trackers, used to collect image information, sound information, and eye movement information; The unified vector space processing layer uses a multimodal large model to achieve unified cross-modal information representation of the multimodal information collected by the multimodal perception layer; The agent collaborative decision-making layer includes: Emotion-aware intelligent agents: used to analyze users' emotional states and build emotional knowledge graphs; Scene-understanding agents are used to analyze the environment and activities and build context for user experiences. Diary-generating AI: Used to generate insightful content through chain-thinking and reflection mechanisms; Personalized Adaptive Agent: Used to continuously optimize content style and preferences.

[0006] As a further improvement of the present invention, each agent in the agent collaborative decision-making layer is based on the attention mechanism of the Transformer architecture. The agents work together through three methods: vector retrieval, semantic understanding, and conditional probability reasoning, forming a closed loop of information flow and decision flow.

[0007] As a further improvement of the present invention, the intelligent agent collaborative decision-making layer is represented by the following mathematical model: Suppose there are n agents $A = \{A_1,A_2,...,A_n\}$ in the system. Each agent $A_i$ observes a state of $s_i^t$ at time t and takes an action of $a_i^t$. The global state is $S^t$. The agent cooperative decision-making process can be formalized as follows: $a_i^t=\pi_i(s_i^t,M_i^t,C^t)$ Where $\pi_i$ is the policy function of agent $A_i$, $M_i^t$ is the internal memory state of the agent, and $C^t$ is the context information shared among agents through the attention mechanism, defined as: $C^t=\text{Attention}(Q_i^t,K^t,V^t)$ $Q_i^t=W_Q^i\cdots_i^t$ $K^t=[W_K^1\cdots_1^t;W_K^2\cdots_2^t;...;W_K^n\cdots_n^t]$ $V^t=[W_V^1\cdots_1^t;W_V^2\cdots_2^t;...;W_V^n\cdots_n^t]$ Where $W_Q^i$, $W_K^j$, and $W_V^j$ are learnable parameter matrices.

[0008] As a further improvement of the present invention, the multi-agent consensus mechanism in the agent collaborative decision-making layer optimizes the overall decision-making efficiency through the following objective function: $J(\pi_1,\pi_2,...,\pi_n)=\mathbb{E}[\sum_{t=0}^{T}\gamma^t\cdotR(S^t,a_1^t,a_2^t,...,a_n^t)]$ Where $R$ is the global reward function and $\gamma$ is the discount factor.

[0009] As a further improvement of the present invention, the unified vector space processing layer adopts an end-to-end pre-trained multimodal large model to establish a unified vector space for processing multimodal information, specifically including: Modality-specific encoding: Preliminary feature extraction is performed using a specialized encoder; Cross-modal alignment: achieving semantic-level modal alignment through contrastive learning; Multi-head attention interaction: Enables the fusion of cross-modal information such as vision, speech, and motion; Unified semantic representation: Constructing a unified vector that contains multimodal information.

[0010] As a further improvement of the present invention, the mathematical expression of the multimodal information processing is as follows: Given visual input $X_v$, audio input $X_a$, eye-tracking input $X_e$, and motion input $X_m$, we first obtain modality-specific representations through their respective encoders: $H_v=E_v(X_v)$ $H_a=E_a(X_a)$ $H_e=E_e(X_e)$ $H_m=E_m(X_m)$ Then, cross-modal alignment maps features from different modalities to a unified semantic space: $Z_v=\phi_v(H_v)$ $Z_a=\phi_a(H_a)$ $Z_e=\phi_e(H_e)$ $Z_m=\phi_m(H_m)$ Where $\phi_v,\phi_a,\phi_e,\phi_m$ are the corresponding mapping functions, which are optimized through contrastive learning to minimize the following contrastive loss: $\mathcal{L}_{contrast}=-\sum_{i=1}^{B}\log\frac{e^{sim(Z_v^i,Z_a^i) / \tau}}{\sum_{j=1}^{B}e^{sim(Z_v^i,Z_a^j) / \tau}}$ Where $B$ is the batch size, $sim$ is the cosine similarity function, and $\tau$ is the temperature parameter.

[0011] Finally, the multimodal fusion representation $Z_{fused}$ is calculated through a cross-attention mechanism: $Z_{fused}=\text{MultiheadAttention}(Z_v,[Z_v;Z_a;Z_e;Z_m],[Z_v;Z_a;Z_e;Z_m])$.

[0012] As a further improvement of the present invention, the emotion-sensing agent adopts a hierarchical design, specifically including: Base layer: Zero-shot sentiment recognition, supporting real-time multimodal sentiment analysis; Reasoning layer: Complex sentiment analysis, taking into account socio-cultural background and historical information; Memory layer: Constructs an emotional knowledge graph to record emotional patterns and triggering factors.

[0013] As a further improvement of the present invention, the emotional state label and calculation processed by the emotion-perceiving intelligent agent are as follows: Emotional state $E$ is represented as a multidimensional vector: $E=[e_1,e_2,...,e_k]$ Where $e_i$ represents the intensity value of different dimensions of emotion.

[0014] Emotion recognition based on multimodal input can be represented as: $E=f_{\theta}(Z_{fused})$ Where $f_{\theta}$ is the emotion recognition function with parameter $\theta$.

[0015] For complex emotional reasoning, a chain-thinking mechanism is employed: $E_{complex}=\text{CoT}(E,C,H)$ Where $C$ represents the current context information and $H$ represents the historical sentiment state sequence, the CoT process can be expanded as follows: $T_1=g_1(Z_{fused},C)$ $T_2=g_2(T_1,C,H)$ $T_3=g_3(T_2,C,H)$ $E_{complex}=g_4(T_3,C,H)$ In the sentiment knowledge graph $G=(V,R)$, the node set $V$ represents sentiment states and triggering events, and the edge set $R$ represents the relationships between them. The dynamic update process of the graph is as follows: $G^{t+1}=\text{Update}(G^t,E^t,C^t)$ Here, $\text{Update}$ is an update function of the graph based on newly observed sentiment state $E^t$ and context $C^t$.

[0016] As a further improvement of the present invention, the process for generating the diary-generating intelligent agent is as follows: Planning phase: TaskDecomposition breaks down the diary generation task into a set of subtasks and filters key events through an importance scoring mechanism; Generation phase: Content is generated in parallel using four modules: scene reconstruction, event narration, emotion expression, and insight generation; Optimization phase: Through a self-assessment and correction cycle, ensure content quality and consistency; Reflection phase: Analyze the connection between the emotions and history of the day to provide insights into personal growth.

[0017] As a further improvement of the present invention, the mathematical representation of the diary generation process of the diary generation agent is as follows: First, the event importance scoring function is used to filter key events: $\text{Score}(e_i)=\alpha\cdot\text{Emotion\_Intensity}(e_i)+\beta\cdot\text{Novelty}(e_i)+\gamma\cdot\text{Duration}(e_i)$ Where $\alpha,\beta,\gamma$ are weight parameters; Content generation uses a Transformer-based language model, with conditional generation probabilities as follows: $P(y|X,E,C)=\prod_{t=1}^{T}P(y_t|y_{ <t},X,E,C)$ Where $y$ is the generated diary text, $X$ is the multimodal input, $E$ is the emotional state, and $C$ is the contextual information; Chain thinking-enhanced journal generation can be represented as: $S_1=\text{Describe}(X)$ $S_2=\text{Narrate}(X,S_1)$ $S_3=\text{Express}(E,S_2)$ $S_4=\text{Reflect}(S_3,H)$ $y=\text{Compose}(S_1,S_2,S_3,S_4)$ The self-assessment mechanism evaluates the generated content using the following quality scoring function: $\text{Quality}(y)=w_1\cdot\text{Coherence}(y)+w_2\cdot\text{Relevance}(y,X)+w_3\cdot\text{Emotion\_Alignment}(y,E)+w_4\cdot\text{Insight}(y)$ Where $w_1, w_2, w_3, w_4$ are the weights of each evaluation dimension.

[0018] The beneficial effects of this invention are that, through the setup of a multimodal perception layer, a unified vector space processing layer, and an intelligent agent collaborative decision-making layer, it can effectively achieve real-time emotion perception, scene understanding, and personalized diary generation through an edge computing-cloud collaborative architecture. Furthermore, this generation method has the following beneficial effects: A multimodal understanding framework in a unified vector space enables deep integration of emotion, scene, and language; Based on a large model, the agent collaborative decision-making mechanism allows each agent to make independent decisions and collaborate efficiently. Chained thinking and reflection enhance content generation, going beyond simple recording to provide deep insights; A continuously learning, personalized adaptation system ensures that the diary style becomes increasingly tailored to the user's characteristics with each use. Attached Figure Description

[0019] Figure 1 This is a block diagram of the multimodal intelligent agent-driven emotional diary smart glasses system of the present invention; Figure 2 A flowchart illustrating the unified representation of cross-modal information; Figure 3 A flowchart illustrating the process of an emotion-aware intelligent agent; Figure 4 A schematic diagram illustrating the process of generating an intelligent agent for diary entries; Figure 5 A flowchart illustrating the process of personalization adaptation. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings.

[0021] Reference Figures 1 to 5 As shown in this embodiment, a multimodal intelligent agent-driven emotional diary smart glasses system adopts a "lightweight edge, heavy cloud" computing architecture. The smart glasses are responsible for data collection and preprocessing, while the edge server or cloud is responsible for deep computing tasks. Specifically, it includes: Multimodal perception layer: Integrates cameras, microphone arrays, and eye trackers to construct a full-dimensional perception network; Unified Vector Space Processing Layer: Employs a multimodal large model to achieve unified representation of cross-modal information; Intelligent agent collaborative decision-making layer: Emotion-aware intelligent agents: analyze users' emotional states and build emotional knowledge graphs; Scene-understanding intelligent agents: analyze the environment and activities to construct context for user experiences; Diary-generating AI: Generates insightful content through chain-thinking and reflection mechanisms; Personalized Adaptive Agents: Enable continuous optimization of content style and preferences.

[0022] The aforementioned agent collaborative decision-making framework, based on the attention mechanism of the Transformer architecture, achieves the following key capabilities: Autonomous task decomposition and planning; Efficient communication based on shared vector space; Adaptive resource allocation dynamically adjusts computing resources based on task complexity; A multi-agent consensus mechanism ensures consistency in decision-making.

[0023] Intelligent agents collaborate through three methods: vector retrieval, semantic understanding, and conditional probability reasoning, forming a closed loop of information flow and decision flow.

[0024] The collaborative decision-making of intelligent agents can be represented by the following mathematical model: Suppose there are n agents $A = \{A_1,A_2,...,A_n\}$ in the system. Each agent $A_i$ observes a state of $s_i^t$ at time t and takes an action of $a_i^t$. The global state is $S^t$. The agent cooperative decision-making process can be formalized as follows: $a_i^t=\pi_i(s_i^t,M_i^t,C^t)$ Where $\pi_i$ is the policy function of agent $A_i$, $M_i^t$ is the internal memory state of the agent, and $C^t$ is the context information shared among agents through the attention mechanism, defined as: $C^t=\text{Attention}(Q_i^t,K^t,V^t)$ $Q_i^t=W_Q^i\cdots_i^t$ $K^t=[W_K^1\cdots_1^t;W_K^2\cdots_2^t;...;W_K^n\cdots_n^t]$ $V^t=[W_V^1\cdots_1^t;W_V^2\cdots_2^t;...;W_V^n\cdots_n^t]$ Where $W_Q^i$, $W_K^j$, and $W_V^j$ are learnable parameter matrices.

[0025] Multi-agent consensus mechanisms optimize overall decision-making efficiency through the following objective function: $J(\pi_1,\pi_2,...,\pi_n)=\mathbb{E}[\sum_{t=0}^{T}\gamma^t\cdotR(S^t,a_1^t,a_2^t,...,a_n^t)]$ Where $R$ is the global reward function and $\gamma$ is the discount factor.

[0026] Building upon the aforementioned agent-based collaborative work, this system overcomes the limitations of traditional modal fusion by employing end-to-end pre-trained multimodal large models to establish a unified vector space. The core processing flow includes: 1. Modality-specific encoding: Preliminary feature extraction is performed using a specialized encoder (ViT / Whisper); 2. Cross-modal alignment: Achieving semantic-level modal alignment through contrastive learning; 3. Multi-head attention interaction: Enables the fusion of cross-modal information such as vision, speech, and motion; 4. Unified semantic representation: Construct a unified vector that contains multimodal information.

[0027] The model architecture adopts a multi-stream design, with each modality corresponding to an Encoder stream. Information exchange is achieved through a cross-attention mechanism, and finally, a unified Decoder is used to generate modality-independent semantic representations.

[0028] The mathematical expression for multimodal information processing is as follows: Given visual input $X_v$, audio input $X_a$, eye-tracking input $X_e$, and motion input $X_m$, we first obtain modality-specific representations through their respective encoders: $H_v=E_v(X_v)$ $H_a=E_a(X_a)$ $H_e=E_e(X_e)$ $H_m=E_m(X_m)$ Then, cross-modal alignment maps features from different modalities to a unified semantic space: $Z_v=\phi_v(H_v)$ $Z_a=\phi_a(H_a)$ $Z_e=\phi_e(H_e)$ $Z_m=\phi_m(H_m)$ Where $\phi_v,\phi_a,\phi_e,\phi_m$ are the corresponding mapping functions, which are optimized through contrastive learning to minimize the following contrastive loss: $\mathcal{L}_{contrast}=-\sum_{i=1}^{B}\log\frac{e^{sim(Z_v^i,Z_a^i) / \tau}}{\sum_{j=1}^{B}e^{sim(Z_v^i,Z_a^j) / \tau}}$ Where $B$ is the batch size, $sim$ is the cosine similarity function, and $\tau$ is the temperature parameter.

[0029] Finally, the multimodal fusion representation $Z_{fused}$ is calculated through a cross-attention mechanism: $Z_{fused}=\text{MultiheadAttention}(Z_v,[Z_v;Z_a;Z_e;Z_m],[Z_v;Z_a;Z_e;Z_m])$.

[0030] Based on the unified representation of cross-modal information completed above, this embodiment provides the following emotion-aware agent, which adopts a hierarchical design: 1. Base layer: Zero-shot sentiment recognition, supporting real-time multimodal sentiment analysis; 2. Reasoning layer: Complex sentiment analysis, considering socio-cultural background and historical information; 3. Memory layer: Construct an emotional knowledge graph to record emotional patterns and triggering factors.

[0031] Based on the above three-layer structure, complex sentiment reasoning based on ICL and CoT can be realized, overcoming the limitations of traditional classification methods; the dynamic update mechanism of the sentiment knowledge graph enables sentiment pattern learning; and the sentiment prediction model predicts possible sentiment responses in specific situations based on historical data. Specifically, the representation and calculation of sentiment states can be formalized as follows: Emotional state $E$ is represented as a multidimensional vector: $E=[e_1,e_2,...,e_k]$ Where $e_i$ represents the intensity value of different dimensions of emotion (such as joy, sadness, anger, etc.).

[0032] Emotion recognition based on multimodal input can be represented as: $E=f_{\theta}(Z_{fused})$ Where $f_{\theta}$ is the emotion recognition function with parameter $\theta$.

[0033] For complex emotional reasoning, the Chain Thinking (CoT) mechanism is employed: $E_{complex}=\text{CoT}(E,C,H)$ Where $C$ represents the current context information, and $H$ represents the historical sentiment state sequence. The CoT process can be expanded as follows: $T_1=g_1(Z_{fused},C)$(Preliminary sentiment assessment) $T_2=g_2(T_1,C,H)$ (Context-dependent analysis) $T_3=g_3(T_2,C,H)$(Social and cultural considerations) $E_{complex}=g_4(T_3,C,H)$(Final emotional state) In the sentiment knowledge graph $G=(V,R)$, the node set $V$ represents sentiment states and triggering events, and the edge set $R$ represents the relationships between them. The dynamic update process of the graph is as follows: $G^{t+1}=\text{Update}(G^t,E^t,C^t)$ Where $\text{Update}$ is an update function of the graph based on newly observed sentiment state $E^t$ and context $C^t$.

[0034] Based on the sentiment analysis performed by the aforementioned sentiment-perceiving agent, this embodiment employs a diary-generating agent to generate insightful content, specifically as follows: This embodiment of the diary generation agent integrates the latest LLM technology to form a four-stage generation process: 1. Planning phase: TaskDecomposition breaks down the journal generation task into a set of subtasks and filters key events through an importance scoring mechanism; 2. Generation Phase: Content is generated in parallel using four modules: scene reconstruction, event narration, emotional expression, and insight generation; 3. Optimization Phase: Through a self-assessment and correction cycle, ensure content quality and consistency; 4. Reflection Phase: Analyze the connection between the emotions and history of the day to provide insights into personal growth.

[0035] Through the above four stages, the introduction of the Chain of Thought (CoT) and self-reflection mechanisms enables the system not only to record facts but also to analyze emotional patterns and provide regulatory strategies, forming a complete cognitive chain of "recording-analysis-insight." This can be represented mathematically as follows: First, the event importance scoring function is used to filter key events: $\text{Score}(e_i)=\alpha\cdot\text{Emotion\_Intensity}(e_i)+\beta\cdot\text{Novelty}(e_i)+\gamma\cdot\text{Duration}(e_i)$ Where $\alpha,\beta,\gamma$ are weight parameters.

[0036] Content generation uses a Transformer-based language model, with conditional generation probabilities as follows: $P(y|X,E,C)=\prod_{t=1}^{T}P(y_t|y_{ <t},X,E,C)$ Where $y$ is the generated diary text, $X$ is the multimodal input, $E$ is the sentiment state, and $C$ is the context information.

[0037] Chain Thinking (CoT) enhanced journal generation can be represented as: $S_1=\text{Describe}(X)$(Scene Description) $S_2=\text{Narrate}(X,S_1)$(Event Narration) $S_3=\text{Express}(E,S_2)$(Emotional Expression) $S_4=\text{Reflect}(S_3,H)$(Reflection and Insight) $y=\text{Compose}(S_1,S_2,S_3,S_4)$(Final Diary) The self-assessment mechanism evaluates the generated content using the following quality scoring function: $\text{Quality}(y)=w_1\cdot\text{Coherence}(y)+w_2\cdot\text{Relevance}(y,X)+w_3\cdot\text{Emotion\_Alignment}(y,E)+w_4\cdot\text{Insight}(y)$ Where $w_1, w_2, w_3, w_4$ are the weights of each evaluation dimension.

[0038] In addition to the layers described above, this embodiment also provides a personalized adaptation system to further enhance the user experience. The personalized adaptation employs a three-layer reinforcement learning framework: 1. Perception Layer: Captures user editing behavior and feedback signals; 2. Strategy Layer: Optimize content generation strategies based on user feedback; 3. Evaluation layer: Self-evaluation of the quality of generated content.

[0039] The system constructs an accurate user preference model by analyzing user editing patterns, interaction behaviors, and explicit feedback. It employs few-shot learning to quickly adapt to user styles and uses comparative learning to identify differences between user-generated content and system-generated content, continuously adjusting its generation strategy accordingly.

[0040] The mathematical representation of the personalized adaptation process is as follows: The user preference model $U$ can be represented as: $U=\{u_{style},u_{focus},u_{format},u_{depth}\}$ Each component represents a user's preferences across different dimensions.

[0041] Feedback signals based on user editing behavior are defined as follows: $F=\text{EditDistance}(y_{generated},y_{edited})$ The system continuously optimizes the generation strategy $\pi_{\theta}$ through reinforcement learning, with the objective function being: $J(\theta)=\mathbb{E}_{X,E,C,U}[r(y_{\theta},y_{user})]$ Where $r$ is the reward function, calculated as follows: $r(y_{\theta},y_{user})=-\text{Distance}(y_{\theta},y_{user})+\lambda\cdot\text{Quality}(y_{\theta})$ Parameter updates are performed using the policy gradient method: $\theta_{t+1}=\theta_t+\alpha\nabla_{\theta}J(\theta)|_{\theta_t}$ The few-shot learning adaptation mechanism can be represented as: $\theta_{personalized}=\theta_{base}+\Delta\theta$ Where $\Delta\theta$ is obtained through meta-learning on a small number of user-edited samples: $\Delta\theta=\text{MetaLearner}(\{(X_i,y_{user,i})\}_{i=1}^k)$.

[0042] Accordingly, the system in this embodiment also employs privacy protection technology to protect user privacy, specifically: Adopting a multi-layered privacy protection architecture: 1. Local processing layer: Sensitive data is processed locally on the device first; 2. Federated learning layer: Model parameters are transmitted instead of raw data, protecting user privacy; 3. Differential privacy layer: Add carefully designed noise to mask individual data characteristics; 4. Encrypted storage layer: End-to-end encryption ensures data security.

[0043] The innovation lies in achieving a balance between privacy protection and AI performance, ensuring that system functionality is not degraded due to privacy protection.

[0044] The mathematical representation of the privacy protection mechanism is as follows: Differential privacy processing can be represented as: $f_{DP}(X)=f(X)+\mathcal{N}(0,S^2\cdot\sigma^2)$ Where $f$ is the primitive function, $S$ is the sensitivity, and $\sigma$ is the noise ratio.

[0045] The federated learning process can be represented as: $\theta_{global}=\frac{1}{N}\sum_{i=1}^{N}w_i\cdot\theta_i$ Where $\theta_i$ are the parameters of the local model trained on device $i$, and $w_i$ are the weights.

[0046] Encrypted data storage uses homomorphic encryption, allowing computation to be performed within the encrypted domain: $E(x+y)=E(x)\circE(y)$ $E(a\cdotx)=E(x)^a$ Where $E$ is the encryption function and $\circ$ is the operation on the encryption field.

[0047] In summary, the multimodal intelligent agent-driven emotional diary smart glasses system of this embodiment can achieve real-time emotion perception, scene understanding, and personalized diary generation through an edge computing-cloud collaboration architecture.

[0048] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal intelligent agent-driven emotional diary smart glasses system, characterized in that: include: The multimodal perception layer is a full-dimensional perception network built by integrating cameras, microphone arrays, and eye trackers, used to collect image information, sound information, and eye movement information; The unified vector space processing layer uses a multimodal large model to achieve unified cross-modal information representation of the multimodal information collected by the multimodal perception layer; The agent collaborative decision-making layer includes: Emotion-aware intelligent agents: used to analyze users' emotional states and build emotional knowledge graphs; Scene-understanding agents are used to analyze the environment and activities and build context for user experiences. Diary-generating AI: Used to generate insightful content through chain-thinking and reflection mechanisms; Personalized Adaptive Agent: Used to continuously optimize content style and preferences.

2. The multimodal intelligent agent-driven emotional diary smart glasses system according to claim 1, characterized in that: The agents in the collaborative decision-making layer are based on the attention mechanism of the Transformer architecture. The agents work together through three methods: vector retrieval, semantic understanding, and conditional probability reasoning, forming a closed loop of information flow and decision flow.

3. The multimodal intelligent agent-driven emotional diary smart glasses system according to claim 2, characterized in that: The agent collaborative decision-making layer is represented by the following mathematical model: Suppose there are n agents $A = \{A_1, A_2, ..., A_n\}$ in the system. Each agent $A_i$ observes a state of $s_i^t$ at time t and takes an action of $a_i^t$. The global state is $S^t$. The agent cooperative decision-making process can be formalized as follows: $a_i^t = \pi_i(s_i^t, M_i^t, C^t)$ Where $\pi_i$ is the policy function of agent $A_i$, $M_i^t$ is the internal memory state of the agent, and $C^t$ is the context information shared among agents through the attention mechanism, defined as: $C^t = \text{Attention}(Q_i^t, K^t, V^t)$ $Q_i^t = W_Q^i \cdot s_i^t$ $K^t = [W_K^1 \cdot s_1^t; W_K^2 \cdot s_2^t; ...; W_K^n \cdot s_n^t]$ $V^t = [W_V^1 \cdot s_1^t; W_V^2 \cdot s_2^t; ...; W_V^n \cdot s_n^t]$ Where $W_Q^i$, $W_K^j$, $W_V^j$ are learnable parameter matrices.

4. The multimodal intelligent agent-driven emotional diary smart glasses system according to claim 3, characterized in that: The multi-agent consensus mechanism in the agent collaborative decision-making layer optimizes overall decision-making efficiency through the following objective function: $J(\pi_1, \pi_2, ..., \pi_n) = \mathbb{E}[\sum_{t=0}^{T} \gamma^t \cdot R(S^t, a_1^t, a_2^t, ..., a_n^t)]$ Where $R$ is the global reward function and $\gamma$ is the discount factor.

5. The multimodal agent-driven emotional diary smart glasses system according to any one of claims 1 to 4, characterized in that: The unified vector space processing layer employs an end-to-end pre-trained multimodal large model to establish a unified vector space for processing multimodal information, specifically including: Modality-specific encoding: Preliminary feature extraction is performed using a specialized encoder; Cross-modal alignment: achieving semantic-level modal alignment through contrastive learning; Multi-head attention interaction: Enables the fusion of cross-modal information such as vision, speech, and motion; Unified semantic representation: Constructing a unified vector that contains multimodal information.

6. The multimodal intelligent agent-driven emotional diary smart glasses system according to claim 5, characterized in that: The mathematical expression for the multimodal information processing is as follows: Given visual input $X_v$, audio input $X_a$, eye-tracking input $X_e$, and motion input $X_m$, we first obtain modality-specific representations through their respective encoders: $H_v = E_v(X_v)$ $H_a = E_a(X_a)$ $H_e = E_e(X_e)$ $H_m = E_m(X_m)$ Then, cross-modal alignment maps features from different modalities to a unified semantic space: $Z_v = \phi_v(H_v)$ $Z_a = \phi_a(H_a)$ $Z_e = \phi_e(H_e)$ $Z_m = \phi_m(H_m)$ Where $\phi_v, \phi_a, \phi_e, \phi_m$ are the corresponding mapping functions, which are optimized through contrastive learning to minimize the following contrastive loss: $\mathcal{L}_{contrast} = -\sum_{i=1}^{B} \log \frac{e^{sim(Z_v^i, Z_a^i) / \tau}}{\sum_{j=1}^{B} e^{sim(Z_v^i, Z_a^j) / \tau}}$ Where $B$ is the batch size, $sim$ is the cosine similarity function, and $\tau$ is the temperature parameter: Finally, the multimodal fusion representation $Z_{fused}$ is calculated using a cross-attention mechanism: $Z_{fused} = \text{MultiheadAttention}(Z_v, [Z_v; Z_a; Z_e; Z_m], [Z_v;Z_a; Z_e; Z_m])$.

7. The multimodal agent-driven emotional diary smart glasses system according to any one of claims 1 to 4, characterized in that: The emotion-aware intelligent agent adopts a hierarchical design, specifically including: Base layer: Zero-shot sentiment recognition, supporting real-time multimodal sentiment analysis; Reasoning layer: Complex sentiment analysis, taking into account socio-cultural background and historical information; Memory layer: Constructs an emotional knowledge graph to record emotional patterns and triggering factors.

8. The multimodal intelligent agent-driven emotional diary smart glasses system according to claim 7, characterized in that: The emotional state label and calculation processed by the emotion-perceiving agent are as follows: Emotional state $E$ is represented as a multidimensional vector: $E = [e_1, e_2, ..., e_k]$ Where $e_i$ represents the intensity value of different dimensions of emotion.

9. Emotion recognition based on multimodal input can be represented as: $E = f_{\theta}(Z_{fused})$ Where $f_{\theta}$ is the sentiment recognition function with parameter $\theta$; For complex emotional reasoning, a chain-thinking mechanism is employed: $E_{complex} = \text{CoT}(E, C, H)$ Where $C$ represents the current context information and $H$ represents the historical sentiment state sequence, the CoT process can be expanded as follows: $T_1 = g_1(Z_{fused}, C)$ $T_2 = g_2(T_1, C, H)$ $T_3 = g_3(T_2, C, H)$ $E_{complex} = g_4(T_3, C, H)$ In the sentiment knowledge graph $G = (V, R)$, the node set $V$ represents sentiment states and triggering events, and the edge set $R$ represents the relationships between them. The dynamic update process of the graph is as follows: $G^{t+1} = \text{Update}(G^t, E^t, C^t)$ in, $\text{Update}$ is an update function for the graph based on newly observed sentiment state $E^t$ and context $C^t$; The multimodal intelligent agent-driven emotional diary smart glasses system according to any one of claims 1 to 4 is characterized in that: the diary generation intelligent agent generation process is as follows: Planning phase: Task Decomposition breaks down the diary generation task into a set of subtasks and filters key events through an importance scoring mechanism; Generation phase: Content is generated in parallel using four modules: scene reconstruction, event narration, emotion expression, and insight generation; Optimization phase: Through a self-assessment and correction cycle, ensure content quality and consistency; Reflection phase: Analyze the connection between the emotions and history of the day to provide insights into personal growth.

10. The multimodal intelligent agent-driven emotional diary smart glasses system according to claim 9, characterized in that: The mathematical representation of the diary generation process of the diary-generating agent is as follows: First, the event importance scoring function is used to filter key events: $\text{Score}(e_i) = \alpha \cdot \text{Emotion\_Intensity}(e_i) + \beta\cdot \text{Novelty}(e_i) + \gamma \cdot \text{Duration}(e_i)$ Where $\alpha, \beta, \gamma$ are weight parameters; Content generation uses a Transformer-based language model, with conditional generation probabilities as follows: $P(y|X, E, C) = \prod_{t=1}^{T} P(y_t | y_{ Where $y$ is the generated diary text, $X$ is the multimodal input, $E$ is the emotional state, and $C$ is the contextual information; Chain thinking-enhanced journal generation can be represented as: $S_1 = \text{Describe}(X)$ $S_2 = \text{Narrate}(X, S_1)$ $S_3 = \text{Express}(E, S_2)$ $S_4 = \text{Reflect}(S_3, H)$ $y = \text{Compose}(S_1, S_2, S_3, S_4)$ The self-assessment mechanism evaluates the generated content using the following quality scoring function: $\text{Quality}(y) = w_1 \cdot \text{Coherence}(y) + w_2 \cdot \text{Relevance}(y, X) + w_3 \cdot \text{Emotion\_Alignment}(y, E) + w_4 \cdot \text{Insight}(y)$ Where $w_1, w_2, w_3, w_4$ are the weights of each evaluation dimension.

Images