A multimodal interactive smart glasses control method and system

Abstract

This invention provides a multimodal interactive smart glasses control method and system, relating to the field of smart interactive device technology. The multimodal interactive smart glasses control method includes: acquiring multimodal perception data, wherein the multimodal perception data includes at least two of the following: audio information, image information, time information, geographic location information, inertial motion information, spatial posture information, eye-tracking information, and biometric information; constructing a dynamic context model based on the multimodal perception data; outputting potentially needed auxiliary information and its priority level based on the dynamic context model, combined with the user's historical behavioral habits and / or preset preferences; and determining the information output method based on the auxiliary information and its priority level. This invention can proactively provide appropriate information to the user at the appropriate time and in the appropriate manner.

Description

Technical Field

[0001] This invention relates to the field of intelligent interactive device technology, specifically to a multimodal interactive intelligent glasses control method and system. Background Technology

[0002] Existing smart glasses are mostly passive response devices, their operation heavily reliant on explicit user commands, primarily performing single, pre-defined tasks such as taking photos, answering calls, and voice navigation. In a typical interaction, users must actively trigger the device through specific wake words, physical buttons, or touch gestures. To further enhance the interactive experience of smart glasses, a new type of glasses is needed that overcomes the limitations of the current "passive response" approach.

[0003] The above description of the problem discovery process is only used to assist in understanding the technical solution of the present invention and does not mean that the above content is prior art. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal interactive smart glasses control method and system to solve the technical problem that smart glasses in the prior art can only passively respond to a single command.

[0005] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a multimodal interactive smart glasses control method, comprising: acquiring multimodal perception data, wherein the multimodal perception data includes at least two of the following: audio information, image information, time information, geographic location information, inertial motion information, spatial posture information, eye-tracking information, and biometric information; constructing a dynamic context model based on the multimodal perception data; outputting potential auxiliary information needed by the user and the priority level of the auxiliary information based on the dynamic context model and, in conjunction with the user's historical behavioral habits and / or preset preferences; and determining the information output method based on the auxiliary information and the priority level.

[0006] Optionally, acquiring multimodal perception data includes: acquiring audio information through a microphone array mounted on the glasses, acquiring image information through a camera, acquiring inertial motion information and spatial attitude information through an inertial measurement unit, acquiring eye-tracking information through an eye-tracking sensor, and acquiring biometric information through a biosensor; wherein the microphone array includes at least one bone conduction microphone for picking up bone vibration signals generated when the user speaks.

[0007] Optionally, the construction of the dynamic context model includes: performing fusion analysis on the multimodal perception data to identify the current environmental scene type, dialogue topic and key entities, as well as the user's attention state and cognitive load level in real time; and generating a dynamic context model containing three dimensions: environmental context, social dialogue flow and user's internal state based on the identification results.

[0008] Optionally, identifying the user's attention state and cognitive load level includes: judging the user's current level of attention concentration and cognitive busyness based on the gaze duration, blink frequency, and changes in gaze point in the eye-tracking information, combined with changes in head posture in the inertial motion information.

[0009] Optionally, the biometric information includes at least one of heart rate information, heart rate variability information, skin electrical activity information, and voiceprint feature information; the voiceprint feature information is obtained by analyzing the bone vibration signal picked up by the bone conduction microphone.

[0010] Optionally, the step of outputting the auxiliary information that the user may need and the priority level of the auxiliary information includes: generating at least one piece of auxiliary information based on the current scenario and user state identified by the dynamic scenario model, combined with the user's historical behavior habits and / or preset preferences; and calculating the priority level of the auxiliary information based on the closeness of the correlation between the auxiliary information and the current scenario, the time urgency, and the user's preset importance weight.

[0011] Optionally, the auxiliary information includes at least one of identity prompt information, dialogue summary information, task reminder information, physiological state warning information, and navigation guidance information; the historical behavior habits include records of information types actively queried by the user in different scenarios, records of response rates to different types of prompts, and information on frequently contacted persons and high-frequency locations set by the user.

[0012] Optionally, determining the information output mode based on the auxiliary information and the priority level includes: selecting at least one preset information output mode from a variety of modes based on the priority level and the user's current cognitive load level and environmental noise level in the dynamic context model; the variety of information output modes include bone conduction private voice broadcast, visual field dim light text prompts, prompt sound broadcasts, and silent storage to local memory.

[0013] Optionally, determining the information output method includes: if the dynamic context model determines that there is a risk of privacy leakage in the current environment, or the priority level of the auxiliary information corresponds to a high privacy level, then the bone conduction private voice broadcast is selected as the information output method; if the dynamic context model determines that the user is in a state of high cognitive load, then the output frequency of non-urgent auxiliary information is reduced, or it is temporarily stored in the local memory for the user to review later.

[0014] Optionally, the method further includes: driving a corresponding actuator to present the auxiliary information according to the determined information output mode; the actuator includes at least one of a bone conduction speaker, a display module, an indicator light, and a vibration module; wherein the bone conduction speaker and the bone conduction microphone are integrated on the inner side of the temple of the eyeglasses, and are used to play auxiliary information and collect bone vibration signals, respectively.

[0015] Secondly, embodiments of the present invention provide a multimodal interactive smart glasses control system, comprising: an information acquisition module for acquiring multimodal perception data, wherein the multimodal perception data includes at least two of the following: audio information, image information, time information, geographical location information, inertial motion information, spatial posture information, eye-tracking information, and biometric information; a data analysis module for constructing a dynamic context model based on the multimodal perception data; an information generation module for outputting auxiliary information that the user may need and the priority level of the auxiliary information based on the dynamic context model and in combination with the user's historical behavioral habits and / or preset preferences; and an information output module for determining the information output method based on the auxiliary information and the priority level.

[0016] Compared with the prior art, the beneficial effects of the present invention are: The present invention provides a multimodal interactive smart glasses control method, comprising: acquiring multimodal perception data, wherein the multimodal perception data includes at least two of the following: audio information, image information, time information, geographical location information, inertial motion information, spatial posture information, eye-tracking information, and biometric information; constructing a dynamic context model based on the multimodal perception data; outputting, based on the dynamic context model and in conjunction with the user's historical behavioral habits and / or preset preferences, auxiliary information that the user may need and the priority level of the auxiliary information; and determining the information output method based on the auxiliary information and the priority level. The present invention can proactively provide appropriate information to the user at the appropriate time and in the appropriate manner. Attached Figure Description

[0017] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1(a) and Figure 1(b) are schematic diagrams of the structure of a multimodal interactive smart glasses provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating a multimodal interactive smart glasses control method provided in an embodiment of the present invention. Detailed Implementation

[0018] The present invention will now be described in detail with reference to the accompanying drawings of the embodiments, and the technical solutions of the embodiments will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To facilitate understanding of how the multimodal interactive smart glasses control method is implemented in the embodiments of the present invention, please refer to Figures 1(a) and 1(b). Figure 1 is a schematic diagram of the structure of a multimodal interactive smart glasses provided in an embodiment of the present invention. The glasses include a speaker 1, a touch button 2, a first physical button 3, an infrared sensor 4, a first microphone 5, a camera 6, a biosensor 7, a second physical button 8, an eye-tracking sensor 9, and a second microphone 10. When worn, the speaker 1 is located above and behind the user's ears, the touch button 2 is located on the outside of the temples, the first physical button 3 is located in the front middle section of the right temple, the infrared sensor 4 is located on the outside of the frame to acquire environmental information, the first microphone 5 is located on the temples, the camera 6 is located on the outside of the frame to acquire image information, the biosensor 7 is located in the middle and rear section of the temples to acquire the user's biometric information, the second physical button 8 is located in the front middle section of the left temple, the eye-tracking sensor 9 is located on the inside of the frame, and the second microphone 10 is located near the nose pad.

[0020] In practical implementation, the above-mentioned components work together to complete the multimodal interactive smart glasses control method described in this invention: The acquisition of multimodal perception data mainly relies on the collaboration of the camera system, audio system, and biosensing system. Camera 6 continuously acquires image information from the user's first-person perspective, while infrared sensor 4 assists in acquiring environmental depth and heat source information; together, they constitute the visual perception front end. For audio information, a microphone array is formed by a first microphone 5 located on the temple and a second microphone 10 located near the nose pad. One microphone is responsible for far-field environmental sound pickup, while the other is close to the mouth to enhance near-field human voice. In some embodiments, the second microphone 10 is a bone conduction microphone, which can directly pick up the bone vibration signal when the user speaks, serving as a source of voiceprint biometric information. Eye-tracking sensor 9 is integrated into the inner side of the frame using infrared illumination and a miniature camera module to capture the user's gaze point, fixation duration, and blink frequency in real time. Biosensor 7 is attached to the temporal skin in the middle and rear section of the temple, acquiring physiological signals such as heart rate and heart rate variability through photoplethysmography (PPG). An inertial measurement unit (IMU) is integrated inside the temple (not shown in the figure) to acquire inertial motion information and spatial posture information of head movements.

[0021] Data processing and the construction of the dynamic context model are accomplished by the local neural network processor (NPU) built into the glasses. This processor receives the aforementioned multimodal perception data streams and performs fusion analysis: it combines visual data from camera 6 and infrared sensor 4 with audio data from second microphone 10 to identify the current environmental scene type; it performs semantic analysis and key entity extraction on the dialogue voice picked up by the array of first microphone 5 and second microphone 10; simultaneously, it combines attention data provided by eye-tracking sensor 9 and physiological state data provided by biosensor 7 with head posture changes collected by inertial measurement unit to comprehensively assess the wearer's current cognitive load and emotional state. Based on these three analytical clues, the local NPU generates a dynamically updated context model that includes three dimensions: environmental context, social dialogue flow, and the user's internal state.

[0022] Furthermore, in some practical examples, to achieve lightweight glasses and reduce heat generation, the glasses hardware structure may not include a processor. Instead, a cloud processor or an external user phone can be used. In such embodiments, the computational tasks of multimodal perception data fusion analysis, dynamic context model construction, and auxiliary information generation and presentation decisions are all or partly performed by an external computing device connected to the glasses. This external computing device includes, but is not limited to: a cloud processor: The glasses upload the collected multimodal perception data to a cloud server in real-time or near real-time via a built-in low-power Bluetooth chip or Wi-Fi module. The cloud server, equipped with a high-performance AI inference engine, completes the construction of the context model and the generation of auxiliary information, and sends the results back to the glasses to drive information presentation. To protect privacy, sensitive data can be anonymized or federated learning feature extraction can be performed before uploading. Alternatively, an external user phone: The glasses establish a high-speed data link with the user's smartphone via near-field communication methods such as Bluetooth or Ultra-Wideband (UWB). The accompanying application (APP) on the phone calls the phone's built-in NPU, CPU, or GPU to proxy the AI ​​computational tasks that should have been completed by the glasses' local processor. Since modern smartphones generally possess strong on-device AI computing power, this approach can maintain low response latency without increasing the weight and power consumption of the glasses, while ensuring that data does not leave the phone, thus protecting privacy and security. In this type of embodiment without a local processor, the main hardware of the glasses is decoupled into a highly integrated sensing and interaction front-end. Its core tasks are simplified to: collecting and transmitting multimodal data, receiving external computing results, and driving actuators such as speakers and display modules to present information. This architecture significantly reduces the weight of the glasses themselves, the heat dissipation burden, and material costs, which is conducive to achieving a lighter, all-day wearing experience.

[0023] The generation and presentation of auxiliary information are also decided within the processor, with a preferred local neural network processor ensuring both privacy and response speed. The local NPU, based on a dynamic context model, retrieves locally stored user history and preset preferences to generate auxiliary information that the user may currently need, such as identification of the person in front of them, summaries of recently missed conversation points, and rest reminders based on elevated heart rate. It then calculates the priority of each piece of auxiliary information. Finally, based on the priority levels and factors such as cognitive load and environmental privacy within the current context model, it determines the most appropriate way to output the information.

[0024] The final information presentation is adaptively accomplished through multiple actuators. When the decision is a private voice output, the speaker 1 (preferably a bone conduction speaker) located above and behind the ear plays the auxiliary information in a manner audible only to the wearer; when the decision is a visual cue, it is presented as luminous text through a display module (not shown in the figure) integrated into the lens. Touch key 2, first physical button 3, and second physical button 8 provide users with supplementary channels for manual interaction, which can be used to confirm, ignore, or further query information actively pushed by the system.

[0025] Please see Figure 2 , Figure 2 This is a flowchart illustrating a multimodal interactive smart glasses control method provided in an embodiment of the present invention. In this embodiment, the multimodal interactive smart glasses control method includes the following steps: S1, acquire multimodal perception data, wherein the multimodal perception data includes at least two of the following: audio information, image information, time information, geographical location information, inertial motion information, spatial posture information, eye tracking information, and biometric information; S2, Construct a dynamic context model based on the multimodal perception data; S3. Based on the dynamic context model, combined with the user's historical behavioral habits and / or preset preferences, output the auxiliary information that the user may need and the priority level of the auxiliary information; S4. Determine the information output method based on the auxiliary information and the priority level.

[0026] In one embodiment, step S1 is the perception foundation layer for multimodal interaction, its core being to overcome the limitations of existing smart glasses that rely solely on a single or a few sensors for passive command responses. Specifically, audio information is acquired through a microphone array positioned near the temples and nose pads; some embodiments include bone conduction microphones to pick up bone vibration signals. Image information is captured by a camera on the outside of the frame, providing the user with first-person visual content. Time and geolocation information are acquired through the system clock and positioning module, respectively, providing a spatiotemporal reference for contextual judgment. Inertial motion and spatial posture information are acquired by an integrated inertial measurement unit (IMU), reflecting the user's head movement and body posture. Eye-tracking information is acquired through an infrared camera module on the inside of the frame, including gaze point, gaze duration, and blink frequency. Biometric information is acquired through biosensors, which may include physiological signals such as heart rate, heart rate variability, and skin conduction activity, as well as voiceprint features extracted from bone conduction speech. This step emphasizes the acquisition of "at least two pieces of information," meaning that the present invention is not limited to a fixed combination of sensors, but rather constructs a multi-dimensional perception data pool that can be flexibly combined according to device configuration, providing rich raw materials for subsequent deep contextual understanding.

[0027] In one embodiment, step S2 is the core processing layer of multimodal fusion analysis. The system aligns and fuses the heterogeneous perceptual data acquired in S1 in terms of time and space, and inputs it to a local neural network processor or external computing device for analysis. This analysis process is not a simple superposition of various types of data, but rather a deep correlation mining: for example, combining scene recognition results from image information with geographical location information and environmental audio information to determine whether the user is "in a noisy coffee shop" or "in a quiet meeting room"; associating frequent gaze drifts in eye-tracking information with increased central rate variability in biometric information to infer that the user may be in a state of inattention or anxiety; and performing semantic analysis on the conversational speech captured by the microphone array to extract key entities (such as names, time, location, and tasks). The result of the fusion analysis is the construction of a dynamic context model that continuously updates over time, including three dimensions: environmental context, social dialogue flow, and the user's internal state. This model is the digital foundation for the glasses to "understand" the user's current situation.

[0028] In one embodiment, step 3 is the intelligent decision-making and intent reasoning layer. Based on the dynamic context model constructed in S2, the system further retrieves and references the user's historical behavioral habits and / or preset preferences. For example, if the context model identifies that the user is in a meeting, and historical data shows that the user often needs to record and generate meeting minutes in this scenario, the system will proactively infer the potential need that "the user may need a meeting summary." If the context model identifies that the person the user is talking to is a contact in their address book, but the user stares at the other person for a long time without speaking, the system may infer that "the user may have forgotten the other person's name and needs an identity prompt." Based on this, the system generates a series of auxiliary information, such as identity prompts, dialogue summaries, task reminders, or physiological state warnings. At the same time, the system calculates a priority level for each piece of auxiliary information based on the urgency of the information, its relevance to the current context, and the user's preset importance weight, providing a basis for subsequent presentation timing and modal decisions.

[0029] In one embodiment, step 4 is the delivery layer for adaptive information presentation. Based on the auxiliary information output in S3 and its priority level, combined with the user's current cognitive load level and environmental privacy status reflected in the dynamic context model in S2, the system intelligently selects the most appropriate information output method. For example, for a high-priority identity forgetting prompt, if the user is in a conversation with high cognitive load, the system may choose to overlay a faintly lit name prompt at the edge of the user's field of vision, rather than interrupting the user with voice; if a noisy environment with other people is detected, for highly private messages, the system will prioritize using a bone conduction speaker for private voice broadcasting; and for non-urgent health reminders, the system may choose to silently record and store them in local memory, waiting for the user to access them when they are free. This dynamic and adaptive output decision ensures that information is presented in the most appropriate way at the most appropriate time, providing timely assistance while minimizing interference with the user.

[0030] As an optional implementation, the acquisition of multimodal perception data includes: acquiring the audio information through a microphone array mounted on the glasses, acquiring the image information through a camera, acquiring the inertial motion information and spatial attitude information through an inertial measurement unit, acquiring the eye tracking information through an eye tracking sensor, and acquiring the biometric information through a biosensor; wherein the microphone array includes at least one bone conduction microphone for picking up bone vibration signals generated when the user speaks.

[0031] As an optional implementation, the construction of the dynamic context model includes: performing fusion analysis on the multimodal perception data to identify the current environmental scene type, dialogue topic and key entities, as well as the user's attention state and cognitive load level in real time; and generating a dynamic context model containing three dimensions, namely environmental context, social dialogue flow and user's internal state, based on the identification results.

[0032] As an optional implementation, the construction of the dynamic context model specifically includes the following processing steps: (1) Spatiotemporal alignment and synchronization of multimodal data. Since the sampling frequencies and delays of different sensors vary (e.g., camera 30fps, microphone 16kHz, IMU 100Hz), it is first necessary to align the timestamps of each modal data stream. Based on the system clock, the data frames of each sensor are interpolated or truncated to a unified time axis according to the generation time. At the same time, the camera image and the eye-tracking image are spatially calibrated to determine the transformation matrix between their coordinate systems, so that the gaze point can be accurately mapped to the corresponding pixel area of ​​the scene image. (2) Single-modal feature extraction: The original data of each modality are input into the corresponding feature extraction sub-network: Image features: The RGB image frames captured by the camera are input into a lightweight convolutional neural network (such as MobileNetV3) to extract scene-level visual feature vectors; Audio features: The audio stream picked up by the microphone array is converted into a Mel spectrogram through short-time Fourier transform and input into a convolutional recurrent neural network to extract acoustic embedding features; Speech semantic features: The text sequence obtained by automatic speech recognition is input into a pre-trained language model (such as a lightweight variant of BERT) to extract the semantic embedding vectors and named entity labels of the dialogue; Motion and posture features: The three-axis acceleration and angular velocity sequences output by the IMU are input into a one-dimensional temporal convolutional network or a long short-term memory network to extract motion pattern embedding features; Eye movement features: The structured signals such as pupil coordinates, gaze duration, and blink count output by the eye movement tracking sensor are normalized and statistically analyzed by a sliding window to generate eye movement behavior feature vectors; and other information. (3) Cross-modal feature fusion: After mapping the above single-modal feature vectors to a common semantic space of the same dimension through linear projection, they are input into the multi-head cross-attention fusion module. This module calculates the attention weight between any two modal features, adaptively captures the complementarity and correlation between modalities, and outputs a compact multi-modal fusion feature vector. (4) Multi-task joint reasoning: The fusion feature vector is simultaneously input into three parallel task head networks: output the probability distribution of the current environment scene type (e.g., meeting room / coffee shop / street / car interior, etc.), trained using the cross-entropy loss function; output the sequence labeling results of dialogue topic categories and key entities, optimized for entity boundaries using conditional random fields; output the classification results of attention state (focused / distracted) and cognitive load level (high / medium / low), using the focus loss function to alleviate class imbalance. (5) Structured output and dynamic update of the context model: The above reasoning results are assembled into a dynamic context model containing three dimensions according to the preset data structure: environmental context, social dialogue flow, and user internal state. The environmental context includes scene category, GPS coordinates, timestamp, and environmental noise level; the social dialogue flow includes dialogue topic vector, key entity list (entity type, entity value, confidence level), and speaker separation results; the user's internal state includes attention score, cognitive load level, emotional valence, and arousal.

[0033] The model update strategy combines event-driven and timed polling. When the change in the feature vector of any modality exceeds a preset threshold, an immediate update is triggered. At the same time, a maximum update interval (e.g., every 5 seconds) is set to ensure that the contextual model does not stagnate for a long time. During the update, an exponential moving average is used to smooth the historical state to avoid model jumps caused by misjudgments in a single frame.

[0034] As an optional implementation, identifying the user's attention state and cognitive load level includes: judging the user's current level of attention concentration and cognitive busyness based on the gaze duration, blink frequency, and changes in gaze point in the eye-tracking information, combined with changes in head posture in the inertial motion information.

[0035] As an optional implementation, the biometric information includes at least one of heart rate information, heart rate variability information, skin electrical activity information, and voiceprint feature information; the voiceprint feature information is obtained by analyzing the bone vibration signals picked up by the bone conduction microphone.

[0036] As an optional implementation, the step of outputting the auxiliary information that the user may need and the priority level of the auxiliary information includes: generating at least one piece of auxiliary information based on the current scenario and user state identified by the dynamic scenario model, combined with the user's historical behavior habits and / or preset preferences; and calculating the priority level of the auxiliary information based on the closeness of the correlation between the auxiliary information and the current scenario, the time urgency, and the user's preset importance weight.

[0037] As an optional implementation, the auxiliary information includes at least one of identity prompt information, dialogue summary information, task reminder information, physiological state warning information, and navigation guidance information; the historical behavior habits include records of information types actively queried by the user in different scenarios, records of response rates to different types of prompts, and information on frequently contacted persons and high-frequency locations set by the user.

[0038] As an optional implementation, determining the information output mode based on the auxiliary information and the priority level includes: selecting at least one preset information output mode from a variety of modes based on the priority level and the user's current cognitive load level and environmental noise level in the dynamic context model; the variety of information output modes includes bone conduction private voice broadcast, visual field dim light text prompts, prompt sound broadcasts, and silent storage to local memory.

[0039] As an optional implementation, determining the information output method includes: if the dynamic context model determines that there is a risk of privacy leakage in the current environment, or the priority level of the auxiliary information corresponds to a high privacy level, then the bone conduction private voice broadcast is selected as the information output method; if the dynamic context model determines that the user is in a state of high cognitive load, then the output frequency of non-urgent auxiliary information is reduced, or it is temporarily stored in the local memory for the user to review later.

[0040] As an optional implementation, the method further includes: driving a corresponding actuator to present the auxiliary information according to the determined information output mode; the actuator includes at least one of a bone conduction speaker, a display module, an indicator light, and a vibration module; wherein the bone conduction speaker and the bone conduction microphone are integrated on the inner side of the temple of the eyeglasses, and are used to play auxiliary information and collect bone vibration signals, respectively.

[0041] In one specific embodiment, a user wears the multimodal interactive smart glasses described in this invention to attend an industry exchange meeting. The venue is filled with approximately one hundred people, the environment is noisy, and conversations are frequent. The user is having a face-to-face conversation with a familiar face but whose name they cannot immediately recall. The glasses' various sensors work together. First, multimodal perception data is acquired: the camera captures facial image information of the person being spoken to; an array of first and second microphones captures audio information, including speech features, while the eye-tracking sensor detects that the user's gaze lingers on the other person's face for an extended period, exceeding a normal threshold, and with a slightly increased blinking frequency. A dynamic context model is constructed: the processor fuses and analyzes the above data: comparing the facial image with the local address book, it identifies the other person as "General Manager Zhang," a contact added by the user three months prior at an event, but the matching confidence level does not reach the automatic push threshold. Combining the prolonged gaze, slight increase in heart rate, and increased blinking frequency, the system infers that the user is currently under social cognitive load and is likely trying to recall the other person's identity but has temporarily forgotten it. The system comprehensively assesses the current situation: a noisy social environment, facing a known contact, and the user experiencing difficulty recalling their identity. The generation and priority calculation of auxiliary information are as follows: Based on the above situational model, the system generates the following auxiliary information: Identity prompt – “Mr. Zhang, General Manager of XX Technology Company, last met: a summit in March 2024.” Priority level: High. Because the user is in the middle of a conversation, and the social pressure of forgetting the other person's identity increases over time, timeliness is crucial. The processor determines the information output method based on the current situation and priority: immediate presentation. Output method: a low-light text prompt within the user's field of vision. The system determines that the user is currently in a face-to-face conversation; if a voice broadcast were used, it would directly interrupt the conversation and be noticed by the other party, causing greater social embarrassment. Therefore, the prompt “Mr. Zhang, General Manager of XX Technology Company” is presented in a low-brightness, non-intrusive text format at the edge of the user's field of vision. Throughout the process, the user does not perform any manual operation; the conversation is smooth and natural, and the auxiliary information appropriately eliminates social embarrassment, demonstrating the core beneficial effect of “proactively providing appropriate information at the right time and in the right way.”

[0042] Based on the same inventive concept as the above method, another embodiment of this disclosure also discloses a multimodal interactive smart glasses control system, comprising: an information acquisition module for acquiring multimodal perception data, wherein the multimodal perception data includes at least two of the following: audio information, image information, time information, geographical location information, inertial motion information, spatial posture information, eye-tracking information, and biometric information; a data analysis module for constructing a dynamic context model based on the multimodal perception data; an information generation module for outputting auxiliary information that the user may need and the priority level of the auxiliary information based on the dynamic context model and in combination with the user's historical behavioral habits and / or preset preferences; and an information output module for determining the information output method based on the auxiliary information and the priority level.

[0043] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0044] In summary, by implementing the embodiments of the present invention, multimodal perception data is acquired, including at least two of the following: audio information, image information, time information, geographic location information, inertial motion information, spatial posture information, eye-tracking information, and biometric information. Based on the multimodal perception data, a dynamic context model is constructed. Based on the dynamic context model, and combined with the user's historical behavioral habits and / or preset preferences, auxiliary information that the user may need, along with the priority level of the auxiliary information, is output. Based on the auxiliary information and the priority level, the information output method is determined. The present invention can proactively provide appropriate information to users at the appropriate time and in the appropriate manner.

[0045] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0046] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the electronic devices, apparatuses, etc. described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0047] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control method for multimodal interactive smart glasses, characterized in that, include: Acquire multimodal perception data, which includes at least two of the following: audio information, image information, time information, geographic location information, inertial motion information, spatial attitude information, eye tracking information, and biometric information; Based on the multimodal perception data, a dynamic context model is constructed; Based on the dynamic context model, combined with the user's historical behavioral habits and / or preset preferences, the system outputs the auxiliary information that the user may need and the priority level of the auxiliary information. The information output method is determined based on the auxiliary information and the priority level.

2. The multimodal interactive smart glasses control method according to claim 1, characterized in that, The acquisition of multimodal sensing data includes: The audio information is collected through a microphone array mounted on the glasses, the image information is collected through a camera, the inertial motion information and spatial attitude information are collected through an inertial measurement unit, the eye tracking information is collected through an eye tracking sensor, and the biometric information is collected through a biosensor. The microphone array includes at least one bone conduction microphone for picking up bone vibration signals generated when the user speaks.

3. The multimodal interactive smart glasses control method according to claim 1, characterized in that, The construction of the dynamic context model includes: The multimodal perception data is fused and analyzed to identify the current environmental scene type, dialogue topic and key entities, as well as the user's attention state and cognitive load level in real time; Based on the recognition results, a dynamic context model is generated that includes three dimensions: environmental context, social dialogue flow, and user internal state.

4. The multimodal interactive smart glasses control method according to claim 1, characterized in that, The biometric information includes at least one of heart rate information, heart rate variability information, skin electrical activity information, and voiceprint feature information; The voiceprint feature information is obtained by analyzing the bone vibration signals picked up by the bone conduction microphone.

5. The multimodal interactive smart glasses control method according to claim 1, characterized in that, The output of potential user assistance information and the priority level of the assistance information include: Based on the current scenario and user state identified by the dynamic scenario model, and combined with the user's historical behavior habits and / or preset preferences, at least one piece of auxiliary information is generated; The priority level of the auxiliary information is calculated based on its relevance to the current scenario, time urgency, and user-preset importance weight.

6. The multimodal interactive smart glasses control method according to claim 5, characterized in that, The auxiliary information includes at least one of the following: identity prompt information, dialogue summary information, task reminder information, physiological status warning information, and navigation guidance information; The historical behavioral habits include records of the types of information actively queried by the user in different scenarios, records of response rates to different types of prompts, and information on frequently contacted persons and high-frequency locations set by the user.

7. The multimodal interactive smart glasses control method according to claim 1, characterized in that, The step of determining the information output method based on the auxiliary information and the priority level includes: Based on the priority level, and in conjunction with the user's current cognitive load level and environmental noise level in the dynamic scenario model, at least one of the preset information output modes is selected. The various information output modes include bone conduction private voice broadcasting, visual field dim light text prompts, prompt sound broadcasting, and silent storage to local memory.

8. The multimodal interactive smart glasses control method according to claim 7, characterized in that, The method of determining information output includes: If the dynamic scenario model determines that there is a risk of privacy leakage in the current environment, or if the priority level of the auxiliary information corresponds to a high privacy level, then the bone conduction private voice broadcast is selected as the information output method. If the dynamic situation model determines that the user is in a state of high cognitive load, it reduces the output frequency of non-urgent auxiliary information or temporarily stores it in the local memory for the user to review later.

9. The multimodal interactive smart glasses control method according to claim 1, characterized in that, The method further includes: Based on the determined information output method, the corresponding actuator is driven to present the auxiliary information; The actuator includes at least one of a bone conduction speaker, a display module, an indicator light, and a vibration module; wherein the bone conduction speaker and the bone conduction microphone are integrated into the inner side of the temple of the eyeglasses, and are used to play auxiliary information and collect bone vibration signals, respectively.

10. A multimodal interactive smart glasses control system, characterized in that, include: The information acquisition module is used to acquire multimodal perception data, which includes at least two of the following: audio information, image information, time information, geographical location information, inertial motion information, spatial posture information, eye tracking information, and biometric information. The data analysis module is used to construct a dynamic context model based on the multimodal sensing data; The information generation module is used to output auxiliary information that the user may need and the priority level of the auxiliary information based on the dynamic context model and the user's historical behavior habits and / or preset preferences. The information output module is used to determine the information output method based on the auxiliary information and the priority level.

Images