Intelligent glasses based on multi-mode perception and output mechanism

Abstract

The utility model relates to the technical field of intelligent glasses, and discloses intelligent glasses based on a multi-mode perception and output mechanism. According to the intelligent glasses, through multi-modal data fusion and hardware construction innovation, the communication assisting effect in a complex environment is remarkably improved. The overlook view angle sign language recognition solves the problem of shielding of a traditional fixed view angle, and the edge-cloud cooperative processing guarantees the real-time performance and the recognition precision. The transparent OLED display screen and the loudspeaker ensure that information is visually output and does not interfere with a normal visual field, the cruising ability is prolonged through the modular battery system, and all-weather and low-delay barrier-free interaction experience is provided for hearing-impaired people and high-noise environment users.

Description

Technical Field

[0001] This utility model relates to the field of smart glasses technology, specifically to a smart glasses based on a multimodal perception and output mechanism. Background Technology

[0002] In recent years, the rapid development of artificial intelligence and computer vision technologies has driven the widespread application of smart wearable devices, especially in the fields of medical assistance and industry. The communication needs of hearing-impaired individuals, speech-impaired patients, and workers in high-noise environments are becoming increasingly prominent. Traditional solutions such as hearing aids and cochlear implants rely on sound signal amplification, but their performance degrades significantly in complex acoustic environments. Vision-based assistive technologies (such as lip reading and sign language recognition) are gradually becoming a research hotspot. For example, the LipNet model proposed at CVPR 2016 achieved sentence-level lip reading through end-to-end deep learning, while devices like SignAll utilize multiple cameras to capture hand gestures.

[0003] However, existing technologies still have significant limitations: they rely on a single modality, with current smart glasses focusing on a single data source (such as lip reading or sign language alone), making it difficult to handle the multimodal interaction needs in complex scenarios; they are sensitive to perspective and environment, with sign language recognition relying on fixed-view cameras, easily affected by background interference and gesture occlusion, while lip reading recognition is not adaptable enough to changes in lighting and individual pronunciation differences; and they have low hardware integration, with existing devices being bulky and having limited battery life, making it difficult to meet the portability and comfort requirements for everyday wear. To address these limitations, a smart glasses based on a multimodal perception and output mechanism is proposed. Utility Model Content

[0004] To address the shortcomings of existing technologies, this invention provides a smart glasses based on a multimodal sensing and output mechanism, thus solving the problems mentioned in the background technology.

[0005] To achieve the above objectives, this utility model provides the following technical solution: a smart glasses based on a multimodal sensing and output mechanism, including a frame, temples and lenses, and also including an input device module, an output device module, a power supply system and a main control unit;

[0006] The input device module includes:

[0007] A head-up camera, positioned directly in front of the frame, is used to capture images of the lip movements of the person you are talking to;

[0008] The top-down camera, positioned directly in front of the frame, features a wide-angle lens and is used to capture the user's hand movements.

[0009] A microphone, built into the side of the frame, is used to collect ambient voice signals;

[0010] An accelerometer, built into the top front of the frame, is used to detect changes in head posture and switch between the working states of the head-up camera and the head-down camera based on the posture data.

[0011] A touch-sensitive switch, located on the outside of the temple, is used to manually control the opening and closing of the input device module;

[0012] The output device module includes:

[0013] An in-lens display screen, integrated inside the lens, is used to display the recognized text content;

[0014] Speakers, embedded in both sides of the frame, are used for voice output;

[0015] The power supply system includes:

[0016] Multiple micro battery units are distributed within the temples and frame;

[0017] A modular battery management system connects to micro battery cells for power rotation and intelligent charging switching;

[0018] The main control unit, integrated inside the temple, includes:

[0019] An edge-embedded processing chip is electrically connected to a head-up camera, a top-down camera, and an accelerometer. It is used to trigger camera switching logic based on the attitude data from the accelerometer and control the activation state of the head-up camera and the top-down camera.

[0020] The wireless communication module supports dual-mode communication of 5G and Wi-Fi 6, and connects to the cloud server and the edge embedded processing chip to send commands processed by the edge embedded processing chip and the cloud server to the output device module.

[0021] The encryption chip, based on Fast User Datagram Protocol Internet Connection (QUIC) and Transport Layer Security Protocol version 1.3, encrypts the data transmission process of the wireless communication module.

[0022] The main control unit is connected to the input device module and power supply system inside the frame via a bus for data interaction.

[0023] Preferably, the edge-embedded processing chip has a built-in timestamp synchronization circuit for aligning the attitude data of the accelerometer with the video frames of the head-up camera and the top-down camera in time.

[0024] Preferably, the in-mirror display and speaker receive commands from the edge embedded processing chip and the cloud server via a wireless communication module.

[0025] Preferably, the modular battery management system includes a charge / discharge control chip and a temperature sensor. The temperature sensor is located near each micro battery cell and is used to monitor the battery temperature and dynamically distribute the load.

[0026] Preferably, the touch sensor switch is a capacitive touch sensor, electrically connected to the main control unit, and used to control the opening and closing of the input device module through preset gestures.

[0027] Preferably, the wireless communication module includes a Quality of Service (QoS) management unit, which is a hardware priority allocation circuit of the wireless communication module, used to allocate transmission priorities according to data types, wherein sign language keyframes and lip reading video frames have higher priorities than sensor data.

[0028] Preferably, the main control unit is also connected to a storage unit, which is a flash memory chip with a built-in gesture quick recognition database that stores specific gesture feature data numbered 01 to 99 for quickly matching user gestures and outputting preset semantics.

[0029] Preferably, the encryption chip is electrically connected to the touch sensor switch, and when the touch sensor switch closes the input device module, the data transmission channel of the encryption chip is simultaneously cut off.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention's smart glasses significantly improve communication assistance in complex environments through multimodal data fusion and hardware innovation. Its top-down view sign language recognition solves the occlusion problem of traditional fixed-viewpoint systems, while edge-cloud collaborative processing ensures real-time performance and recognition accuracy. The transparent OLED display and speakers ensure intuitive information output without interfering with normal vision, and the modular battery system extends battery life, providing a 24 / 7, low-latency, barrier-free interactive experience for hearing-impaired individuals and users in high-noise environments.

[0032] Other features and advantages of this invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained by means of the structures pointed out in the description, claims, and drawings. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the front-view structure of the smart glasses of this utility model;

[0034] Figure 2 This is a schematic diagram of the inner viewing angle structure of the smart glasses of this utility model;

[0035] Figure 3This is a schematic diagram of the upward viewing angle structure of the smart glasses of this utility model;

[0036] Figure 4 This is a schematic diagram of the interaction process of the multimodal lip-reading and gesture recognition smart glasses of this utility model;

[0037] Figure 5 This is a block diagram of the smart glasses system of this utility model.

[0038] In the image: 1. Frame; 2. Temple; 3. Lens; 4. Speaker; 5. Head-up camera; 6. Microphone; 7. Accelerometer; 8. Touch-sensitive switch; 9. Miniature battery unit; 10. Top-down camera; 11. In-lens display. Detailed Implementation

[0039] The technical solutions of the present utility model will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present utility model, and not all embodiments. Based on the embodiments of the present utility model, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present utility model.

[0040] Please see Figure 1-5 This invention relates to a smart glasses system based on a multimodal perception and output mechanism. By integrating four functional components—an input device module, an output device module, a power supply system, and a main control unit—it provides users with high-precision, low-latency sign language recognition, lip reading, and voice processing services. The aim is to leverage multi-sensor fusion, deep learning algorithms, and high-speed, secure network transmission technology to help target users achieve real-time and accurate communication in complex environments.

[0041] 1. Appearance and Structure

[0042] Frame 1: Made of lightweight alloy material, it integrates a head-up camera 5, a microphone 6, a speaker 4, and a power supply system.

[0043] Temple 2: Rotatably connected to both ends of frame 1, with touch sensor switch 8, main control unit and part of micro battery unit 9 integrated on the left and right temples respectively;

[0044] Lens 3: Mounted on the frame 1, with an internal display screen 11 embedded on the inside, supporting transparent display.

[0045] 2. Input device module

[0046] Head-up camera 5: Mounted above the nose pad in front of the frame 1, it uses an 8-megapixel CMOS sensor to capture dynamic images of the lips of the person you are talking to;

[0047] Top-view camera 10: Installed in front of the frame, below the head-up camera 5, it uses a 120° wide-angle lens to capture the user's hand movements from a top-down perspective;

[0048] Microphone 6: Embedded in the ear hooks on both sides of the frame 1, equipped with active noise cancellation circuitry, used to collect clear voice signals in noisy environments;

[0049] Accelerometer 7: Integrated on the top front of the frame 1, it detects the head tilt angle. When the head tilt angle exceeds 15°, it triggers the activation of the downward-looking camera 10.

[0050] Touch sensor switch 8: Located on the outer surface of the right temple 2, it adopts a capacitive touch sensor. Users can turn the input device module on / off by double-tapping gestures. Touch sensor switch 8 is used to identify the on and off of the function to meet the user's privacy and autonomy needs in different situations.

[0051] The output data from the head-up camera 5, the top-down camera 10, the microphone 6, the accelerometer 7, and the touch sensor switch 8 will be aggregated through the internal bus to the edge embedded processing chip of the main control unit, forming a multimodal input data stream, which lays the foundation for subsequent data processing.

[0052] 3. Power supply system

[0053] Micro battery unit 9: consists of 4 lithium polymer batteries, which are respectively arranged inside the left and right temples 2 and the nose pad of the frame 1;

[0054] Modular battery management system: Includes charge and discharge control chip (such as TI BQ25895) and temperature sensor (such as NTC thermistor) to monitor the temperature of each battery cell in real time and switch the power supply battery pack through dynamic load distribution algorithm to avoid overheating or over-discharge.

[0055] 4. Main Control Unit

[0056] 4.1 The main control unit is the intelligent core of this utility model. The main control unit consists of two parts: an edge embedded processing chip and a cloud data processing platform.

[0057] Edge-embedded processing chip: Utilizing an NVIDIA Jetson Nano module, integrated inside the left temple 2, it connects to the head-up camera 5, the top-down camera 10, and the accelerometer 7 via an internal frame bus (such as the I2C protocol), and is connected to the internal display screen 11 via flexible circuitry (such as an FPC cable), performing the following functions:

[0058] Based on the attitude data from the accelerometer 7, the camera switching logic is controlled (camera 5 is activated in head-up mode, and camera 10 is activated in top-down mode).

[0059] The attitude data of the accelerometer 7 is aligned with the camera video frame by a built-in hardware clock synchronization circuit (such as an FPGA timing controller). The core function of the accelerometer is to detect the wearer's head posture and movement state and synchronize this information with the camera image stream. When the system determines that the user is in sign language input mode, it will automatically enable the sign language recognition algorithm. When the user is facing the interlocutor and the head posture is stable, it can switch to lip reading recognition mode.

[0060] The display signals (such as lip reading text and sign language translation results) output by the edge embedded processing chip are transmitted to the in-lens display screen (11) through a flexible circuit. The display screen driver IC converts the signals into pixel control instructions for the transparent OLED panel to achieve text overlay display without affecting the light transmittance of the lens.

[0061] After the edge-embedded processing chip preprocesses the data, the system selectively packages and uploads video keyframes, audio streams, or sensor data to the cloud based on real-time requirements and network bandwidth. The cloud data processing platform consists of a high-performance GPU cluster, and its software environment deploys various deep learning models, including YOLOv5 (You Only LookOnce Version 5) for hand and lip region detection, ResNet (Residual Network) for feature extraction, and CTC (Connectionist Temporal Classification) and temporal-space attention mechanisms for the recognition and semantic inference of continuous sign language or lip-reading sequences.

[0062] 4.2 Wireless Communication Module and Encryption Chip

[0063] 4.21. The wireless communication module uses a 5G / Wi-Fi 6 dual-mode communication chip (such as Qualcomm SDX55) to transmit instructions from the edge embedded processing chip and the cloud server to the output device module;

[0064] In this invention, the wireless communication module plays a crucial role in data interaction between the terminal input device module and the main control unit. To meet the demands of high concurrency and high-volume real-time transmission, a dual-channel communication strategy integrating 5G and Wi-Fi 6 is adopted: in outdoor or large-scale event scenarios, the system prioritizes the use of the 5G network (fifth-generation mobile communication technology), leveraging its high bandwidth and low latency to upload audio and video data and distribute model parameters; indoors or in fixed locations, Wi-Fi 6 (sixth-generation wireless local area network standard) can be prioritized to further improve the throughput and stability of the local area network.

[0065] To prevent data packet loss or excessive latency caused by network congestion, a Quality of Service (QoS) management unit is built into the wireless communication module. This QoS management unit is a hardware priority allocation circuit for the wireless communication module, capable of prioritizing different types of data (such as image frames, voice streams, and sensor data). Especially during the transmission of keyframes for lip-reading and sign language recognition, the system automatically allocates higher bandwidth to ensure the continuity and accuracy of the recognition process.

[0066] In addition, to adapt to dynamic changes in network conditions, a software-defined networking (SDN) architecture is adopted. Real-time traffic monitoring and intelligent scheduling algorithms are used to assess the current bandwidth usage and automatically allocate or release network resources to ensure that model delivery and data backhaul can be completed in a very short time.

[0067] 4.22. Regarding the communication protocol, an encryption chip (ATECC608 security chip) is used. Data encryption is performed based on the QUIC (Quick UDP Internet Connections) transmission protocol combined with TLS1.3 (Transport Layer Security 1.3) to ensure that audio and video data and user privacy information are not intercepted or tampered with during transmission. The encryption chip is electrically connected to the touch sensor switch 8. When the touch sensor switch 8 closes the input device module, the data transmission channel of the encryption chip is simultaneously cut off.

[0068] 4.3. Storage Unit: A 64GB eMMC flash memory chip stores a pre-trained gesture quick recognition database (specific gesture feature data numbered 01-99). Users do not need to type out the complete sign language; they only need to make gestures with specific numbers (such as 01 to 99) to quickly convey common semantics. The edge embedded processing chip calls the specific gesture feature data in the database stored in the storage unit.

[0069] 5. Output device module

[0070] The output device module serves as the business presentation and interaction interface for users of this utility model. The module includes an in-lens display screen 11 (text display screen or transparent OLED display panel) and a speaker 4 (text-to-speech based on neural network).

[0071] The in-lens display screen 11 is responsible for presenting the recognized lip reading, sign language, and speech content in high-resolution graphics and text in real time within the user's field of vision. To enhance readability, this invention features an adaptive design in interface layout and font selection, allowing users to clearly view the displayed information under different lighting conditions, whether indoors or outdoors. The in-lens display screen 11 uses a 0.5-inch transparent OLED panel, embedded inside the lens 3, to display the lip reading text or sign language translation results in real time;

[0072] Speakers 4 are built into both sides of the frame 1, playing the voice content parsed by the system, such as converting recognized sign language into speech, realizing the output assistance from sign language to hearing, and facilitating real-time feedback for the speaker. In this process, a neural network TTS (Text-to-Speech) algorithm is introduced to ensure that the voice output is natural and fluent with a certain emotional rhythm, thereby enhancing the auditory experience.

Claims

1. A smart glasses based on a multimodal sensing and output mechanism, comprising a frame (1), temples (2), and lenses (3), characterized in that, It also includes an input device module, an output device module, a power supply system, and a main control unit; The input device module includes: A head-up camera (5) is set in front of the frame (1) to capture images of the lip movements of the person being spoken to; The top-view camera (10) is located directly in front of the frame (1) and is designed with a wide-angle lens to capture the user's hand movements; Microphone (6), built into the side of the frame (1), is used to collect environmental voice signals; An accelerometer (7) is built into the top front of the frame (1) to detect changes in head posture and switch the working state of the head-up camera (5) and the head-down camera (10) according to the posture data. A touch-sensitive switch (8) is located on the outside of the temple (2) and is used to manually control the opening and closing of the input device module; The output device module includes: An in-lens display screen (11) is integrated inside the lens (3) and is used to display the recognized text content; Speakers (4) are embedded in both sides of the frame (1) and are used to play voice output; The power supply system includes: Multiple micro battery units (9) are distributed inside the temple (2) and frame (1); A modular battery management system is connected to a micro battery cell (9) for power switching and intelligent charging switching; The main control unit, integrated inside the temple (2), includes: An edge-embedded processing chip is electrically connected to a head-up camera (5), a top-down camera (10), and an accelerometer (7). It is used to trigger camera switching logic based on the attitude data of the accelerometer (7) and control the activation state of the head-up camera (5) and the top-down camera (10). The wireless communication module supports dual-mode communication of 5G and Wi-Fi 6, and connects to the cloud server and the edge embedded processing chip to send commands processed by the edge embedded processing chip and the cloud server to the output device module. The encryption chip, based on Fast User Datagram Protocol Internet Connection (QUIC) and Transport Layer Security Protocol version 1.3, encrypts the data transmission process of the wireless communication module. The main control unit is connected to the input device module and power supply system inside the frame (1) via a bus for data interaction.

2. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The edge-embedded processing chip has a built-in timestamp synchronization circuit for aligning the attitude data of the accelerometer (7) with the video frames of the head-up camera (5) and the top-down camera (10) in time.

3. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The in-mirror display (11) and speaker (4) receive commands from the edge embedded processing chip and cloud server via the wireless communication module.

4. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The modular battery management system includes a charge / discharge control chip and a temperature sensor. The temperature sensor is located near each micro battery cell (9) to monitor the battery temperature and dynamically distribute the load.

5. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The touch sensor switch (8) is a capacitive touch sensor, which is electrically connected to the main control unit and is used to control the opening and closing of the input device module through preset gestures.

6. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The wireless communication module includes a Quality of Service (QoS) management unit, which is a hardware priority allocation circuit for the wireless communication module. It is used to allocate transmission priorities according to data types, wherein sign language keyframes and lip reading video frames have higher priorities than sensor data.

7. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The main control unit is also connected to a storage unit, which is a flash memory chip with a built-in gesture quick recognition database that stores specific gesture feature data numbered 01 to 99, used to quickly match user gestures and output preset semantics.

8. The smart glasses based on a multimodal sensing and output mechanism according to claim 1, characterized in that, The encryption chip is electrically connected to the touch sensor switch (8). When the touch sensor switch (8) closes the input device module, the data transmission channel of the encryption chip is simultaneously cut off.

Images