Small-size AR intelligent glasses display interface user interaction method and system

Abstract

The invention discloses a small-size AR intelligent glasses display interface user interaction method and system. According to the interaction system, through dynamic regional information management layout, multi-mode interface dynamic switching, personalized interaction control and user-defined editing functions, the problems of information congestion, low interaction efficiency and insufficient immersion of a small-size AR display interface are solved. According to the main technical scheme, the system comprises a dynamic partition information management module, a multi-mode interaction control module, a personalized theme and appearance setting module, a self-adaptive layout control module and a DIY UI editing and developing module. The system supports various interaction modes such as gestures, voices and knobs, has environment self-adaptive capability, and can realize real-time visual interface editing of the mobile terminal and the AR terminal. According to the method, the information density and the user control freedom degree of the small-size AR interface are remarkably improved, and the visual fatigue rate is effectively reduced.

Description

Technical Field

[0001] This invention relates to the field of augmented reality (AR) technology, and more specifically, to a user interaction method and system for a small-sized AR smart glasses display interface. Background Technology

[0002] Augmented reality (AR) technology provides users with a seamless interactive experience by overlaying computer-generated information onto the real world as perceived by the user. With the miniaturization of optical modules, the increased integration of near-eye display devices (such as Micro-OLED and LCOS), and the continuous advancement of wireless communication and edge computing technologies, AR smart glasses are gradually evolving towards lightweight and consumer-grade designs, and are widely used in various scenarios such as navigation, sports monitoring, remote guidance, and real-time translation.

[0003] Currently, most mainstream AR glasses on the market have display module sizes within 12mm × 17.5mm. Due to the hardware size constraints of micro-optical imaging and waveguide projection, the design of their display interfaces faces significant challenges. How to effectively organize information layout and achieve efficient interaction within a limited field of view, while also taking into account users' personalized customization needs and immersive experiences, has become a key issue in the design of AR interactive systems.

[0004] Existing small-sized AR display interfaces generally suffer from the following technical bottlenecks:

[0005] Low information density and severe visual obstruction: Traditional fixed window layouts cannot adapt to limited field of view, information hierarchy is mixed, which can easily cause interface congestion, reduced readability, and even interfere with the observation of road conditions ahead, affecting safety.

[0006] Limited interaction channels and insufficient control freedom: Most AR devices currently only support voice or simplified gesture control, lacking a multimodal fusion mechanism based on spatial gestures, haptic feedback, knob interaction, etc., making it difficult to handle complex task switching and multi-window operation.

[0007] Weak environmental adaptability and rigid display mode: It lacks a dynamic display mode adjustment strategy based on factors such as ambient light, time, and user movement status, and cannot flexibly adjust the display style and theme in multiple scenarios such as night and strong outdoor light.

[0008] Low user engagement and non-customizable interface: Users cannot freely edit the interface layout, adjust component sizes and interaction parameters according to their own preferences or task requirements, and there is a lack of cross-device synchronization and community sharing mechanisms, which limits the adaptability and scalability of the AR interface.

[0009] Therefore, there is an urgent need for a display interface and user interaction strategy and development system for small-sized AR smart glasses. By introducing a dynamic partition layout mechanism, multimodal interaction control logic, personalized theme and font appearance settings, environmental adaptive switching mechanism, and a toolchain that supports user visual DIY editing, the system can effectively improve the efficiency of information organization and user operation freedom under limited display area, enhance immersion and system scalability, and meet the intelligent human-computer interaction needs of the next generation of AR terminals. Summary of the Invention

[0010] Objective: To address the aforementioned problems, this invention provides a user interaction method and system for the display interface of small-sized (12mm × 17.5mm) AR smart glasses. This invention can be applied to small-sized AR smart glasses displays, balancing information clarity, ease of interaction, and dynamic adaptability. This solution offers five different interface layout modes suitable for various usage scenarios and enhances the user experience through dynamic UI adaptation.

[0011] Technical Solution: To achieve the above objectives, this invention discloses a user interaction method for the display interface of small-sized AR smart glasses, comprising the following steps:

[0012] Dynamic partitioned display steps: Dynamically divide the physical display area of ​​the smart glasses into three logical partitions: the central main viewing area, the dual foldable side areas, and the bottom context-aware auxiliary area, and configure each partition to display content with different priorities;

[0013] Multimodal interaction processing steps: receiving user input commands by integrating gesture recognition, voice recognition and external device control, wherein the gesture recognition is based on inertial measurement unit data, the voice recognition supports custom commands, and the external device control includes the knob operation of a smart bracelet or ring;

[0014] Adaptive interface control steps: Based on user input commands, environmental sensor data, and the current application scenario, dynamically adjust the display content, layout style, and visual parameters of each logical partition, including adjustments to window size, transparency, and layout mode;

[0015] The output of the dynamic partition display step provides the interface operation basis for the multimodal interaction processing step, and the output of the multimodal interaction processing step provides the adjustment basis for the adaptive interface control step. The three steps work together to achieve efficient human-computer interaction in a small display area.

[0016] Preferably, the dynamic partition display step includes:

[0017] The physical display area is divided into three logical zones according to the area ratio. The central main view area occupies 40% and is used to display real-time captions, navigation instructions or emergency notifications. The two foldable side areas each occupy 30%, which are hidden by default as edge indicator bars and can be expanded to display weather, schedule or multi-task thumbnails. The bottom context-aware auxiliary area occupies 20%, which dynamically loads icon groups according to the current application scenario.

[0018] It supports three-level adjustment of window size for each partition, stepless control of transparency from 50% to 100%, and adaptive switching between near and far focus modes.

[0019] Preferably, the multimodal interaction processing steps include:

[0020] The system detects user gestures using a nine-axis inertial measurement unit, defines left / right swipes to switch windows, up / down swipes to adjust brightness, pinch gestures to zoom in / out windows, and circling gestures to bring up the shortcut menu, and sets acceleration thresholds, duration thresholds, and trajectory characteristic parameters for each gesture.

[0021] The speech recognition engine supports user-defined voice commands. After receiving voice input, it performs noise reduction, semantic decomposition, and parameter extraction to generate a system API call chain to execute the corresponding operation.

[0022] The system receives rotational encoding signals through the knob interface of a smart bracelet or ring, controls window transparency and Z-axis layer in real time, and designs a dynamic adjustment algorithm based on rotational angular velocity.

[0023] Preferably, the adaptive interface control step includes:

[0024] It comes pre-installed with a variety of scenario-based interface templates, including classic HUD mode, immersive information flow mode, multi-tasking floating mode, minimalist power-saving mode and touch desktop mode;

[0025] Based on ambient light sensor data, IMU motion status data, and the current foreground application type, automatically select or recommend a suitable interface template;

[0026] It provides two-way real-time visual editing functionality for both mobile and AR devices, and supports users to customize interface layout, component attributes, and interaction logic.

[0027] Preferably, the scenario-based interface template includes at least:

[0028] The classic HUD mode is configured to display core information in the center and secondary information floating on the side with high transparency, and supports core information anchoring, intelligent anti-obstruction and context-aware simplified interaction;

[0029] Immersive information flow mode, which is configured to display vertically scrolling text in most areas, supports scroll-adaptive loading, focus-tracking typography, and semantic interactive tags;

[0030] The multi-tasking floating mode is configured to support a dynamic layout that avoids multiple floating windows, providing window hierarchy management, state adjustment, and intelligent avoidance mechanisms.

[0031] Preferably, the method further includes an environment adaptation step:

[0032] The ambient light sensor detects light intensity in real time and automatically switches the interface theme and display parameters based on the detection results, including enabling dark light optimization mode when the light intensity is below the first threshold and enabling bright contrast mode when the light intensity is above the second threshold.

[0033] The user's motion state is identified by the inertial measurement unit, and the interface layout and interaction method are dynamically adjusted according to whether the user is walking, stationary, or in a vehicle.

[0034] The theme switching process uses a smooth, gradual animation; if the user is detected performing an interface operation, the switching is delayed.

[0035] Preferably, the method further includes a DIY editing and sharing step:

[0036] It provides a visual editor for mobile apps, supporting component drag-and-drop, attribute binding, and theme switching. Configuration parameters are synchronized to AR glasses via low-latency Bluetooth protocol.

[0037] It provides a real-time visual editing module for AR devices, supports direct adjustment of UI elements through spatial gestures, and provides auxiliary tools for component snapping and grid alignment.

[0038] It supports exporting layout schemes as standard format files and distributing them through a sharing mechanism, and provides an open interface for third-party UI plugins to load external theme packages and interaction scripts.

[0039] The present invention also provides a user interaction system for a small-sized AR smart glasses display interface, used to implement the method, comprising:

[0040] The display management module is configured to execute the dynamic partition display steps and manage the logical division and content rendering of the physical display area.

[0041] A multimodal input processing module is configured to perform the multimodal interaction processing steps, processing gestures, voice, and external device input signals;

[0042] An adaptive control engine is configured to execute the adaptive interface control steps, controlling the interface to adaptively adjust based on input signals and environmental data.

[0043] An environmental sensing module, configured to collect and process environmental sensor data;

[0044] The modules communicate and collaborate through a data bus and interface protocols to jointly achieve an optimized interactive experience in a small display area.

[0045] Preferably, the display management module includes:

[0046] The partition management submodule is used to maintain the layout parameters and display attributes of the three logical partitions;

[0047] The rendering optimization submodule is used to optimize rendering effects based on the type of content being displayed and environmental conditions.

[0048] The transparency control submodule is used to dynamically adjust the transparency of each section and interface element;

[0049] The multimodal input processing module includes a gesture recognition unit, a voice parsing unit, and an external device interface unit.

[0050] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the method described thereon.

[0051] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: The small-sized AR smart glasses display interface and user interaction design strategy and development system proposed in this invention addresses key issues such as severe visual occlusion, single interaction channels, low user participation, and poor environmental adaptability under existing micro-displays. By constructing a collaborative strategy system of "dynamic partition information management + multimodal input interaction + personalized appearance theme + scene-based layout template + DIY UI editing platform," it provides a comprehensive, efficient, and scalable AR smart glasses interaction system design and development strategy. It fully considers the information organization efficiency, interaction control freedom, personalized configuration requirements, and system openness under the constraints of small-sized hardware, significantly improving the practicality, flexibility, and user experience of AR glasses at the human-computer interface layer, and possesses good market application prospects and technology promotion value. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the overall development process of AR smart glasses in this invention;

[0053] Figure 2 This is a schematic diagram of the dynamic partition information management and display module in this invention;

[0054] Figure 3 This is a schematic diagram of the multimodal interaction control module in this invention;

[0055] Figure 4 This is a schematic diagram of the personalized theme and appearance adjustment module in this invention;

[0056] Figure 5This is a design diagram of the classic HUD mode interface in this invention;

[0057] Figure 6 This is a design diagram of the immersive information flow mode interface in this invention;

[0058] Figure 7 This is a design diagram of the multi-task floating mode in this invention;

[0059] Figure 8 This is a diagram showing the interface design of the simplified power-saving mode in this invention.

[0060] Figure 9 This is a design diagram of the touch desktop mode interface in this invention;

[0061] Figure 10 This is a schematic diagram of the UI editing and development module in this invention;

[0062] Figure 11 This is a schematic diagram of the code structure framework in this invention;

[0063] Figure 12 This is a diagram illustrating a practical application of the present invention on a mobile device. Detailed Implementation

[0064] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0065] Example 1

[0066] This embodiment discloses a user interaction method and system for the display interface of small-sized (12mm × 17.5mm) AR smart glasses, specifically including as follows: Figure 1 The following development modules work together as shown:

[0067] (1) Dynamic partition information management and display module:

[0068] This module is configured to divide the physical display area into three dynamically adjustable logical partitions:

[0069] The central main view area (central area, accounting for 40% of the total area) is used for the fixed display of high-priority content, including live captions, navigation instructions and emergency notifications;

[0070] The dual foldable side areas (each occupying 30% of the horizontal width) are used for auxiliary information floating display. By default, they are hidden as a 2mm edge light strip. After responding to gesture / voice commands, they unfold into a semi-transparent floating panel to display weather, schedule and multi-task thumbnails.

[0071] The bottom context-aware auxiliary area (occupying 20% ​​of the vertical height) dynamically loads icon groups based on the current application scenario, displays the playback control bar in video mode, and provides a route preview hotspot in navigation mode;

[0072] This module supports the following dynamic adjustments for each partition, specifically including:

[0073] The window size can be adjusted in three levels, specifically in full-screen / half-screen / floating mode;

[0074] Transparency is infinitely adjustable to reduce visual obstruction. The transparency setting is adjustable from 50% to 100% in 5% increments.

[0075] refer to Figure 2 The specific operating mechanism of this module includes:

[0076] (1.1) Depth control of the side area;

[0077] When folded, it appears as a 3mm wide edge indicator bar. Its display logic includes: changing the light color according to the current scene (where blue indicates navigation, orange indicates notification to be read, and green indicates a call in progress); touch activation methods support double-click to wake up the smart bracelet knob, voice command "unfold the side", and continuous tap gestures on the edge area (3 touch frame actions with an interval of <500ms).

[0078] Upon touch, it expands into a full-featured floating panel. Its display logic includes: loading a three-layer stacked view, where the top layer is a snapshot of the current multitasking window, the middle layer is a persistent toolset, and the bottom layer is an environment-aware dashboard; activating an edge anti-mistouch algorithm, setting a touch dead zone within 2mm of the outer edge of the panel, and automatically folding when a hand is detected to continuously obscure the area for more than 5 seconds.

[0079] (1.2) Enhanced main view segmentation;

[0080] Split-screen trigger conditions include a two-finger pinch operation (the distance between the two fingers shrinks from >15mm to <5mm), the voice command "split-screen display", and a long press of the smart bracelet knob for 3 seconds followed by a 180° clockwise rotation.

[0081] The dual-view layer after split-screen has differentiated functions. The upper layer retains the original application's main interface, while the lower layer generates a related information flow, and dynamic interaction is supported between the layers.

[0082] (1.3) Intelligent adaptation of the bottom auxiliary area;

[0083] The design incorporates an adaptive mechanism that automatically increases icon contrast and adds a 2px glowing border to operable elements when the ambient light sensor detects a light intensity of less than 300 lux. In motion mode, the IMU identifies whether the user is walking or in a vehicle, expanding the touch area to 150% of its original size and disabling interactive functions requiring precise positioning.

[0084] (2) Multimodal interactive control module:

[0085] This module integrates the following three input channels:

[0086] The nine-axis inertial gesture recognition unit defines swiping, including three basic gestures: swiping left and right to switch windows, swiping up and down to adjust brightness, pinching (window scaling), and drawing circles (bringing up the shortcut menu);

[0087] The voice command parsing engine supports user-defined command phrases and their binding with system operations, including "Hide side" → close the side area, "Focus mode" → activate immersive layout, and "Move left" / "Move right" to instantly adjust the UI layout;

[0088] The smart bracelet features a rotary knob interface, providing knob-style interaction and real-time control of window transparency and Z-axis hierarchy via rotary encoder data stream.

[0089] like Figure 3 As shown, this module specifically includes the following collaborative operation mechanisms:

[0090] (2.1) Spatial gesture-driven window scaling strategy;

[0091] Define left and right swipes to switch windows, and up and down swipes to adjust brightness. Define a linear acceleration threshold of 1.5g and a duration threshold of 100-300ms. Execute the swipe command if both conditions are met. Define screen window scaling via pinch gestures, and define the rate of change of hand micro-motion acceleration as 0.5-2g / s². 2 Define the pinch duration threshold as 200ms, and execute the pinch command if both conditions are met; define the circle drawing to call up the shortcut menu, and form a closed trajectory by integrating the angular velocity. When the circle fitting error is <10% and the main frequency is 2-4Hz, execute the circle drawing command if both conditions are met.

[0092] (2.2) Voice command parsing method, including the following steps:

[0093] (2.2.1) Input custom voice commands, such as "hide side", "focus mode", "move left / right" or custom associated voice commands;

[0094] (2.2.2) Upon receiving voice input, noise reduction processing is performed, the voice signal is converted into text output, and the semantic decomposition module is operated to break down the instruction into "action + object + parameter";

[0095] (2.2.3) Match the preset commands and synonym commands in the custom command library, extract the adjustment parameters (value, direction, percentage, etc.), such as "30" in "shift left 30%", and automatically extract the numerical parameters;

[0096] (2.2.4) Generate system API call chain and execute corresponding operation tasks.

[0097] (2.3) Precise adjustment of transparency controlled by a knob;

[0098] The smart bracelet's knob operation and display window have a mapping relationship between single-window mode and multi-window mode. In single-window mode, the knob controls the transparency of the currently focused window by default, while in multi-window mode, the controlled object can be switched by clicking the knob.

[0099] Design an algorithm for dynamically adjusting rotational angular velocity and transparency:

[0100]

[0101] An exception handling mechanism has been added. If no operation is detected for 10 consecutive seconds, the transparency will be automatically reset to the default (75%).

[0102] (3) Personalized themes and appearance modules;

[0103] This module provides customization options for the user interface's visual effects. (Reference) Figure 4 The specific operating mechanism of this module includes:

[0104] (3.1) Theme Color Mode Customization Mechanism: The system provides a variety of preset display theme modes, which users can choose according to their usage scenarios or which can be automatically switched by the system. The modes include:

[0105] Dark mode: Displays text against a dark gray background with high contrast white text by default, suitable for low-light environments;

[0106] Eye protection mode: Uses a low blue light warm-toned interface, suitable for nighttime or long-term wear;

[0107] Dynamic color matching mode: Based on the ambient light sensor detection results, the saturation, brightness value and contrast parameters of the theme color tone are automatically adjusted to match the current lighting conditions and minimize visual load.

[0108] Users can switch theme modes via voice commands (such as "switch eye protection mode"), touch commands, or rotary menu selection.

[0109] (3.2) Personalized font style and size setting mechanism: The system provides a variety of font styles (such as rounded, bold, and sans-serif fonts) and font size levels for users to set independently.

[0110] The font size is divided into four levels: small (4 vps), medium (6 vps), large (8 vps), and extra large (10 vps) to suit the reading habits of different users.

[0111] Font configuration parameters can be preset in the DIY editor or adjusted in real time on the AR device;

[0112] Based on the user's historical gaze duration and reading frequency, the system can recommend a font size optimization scheme suitable for the user.

[0113] (3.3) Environmental Context Adaptive UI Theme Linkage Mechanism: The system dynamically switches the interface display theme based on environmental sensor data (including light intensity, color temperature, and time period, etc.), specifically including:

[0114] When the ambient light intensity is >800 lux (such as strong outdoor light), the system automatically enables the highlight theme (enhanced outlines, bold borders, and enhanced button contrast).

[0115] When the ambient light is less than 100 lux (such as at night), the system automatically activates the low-light optimization mode, using dark backgrounds and light text and limiting the proportion of blue light on the interface.

[0116] It can combine IMU sensors to determine the current motion state (stationary / walking / vehicle-mounted state) and switch different UI layouts and theme colors to adapt to task requirements.

[0117] (3.4) Gradual transition and user intervention priority mechanism: The theme switching process adopts a smooth gradual transition animation of 250ms to avoid sudden interface changes affecting the continuity of user perception; if the user is detected to be performing interface operation, the theme switching is delayed and the update is performed after the current interaction is completed.

[0118] (4) Adaptive layout control module:

[0119] This module comes pre-loaded with various scenario-based interface templates and automatically selects or recommends suitable templates based on environmental data and user status. Referring to the corresponding attached diagrams, the specific configurations for each template are as follows:

[0120] a) Classic HUD mode (core information centered + high transparency floating side view, see reference) Figure 5 )

[0121] Applicable scenarios: driving, cycling, sports, navigation;

[0122] Layout features: Core information (such as navigation arrows, speed, time, etc.) is always centered and does not obstruct the view; secondary information (weather, battery level, etc.) floats on the side with adjustable transparency; voice / band operation switching window prevents information interference; gesture swipe up to bring up additional information (such as route details).

[0123] b) Immersive information flow mode (80% area vertical scrolling text + 20% fixed operation bar, see reference) Figure 6 )

[0124] Suitable scenarios: reading news, social media, note-taking;

[0125] Layout features: Similar to a mobile phone's scrolling information feed, supporting vertical scrolling to browse long texts;

[0126] c) Multi-tasking floating mode (3-window dynamic avoidance layout, see reference) Figure 7 )

[0127] Applicable scenarios: navigation, prompts, calls;

[0128] Layout features: Up to 3 floating windows can be set, which can be scaled / hung as needed; the core window is centered (such as the navigation), and other windows are semi-transparent and floating;

[0129] d) Minimalist power-saving mode (displays only time / battery charge, transparency >90%, see reference) Figure 8 )

[0130] Suitable for: daily wear, low power mode, outdoor sports;

[0131] Layout features: The display area shows only the most essential information (such as time, incoming calls, and emergency notifications); high information transparency is set to prevent visual interference and improve battery life; this mode is suitable for long-term wear and reduces eye fatigue;

[0132] e) Touch desktop mode (virtual planar projection + spatial gesture interaction, see reference) Figure 9 )

[0133] Applicable scenarios: wristband / finger ring control, augmented reality workbench; integrates AI-driven context-aware logic, automatically switching layout modes based on the following parameters: ambient light intensity (via ALS sensor), user motion status (classified by IMU data as walking / stationary / driving), and current foreground application type;

[0134] Layout features: The UI is presented as a virtual desktop, and the window can be dragged and resized; the UI is controlled by a touch wristband / finger ring knob, which supports precise operation; buttons can be "clicked" in the air, similar to the virtual keyboard experience; it is compatible with 3D UI components, making it suitable for engineering design / remote collaboration.

[0135] (5) The specific mechanism of the classic HUD mode

[0136] refer to Figure 5 The specific operating mechanism of the classic HUD mode includes:

[0137] (5.1) Core information anchoring mechanism;

[0138] Core information elements (such as navigation direction arrows, current speed, and system time) are anchored within the main view area. Their display position is finely adjusted according to the user's gaze focus and head posture, keeping them within ±5° of the center of the field of vision to ensure that the line of sight does not deviate from the road condition observation area.

[0139] (5.2) Dynamic adjustment mechanism for secondary information;

[0140] The side panel displays secondary information (such as weather, battery level, and notification summaries) in a floating format with high transparency (85%-95%) by default. Its transparency can be dynamically adjusted based on ambient brightness, and the content will rotate according to the user-defined priority. The content refresh uses a slide-in and slide-out animation to reduce visual interference.

[0141] (5.3) Contextual voice triggering adjustment mechanism;

[0142] The system has a built-in context recognition algorithm. When driving, cycling or exercising is detected, a fast voice interaction mode is activated, retaining only high-priority commands (such as "show route details" and "hide weather"), and wake-word-free monitoring is enabled. The keyword parsing module is automatically activated when the background noise is <60dB.

[0143] (5.4) Swipe-up gesture expansion area loading mechanism;

[0144] When the user performs an up swipe gesture, the system loads an extended information area in the bottom auxiliary area. This area displays information such as route segment details, estimated time, and traffic events in a card-style interface. The expanded area occupies no more than 25% of the bottom of the display screen and supports automatic collapse after the user finishes reading.

[0145] (5.5) Safe mode prompt logic;

[0146] When the system detects through the IMU that the user is in a high-speed moving state (such as vehicle speed > 30km / h) and has side information interaction behavior, the system will pop up a "safe interaction prompt layer". The transparency of this layer is no more than 40%, and it will remind the user "currently in a high-speed state, minimal interaction is recommended" within 5 seconds, and then automatically fade out;

[0147] (5.6) Semantic compression display strategy;

[0148] To fit the limited main view area display space, a semantic compression algorithm is used for text information, displaying only keywords / key icons (such as "Navigation: Turn right →" "Remaining: 2km"). The complete content is only presented after the user issues the "Expand Details" command or a two-finger zoom gesture, saving space while retaining key perceptions.

[0149] (5.7) Anti-obstruction layout optimization strategy;

[0150] Employing a layout engine based on heatmap analysis, it automatically avoids overlapping displays with navigation arrows and traffic signal areas within the main view area. Combined with vehicle / road condition recognition information, it prevents interface elements from obscuring important visual information. When obscuring is detected, it triggers "micro-displacement adjustment," horizontally shifting unnecessary elements by 5-10 pixels.

[0151] (6) Specific mechanisms of immersive information flow mode

[0152] refer to Figure 6 The specific operational mechanism of the immersive information flow model includes:

[0153] (6.1) Adaptive loading mechanism for scrolling information;

[0154] The main information display area supports vertical scrolling to display long text content. The system dynamically adjusts the text loading density and the rhythm of the text and image layout based on the user's scrolling speed and browsing time. When the system detects that the user has stayed for more than 3 seconds, it activates the content summary level collapse mechanism to display only paragraph titles and key icons.

[0155] (6.2) Immersive focus tracking layout logic;

[0156] The eye-tracking module determines the center of the current gaze area and automatically enhances text clarity (boldening fonts and increasing contrast) within ±5° of it, while reducing the transparency of non-gaze areas by 10%-20% to create a focused reading experience; this mechanism supports smooth dynamic switching and avoids visual jumps.

[0157] (6.3) Auxiliary information side floating layer control mechanism;

[0158] A highly transparent auxiliary layer is set on the right side of the main content area to display supplementary information selected by the user, including the number of comments, author information, tags, etc. This area is minimized by default and is only activated when the user taps the side edge with two fingers or issues a voice command such as "expand comments". After expansion, the floating layer occupies no more than 20% of the main interface.

[0159] (6.4) Semantic interactive tagging system;

[0160] The information stream text contains embedded interactive semantic tags. Users can interact with the content by gazing and tapping gestures or by activating voice commands (such as "save this paragraph" or "forward to X platform"). Tag types include "highlight mark", "save note", "social forwarding", "view original text", etc. After activation, feedback will pop up in the bottom area in the form of icon cards.

[0161] (6.5) Enhanced recognition mechanism for information scrolling gestures;

[0162] Different scrolling behaviors are defined for different swipe amplitudes: short, light swipes result in micro-scrolling (2-3 lines), medium-distance swipes trigger paragraph jumps, and large, rapid swipes trigger a "Quick Overview" mode, which displays only the title and a preview summary; swipe intentions are analyzed using gyroscope and linear acceleration to ensure that the probability of accidental touches is less than 1%.

[0163] (6.6) Nighttime eye protection and high brightness mode switching mechanism;

[0164] Based on ambient light sensor readings, the system automatically activates night mode (dark background, high contrast text, and blue light filtering) when brightness is below 100 lux; and activates high-contrast mode when brightness is above 800 lux, enhancing background grayscale and border thickness to ensure readability in sunlight. Manual mode switching is supported via user gestures (long press for 3 seconds) or voice commands.

[0165] (7) Specific mechanism of multi-task floating mode

[0166] refer to Figure 7 The specific operating mechanism of the multi-tasking floating mode includes:

[0167] (7.1) Hierarchical display mechanism for multi-task windows;

[0168] The system supports up to three floating task windows to be displayed in parallel. Based on content priority and user operation frequency, the task windows are divided into two categories: "core windows" and "auxiliary windows". The core task window is fixed in the center and occupies about 60% of the effective display area of ​​the interface. The auxiliary task windows are located on the side and occupy about 20% of the area on the left and right sides or the bottom, respectively, and are presented in a semi-transparent floating form.

[0169] (7.2) Dynamic control mechanism for window status;

[0170] Each floating window supports three levels of status adjustment, including: ① minimized to an icon tooltip (2mm); ② medium size display of task summary information; ③ maximized to display task details. Status switching can be completed via gesture operations (pinch / expand with two fingers), voice commands (such as "zoom in on navigation", "hide call") or smart knob (long press + rotate).

[0171] (7.3) Intelligent window avoidance and adsorption arrangement mechanism;

[0172] When multiple windows are active at the same time, the system uses a dynamic avoidance algorithm to adjust their positional relationship to avoid overlapping and occlusion. It also supports the edge snapping effect between windows, automatically snapping and aligning them when the user drags the window to the near boundary. The minimum spacing between windows is 5px to ensure visual clarity.

[0173] (7.4) Task type identification and automatic sorting mechanism;

[0174] The system determines the window type based on task semantic tags (such as navigation, schedule, call, etc.) and automatically sorts the display positions according to preset or user-defined priority strategies; the default priority is: navigation > call > schedule > messages > other, and the priority can be customized and adjusted through configuration files or voice commands (such as "call priority");

[0175] (7.5) Focus window highlighting and operation enhancement mechanism;

[0176] The currently focused window is identified by being positioned on top along the Z-axis, displaying a highlighted border, and having a prominent icon. Focus switching can be achieved by tapping the window edge, using the voice command "Focus on a task," or clicking the knob. Once the focused window is activated, a dedicated set of operation buttons is loaded, and the transparency of non-focused windows is automatically reduced to 30%-50%.

[0177] (7.6) Window grouping and pagination switching mechanism;

[0178] When the number of active windows exceeds the limit of the visible area or the system identifies a risk of space overlap, the auxiliary task window is automatically compressed into a tabbed view. Users can switch between task tabs by swiping or using the voice command "next page" to integrate information display and improve space utilization.

[0179] (7.7) Window layout memory and restoration mechanism;

[0180] The system automatically records the multi-tasking window layout state most recently used by the user, including parameters such as window position, size, transparency, and content tabs, and restores the previous view layout when the user enters the mode again. It supports calling multiple memory templates such as "recent layout", "default layout" and "minimized layout".

[0181] (7.8) Edge accidental touch protection mechanism;

[0182] After activating the multitasking window, the system sets a touch shielding layer on its outer edge (about 2mm wide) by default. When the system detects continuous accidental touches or occlusion actions by the user (lasting more than 3 seconds), it triggers a "Collapse this window?" prompt overlay. Users can adjust the activation status of this mechanism by saying "Turn off protection" via voice or by rotating the knob.

[0183] (8) The specific mechanism of the minimalist power-saving mode

[0184] refer to Figure 8 The specific operating mechanism of the minimalist power-saving mode includes:

[0185] (8.1) A simplified display mechanism for core information;

[0186] The system supports up to three floating task windows to be displayed in parallel. Based on content priority and user operation frequency, the task windows are divided into two categories: "core windows" and "auxiliary windows". The core task window is fixed in the center and occupies about 60% of the effective display area of ​​the interface. The auxiliary task windows are located on the side and occupy about 20% of the area on the left and right sides or the bottom, respectively, and are presented in a semi-transparent floating form.

[0187] (8.2) High transparency visual transparency control mechanism;

[0188] In the minimalist mode, the default setting for display layer transparency is greater than 90%, and users can fine-tune it using a smart knob, voice commands (such as "adjust transparency to 95%)", or swipe gestures. The displayed content uses a semi-transparent overlay with smooth edge transitions to enhance environmental integration and reduce visual burden.

[0189] (8.3) Design of a retractable quick operation area;

[0190] The auxiliary interaction area is hidden by default and only unfolds when the user executes a specific wake-up command (such as the voice command "show shortcut bar" or a two-finger swipe gesture); the operation area occupies no more than 20% of the interface and includes controls such as the voice assistant button, UI theme switching, and notification center wake-up, and automatically collapses after 10 seconds of inactivity.

[0191] (8.4) Low power consumption optimization and battery life enhancement mechanisms;

[0192] In minimalist mode, the system automatically reduces the refresh rate to 1Hz and adaptively reduces the screen brightness. It also adaptively controls the brightness according to the scene. When the ambient light is less than 200 lux, the screen brightness is automatically reduced by 50% and when the ambient light is less than 100 lux, the screen brightness is automatically reduced by 70%. At the same time, the number of background processes is limited, and only the core communication and sensor modules are kept active. The system power consumption is reduced by no less than 30%, which is suitable for outdoor or long-term wear scenarios.

[0193] (8.5) Mechanism for enhancing visual comfort during prolonged wear;

[0194] The system enables an eye-protection color mode, limiting blue light output to below 50% and adopting a warm-toned interface color scheme. Combined with adaptive font size, the system automatically enlarges or reduces the size of displayed text based on eye tracking and reading distance to prevent eye strain, dizziness, and other symptoms caused by prolonged wear.

[0195] (8.6) Automatic entry and exit determination mechanism;

[0196] When the system detects that the device has been stationary for more than 10 minutes, or the current application has been in the background for more than 3 minutes, it will automatically switch to the simplified mode and prompt "Simplified mode has been entered"; when the user interacts with voice, activates with gestures, or the IMU detects that the motion state has resumed, it will automatically exit the mode and restore the default layout.

[0197] (8.7) Information burst response strategy under minimalist mode;

[0198] When a high-priority notification is received (such as an incoming call, SOS alarm, or system error message), the system will briefly display a highlighted icon and text prompt at the top or center of the minimalist interface for no more than 5 seconds before fading out without interrupting the current information display rhythm.

[0199] (9) The specific mechanism of touch desktop mode

[0200] refer to Figure 9 The specific operating mechanism of the touch desktop mode includes:

[0201] (9.1) Virtual desktop planar projection mechanism;

[0202] The system constructs a virtual interactive plane within a range of 0.3m to 0.8m in front of the user using a spatial projection algorithm. Multi-window UIs are then fixed onto this virtual desktop in 2D or 3D format, and 6DoF interaction support is provided. The main view area occupies approximately 70% of the virtual interface and is used to load core content such as documents, 3D models, and charts.

[0203] (9.2) Window dragging and dynamic resizing mechanism;

[0204] The UI window supports dragging and resizing at any position in the virtual desktop. The window edge is equipped with drag handles, and users can make precise adjustments to the window width, height, and Z-axis depth through a smart bracelet or ring knob. It supports a scaling ratio of 0.5x to 2x, and the operation error does not exceed ±1mm.

[0205] (9.3) Floating menu and fine control function area;

[0206] The accessibility menu area floats on the right or bottom of the interface, occupying about 30% of the virtual desktop area. It includes commonly used controls such as rotation, zoom in / out, and switching perspectives. Each control is presented as an icon button and supports direct voice command invocation, such as "zoom in on the model" or "switch to top view".

[0207] (9.4) Simulation mechanism of air gesture clicks and virtual keyboard;

[0208] The system has a built-in air gesture recognition module that defines standard operation gestures such as "finger tap", "long press", and "swipe". The tap event is triggered by position trajectory closure detection and fingertip acceleration characteristics to simulate physical button feedback. It supports a virtual keyboard component with haptic vibration feedback and sound prompts for text input and command confirmation.

[0209] (9.5) 3D UI component adaptation mechanism;

[0210] The system supports loading 3D user interface elements, including 3D model preview controls, embedded data panels, transparent shell components, etc.; each component supports rotation, disassembly, and exploded view display, and provides a fusion interface with traditional 2D windows to realize a composite interactive space.

[0211] (9.6) High-precision ring / wristband knob control mechanism;

[0212] The system features an integrated rotary control interface, allowing users to quickly switch menus, adjust model posture, and modify parameter values ​​through rotation, click, and long press operations. The system achieves rotation input with a minimum resolution of 0.5° and a maximum response speed of 100ms through real-time angular velocity sampling and motion recognition algorithms.

[0213] (9.7) Scene adaptation and pose relocalization mechanism;

[0214] When a user moves or changes their posture, such as from standing to sitting, the system automatically repositions the virtual desktop plane to a reasonable viewing area based on IMU sensor data, and maintains the relative layout stability of each window to avoid user interaction interruptions due to changes in spatial position.

[0215] (10) Specific mechanisms of DIY UI editing and development module

[0216] refer to Figure 10 The specific operational mechanism of the DIY UI editing and development module includes:

[0217] (10.1) Mobile UI editing mode;

[0218] Users can edit the AR glasses display interface visually via a mobile app. The editor supports component dragging, attribute binding, preset theme switching, and transparency adjustment, specifically including:

[0219] The APP editor has an embedded layout simulation engine, which can load the AR glasses' partition logic model (main view area, side area, auxiliary area) and edit it synchronously.

[0220] The UI parameters set by the user (including component position, size, transparency, color scheme, font style, etc.) communicate with the AR glasses through a two-way synchronization channel;

[0221] The synchronization channel is built on the Bluetooth 5.2 low latency protocol, with a configuration response latency of less than 30ms, and supports parameter-level incremental synchronization and batch structure updates;

[0222] The editor supports saving solutions, calling solutions, and quick sharing via QR code.

[0223] (10.2) Real-time editing mode on AR devices;

[0224] A visual UI editing module runs on the AR glasses. This module has a "what you see is what you get" capability, allowing users to directly customize the interface through spatial operations. Specifically, it includes:

[0225] The AR real-time editor loads the current system interface state and uses holographic rendering to preview the editing changes in real time.

[0226] Users can select UI elements using spatial gestures such as gaze focus, ring swipe, and two-finger zoom, and then drag their position, resize them, adjust their transparency, and modify their layer order.

[0227] The system provides auxiliary tools for aligning component snapping boundaries with the grid to improve operational accuracy and interface standardization;

[0228] The editor supports automatically saving the current editing state, allowing users to roll back to any historical version of the interface at any time.

[0229] Users can call up theme templates or load custom appearance configuration files saved locally / in the cloud with one click and apply them to the current display interface in real time.

[0230] (10.3) Editing mode switching and permission control mechanism;

[0231] The system features dual-mode switching logic, allowing users to freely switch editing control between mobile and AR terminals. The currently active terminal has primary write permissions for interface parameters, while the other terminal is in read-only mode. It also supports an administrator-configurable permission setting mechanism, allowing device owners to restrict which components, partitions, or themes can be modified.

[0232] The specific development structure framework of the system is as follows: Figure 11 As shown, the development code mainly consists of four modules: cpp source code, ets source code, resources, and module configuration. These modules include the ARImage module for real-time AR image rendering, the multimodal interaction module for human-computer interaction, the UI control library editing module for providing different encapsulated controls, and the AR scene layout master page for storing different scene interfaces.

[0233] The specific mobile display effect is as follows: Figure 12 As shown.

[0234] The specific workflow of the system described in this embodiment is as follows:

[0235] After the system starts up, it first collects sensor data from the ambient light sensor, inertial measurement unit (IMU), and other sensors through the environmental perception module. The display management module initializes the display interface based on the current environmental data and user presets, dynamically dividing the physical display area into three logical partitions: the central main viewing area, the dual foldable side areas, and the bottom context-aware auxiliary area, and loading the default or recommended scenario-based interface templates.

[0236] During user interaction, the multimodal input processing module monitors and processes input signals from gestures, voice, and external devices (such as smart bracelets and rings) in real time. When a user makes a specific gesture using the nine-axis inertial measurement unit (such as swiping left / right to switch windows, swiping up / down to adjust brightness, pinching to zoom in / out of windows, or drawing a circle to bring up a shortcut menu), the gesture recognition unit identifies the gesture based on preset acceleration thresholds, duration thresholds, and trajectory feature parameters, and generates corresponding control commands.

[0237] When a user inputs commands via voice, the voice parsing unit performs noise reduction on the voice signal, converts it into text, performs semantic decomposition, matches it against a preset command library, extracts parameters, and finally generates a system API call chain to execute the corresponding operation. Users can also operate the device via the knob on a smart bracelet or ring. The external device interface unit receives the rotation encoding signal and uses a dynamic adjustment algorithm based on rotational angular velocity to control the window transparency and Z-axis layer in real time.

[0238] The adaptive control engine dynamically adjusts the display content, layout style, and visual parameters of each logical partition based on user input commands, environmental sensor data, and the current application scenario. The engine pre-sets various scenario-based interface templates, including classic HUD mode, immersive information flow mode, multi-tasking floating mode, minimalist power-saving mode, and touch desktop mode. It can automatically select or recommend suitable templates based on ambient light intensity, user movement status (classified as walking, stationary, or driving via IMU data), and the type of foreground application.

[0239] The personalized theme and appearance module allows users to customize the interface theme, colors, font styles, and font sizes in real time via a mobile app or AR device. The system supports environmental context adaptive linkage, dynamically switching the interface display theme based on data such as ambient light intensity and color temperature, and using smooth gradient animations during theme switching. If user interaction is detected, the switching is delayed.

[0240] The DIY UI editing and development module offers two editing modes: mobile and AR. In mobile app editing mode, users can drag and drop components, bind attributes, and switch themes using a visual editor. Configuration parameters are synchronized to the AR glasses via low-latency Bluetooth. In AR real-time editing mode, users can directly adjust UI elements using spatial gestures. The system provides component snapping and grid alignment tools, supports real-time preview and automatic saving.

[0241] The system also supports community sharing and user-generated content (UGC) mechanisms, allowing users to export layout schemes as standard format files (such as encrypted JSON files) and distribute them through the sharing mechanism. At the same time, it opens up third-party UI plugin interfaces to load externally developed theme packages and interactive scripts through RESTful interfaces.

[0242] Through the coordinated operation of the above modules, this invention achieves efficient human-computer interaction in a small AR display area, significantly improving information density and user control freedom, and effectively reducing visual fatigue rate.

[0243] Example 2

[0244] Based on the interaction method disclosed in Embodiment 1, this embodiment provides a user interaction system for a small-sized AR smart glasses display interface. This system includes a display management module, a multimodal input processing module, an adaptive control engine, and an environment perception module. Their specific configurations, functions, and collaborative working methods are completely identical to the descriptions of "(1) Dynamic Partition Information Management Display Module," "(2) Multimodal Interaction Control Module," "(4) Adaptive Layout Control Module," and related environment perception components in Embodiment 1, and can implement the method steps as described in any one of claims 1 to 7. Those skilled in the art can undoubtedly determine the specific implementation of this system based on the description in Embodiment 1.

[0245] Similarly, those skilled in the art can program the method described in Embodiment 1 into a computer program using conventional programming methods and store it on a computer-readable storage medium (e.g., flash memory inside AR glasses, solid-state drive in a device system, or external server memory) to form the storage medium as described in claim 10. When the computer program on the storage medium is loaded and executed by a processor (e.g., the central processing unit or graphics processor of AR glasses), the method as described in any one of claims 1 to 7 can be implemented.

[0246] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A small size AR smart glasses display interface user interaction method, characterized in that, The method comprises the following steps: a dynamic partition display step: dynamically dividing the physical display area of the smart glasses into three logical partitions, namely a central main viewing area, double-sided foldable side area and a bottom context-aware auxiliary area, and configuring different priority contents for each partition; a multi-modal interaction processing step: receiving user input instructions by fusing gesture recognition, voice recognition and external device control, wherein the gesture recognition is based on inertial measurement unit data, the voice recognition supports custom instructions, and the external device control includes the knob operation of a smart bracelet or ring; an adaptive interface control step: dynamically adjusting the display content, layout style and visual parameters of each logical partition based on user input instructions, environmental sensor data and current application scenarios, including window size, transparency and layout mode adjustment; wherein the output of the dynamic partition display step provides an interface operation basis for the multi-modal interaction processing step, the output of the multi-modal interaction processing step provides a basis for adjustment for the adaptive interface control step, and the three steps cooperatively realize efficient human-computer interaction under a small-size display area.

2. The method of claim 1, wherein, The dynamic partition display step comprises: The physical display area is divided into three logical partitions in proportion to the area, wherein the central main viewing area accounts for 40%, used for displaying real-time subtitles, navigation instructions or emergency notifications; the double-sided foldable side area each accounts for 30%, which is hidden by default as an edge indicator, and can be expanded to display weather, schedule or multi-task thumbnails; the bottom context-aware auxiliary area accounts for 20%, dynamically loading icon groups according to the current application scenario; Window size is adjusted in three stages, transparency is controlled steplessly from 50% to 100%, and near-far focus mode is adaptively switched.

3. The method of claim 1, wherein, The multi-modal interaction processing step comprises: User gestures are detected by a nine-axis inertial measurement unit, defining left / right sliding to switch windows, up / down sliding to adjust brightness, pinching to zoom windows, and circling to call a shortcut menu, and setting acceleration threshold, duration threshold and trajectory feature parameters for each gesture; A voice recognition engine supports user-defined voice instructions, and after receiving voice input, noise reduction processing, semantic decomposition and parameter extraction are performed to generate a system API call chain to execute corresponding operations; A rotary encoder signal is received through the knob interface of a smart bracelet or ring to control window transparency and Z-axis level in real time, and a dynamic adjustment algorithm based on rotary angular velocity is designed.

4. The method of claim 1, wherein, The adaptive interface control step comprises: Multiple scenario-based interface templates are preset, including classic HUD mode, immersive information flow mode, multi-task suspension mode, minimalist power-saving mode and touch desktop mode; Based on ambient light sensor data, IMU motion state data and the current foreground application type, an appropriate interface template is automatically selected or recommended; Mobile and AR end-to-end real-time visual editing functions are provided to support user-defined interface layout, component properties and interaction logic.

5. The method of claim 4, wherein, The scenario-based interface template at least comprises: The classic HUD mode is configured to display core information in the center, display secondary information with high transparency on the side, and support core information anchoring, intelligent anti-shielding and context-aware interaction simplification; An immersive information flow mode configured to use most of the area for vertically scrolling text display, supporting scroll adaptive loading, focus tracking typography, and semantic interactive tags; A multitasking floating mode configured to support dynamic avoidance layout of multiple floating windows, providing window hierarchical management, state adjustment, and intelligent avoidance mechanism.

6. The method of claim 1, wherein, The method further comprises an environment adaptation step: Real-time detection of light intensity through an ambient light sensor, automatic switching of interface theme and display parameters according to the detection results, including enabling a dark light optimization mode when the light intensity is below a first threshold, and enabling a high-light contrast mode when the light intensity is above a second threshold; Recognizing user motion state through an inertial measurement unit, dynamically adjusting interface layout and interaction mode according to walking, stationary, or vehicle-mounted state; The theme switching process uses a smooth gradient animation, and if it is detected that the user is performing interface operations, the switching is delayed.

7. The method of claim 1, wherein, The method further comprises a DIY editing and sharing step: Providing a mobile APP visual editor, supporting component dragging, property binding, and theme switching, configuration parameters are synchronized to the AR glasses end through a low-latency Bluetooth protocol; Providing an AR end real-time visual editing module, supporting direct adjustment of UI elements through spatial gestures, providing component adsorption and grid alignment auxiliary tools; Supporting exporting layout schemes to standard format files and distributing them through sharing mechanisms, opening third-party UI plug-in interfaces to load external theme packages and interaction scripts.

8. A small size AR smart glasses display interface user interaction system for implementing the method of any one of claims 1-7, characterized in that, Comprise: A display management module configured to perform the dynamic partition display step, manage the logical division of the physical display area and content rendering; A multi-modal input processing module configured to perform the multi-modal interaction processing step, process gesture, voice, and external device input signals; An adaptive control engine configured to perform the adaptive interface control step, control interface adaptive adjustment based on input signals and environmental data; An environment perception module configured to collect and process environmental sensor data; Wherein, each module communicates and cooperates through a data bus and interface protocol, and together realizes the optimized interaction experience under small size display area.

9. The system of claim 8, wherein, The display management module comprises: A partition management sub-module for maintaining layout parameters and display attributes of the three logical partitions; A rendering optimization sub-module for optimizing rendering effects according to display content types and environmental conditions; A transparency control sub-module for dynamically adjusting the transparency of each partition and interface elements. The multi-modal input processing module includes a gesture recognition unit, a speech analysis unit, and an external device interface unit.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 7.

Images