An augmented reality glasses multi-modal user interaction intention recognition method and system based on modal missing modeling

Abstract

The application belongs to the field of multi-modal user interaction intention understanding, and discloses an augmented reality glasses multi-modal user interaction intention recognition method and system based on modal missing modeling, which comprises the following steps: constructing a multi-modal interaction dataset for an augmented reality glasses scene; extracting features from the multi-modal interaction dataset; statistically analyzing the extracted modal features to obtain multi-modal samples; constructing a multi-modal user interaction intention recognition model; and inputting the multi-modal samples into the trained multi-modal user interaction intention recognition model to recognize the user interaction intention.

Description

Technical Field

[0001] This invention belongs to the field of multimodal user interaction intent understanding, and specifically relates to a method and system for recognizing multimodal user interaction intent for augmented reality glasses based on modality missing modeling. Background Technology

[0002] With the continuous development of Augmented Reality (AR) technology, wearable computing platforms, and mobile sensing technologies, AR glasses are gradually moving from experimental devices to practical application platforms, demonstrating broad application prospects in scenarios such as cultural tourism guidance, smart education, industrial assistance, and medical rehabilitation. In immersive scenarios such as museum tours and office / study assistance, users expect to interact with virtual information in a more natural, less burdensome, and more efficient way, such as through multiple channels like voice, gestures, head movements, gaze, and scene perception to express goals and control operations. This places higher demands on the human-computer interaction capabilities of AR glasses, especially their ability to accurately understand and respond to user interaction intentions in real time. How to build a natural, accurate, and real-time human-computer interaction mechanism for AR glasses platforms has become an important research question in the fields of augmented reality and wearable computing.

[0003] Intent recognition is a key technology for inferring potential goals and operational needs from user input signals, and it is an important component of intelligent human-computer interaction systems, dialogue systems, and context-aware systems. Early intent recognition methods were mostly based on single-modal data such as text or speech. However, in real-world open environments, single-modal solutions are easily affected by factors such as noise, occlusion, changes in viewing angle, wearer jitter, and limited device computing power, leading to a significant decrease in recognition performance. Taking augmented reality glasses as an example, speech signals are easily interfered with in noisy environments, and visual gesture information may be distorted under low light or occlusion conditions. Furthermore, changes in the wearer's motion state further increase the instability of sensor signals. Integrating multi-source heterogeneous information such as speech, images, and inertial sensors (IMUs) to construct a multi-modal intent recognition method with noise resistance, anti-missing information, and dynamic adaptability has become a key path to improve the robustness of augmented reality glasses interactions.

[0004] In recent years, multimodal intent recognition has become an important research direction in multimodal learning and intelligent interaction. Related reviews indicate that researchers are gradually shifting from traditional single-modal recognition methods to deep learning methods that integrate heterogeneous information from multiple sources such as language, vision, acoustics, and sensors, improving the model's adaptability to complex scenes and its recognition accuracy. While significant progress has been made in multimodal intent recognition and augmented reality glasses interaction research, the following shortcomings still exist.

[0005] First, research on the dynamic changes of modal states in mobile scenarios for augmented reality glasses is still insufficient. Many existing methods assume that each modality is stable and usable throughout the interaction process. However, in real-world wearing environments, modalities such as voice, gesture, vision, and IMU may experience short-term failures, signal degradation, and quality fluctuations.

[0006] Second, while existing general multimodal architectures have strong unified modeling capabilities, there is still a lack of research on targeted adaptation for augmented reality glasses interaction tasks. In particular, how to combine the differences in semantic stability, temporal resolution and noise characteristics of different modalities to build a fusion mechanism suitable for device intent recognition still needs further exploration.

[0007] Third, there is still a lack of dedicated multimodal interactive datasets for vertical scenarios such as office and learning assistance and museum interpretation. Related reviews point out that existing publicly available datasets still suffer from issues such as inconsistencies in task definition, idealized collection environments, and insufficient coverage of modal types compared to real-world applications. [3] This, to some extent, restricts the research and verification of the model in practical application scenarios of augmented reality glasses.

[0008] Therefore, constructing a dedicated dataset for augmented reality glasses scenarios and designing a multimodal intent recognition model that balances robustness and real-time performance has clear research value and practical significance.

[0009] From a theoretical perspective, this invention focuses on the problem of multimodal robust intent recognition in open and dynamic environments, emphasizing cross-modal semantic alignment, modal dynamic contribution modeling, and robust fusion mechanisms under modality missing conditions. This contributes to a deeper understanding of multimodal collaborative representation and robust decision-making mechanisms. From an application perspective, achieving high-precision, low-latency, and fault-tolerant intent recognition on augmented reality glasses can significantly improve the interactive experience in scenarios such as museum guided tours and office / study assistance, reduce user operating costs, enhance the naturalness and continuity of AR content services, and provide key technical support for the large-scale deployment of augmented reality glasses in cultural tourism, education, and other fields. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a method and system for recognizing multimodal user interaction intent in augmented reality glasses based on modality missing modeling. The aim is to construct a robust extension model for modality missing. This model uses mechanisms such as anchor dropout, missing token, availability embedding, and selective modality-specific gates for IMU and speech to enable the model to learn decision-making methods under the condition of "partial failure of key modalities" during the training phase, thereby improving robustness in real-world complex environments.

[0011] To achieve the above objectives, the present invention provides the following solution: A method for recognizing multimodal user interaction intent in augmented reality glasses based on modality missing modeling, the method comprising: Construct a multimodal interaction dataset for augmented reality glasses scenarios; Feature extraction is performed on the multimodal interaction dataset; Statistical analysis was performed on the extracted modal features to obtain multimodal samples; Construct a multimodal user interaction intent recognition model; Multimodal samples are input into a trained multimodal user interaction intent recognition model to identify user interaction intents.

[0012] Preferably, the process of constructing a multimodal interaction dataset for augmented reality glasses scenarios includes: To identify application scenarios for augmented reality glasses; Analyze user interaction behavior in augmented reality glasses application scenarios to obtain user interaction intent; Based on the user's interaction intent, HoloLens and an external fisheye camera were used to collect data of different modalities to generate multimodal interaction samples. Data preprocessing and time alignment are performed on multimodal interaction samples to obtain a multimodal interaction dataset for augmented reality glasses scenarios.

[0013] Preferably, the methods for data preprocessing and time alignment of multimodal interaction samples include: The audio stream from the video recorded by HoloLens is selected as the time alignment reference, and effective speech segments are extracted through Voice Activity Detection (VAD) to generate an approximate timestamp for each interaction sample. The approximate timestamp of each interaction sample is used as a unified time anchor point to align external fisheye camera video, audio, text, and IMU modalities.

[0014] Preferably, feature extraction of the multimodal interaction dataset includes: gesture feature extraction, audio feature extraction, text feature extraction, IMU feature extraction, and scene modality feature extraction; The gesture features are represented as follows: Audio features are represented as follows: , The MFCC feature frame number of the current audio window; the text modal features are represented as: The IMU modal characteristics are represented as follows: Scene modal features are represented as follows: .

[0015] Preferably, methods for constructing multimodal user interaction intent recognition models include: A five-modal Perceiver-IO model under real-world conditions is constructed, and an anchor-residual improvement model and a robust extension model are further introduced on the basis of the five-modal Perceiver-IO model. The methods for constructing the anchor-residual improvement model include: Construct an anchor trunk consisting of three modalities: gesture, text, and scene; After the anchor trunk is established, the two supporting modes, IMU and audio, are used as residual supplementary branches, and auxiliary information is injected in a residual manner. Methods for constructing robust expansion models include: An explicit missing modeling mechanism is introduced within the anchor trunk.

[0016] Preferred methods for constructing a five-modal Perceiver-IO model under real-world scenario conditions include: The five modalities are converted into fixed-length token sequences: for the four modalities of IMU, gesture, audio, and text, a temporal representation of length 10 is used; for the scene modality, a single global token is used. They are uniformly projected to the same latent space dimension using modality-specific linear mappings. : ; in, The latent space features after projection of each modality are preserved in the temporal mode. Step, scene modality is One token; It is a learnable linear transformation with no bias; After projection, time position encoding is added to the four time-series modes. At the same time, learnable modality codes are added to each modality. ; Only modal coding is added to the scene modalities This is to represent its semantic identity as a global context token: For timing modes At the t-th time step, t = 1, 2, …, 10: ; in, Let m be the original projected features of mode m at time step t. This is the final input token after encoding; For scene modalities, the encoded representation is as follows: ; in, This serves as the final input token for the scene modality; After projection and encoding are completed, the five modalities are concatenated along the token dimension to form a unified input sequence: ; The model input consists of a total of 41 tokens, including 40 time-series tokens and 1 scene token; After completing the input tokenization, Perceiver-IO is used to uniformly model the five modal inputs. The output of each sample is defined as a joint label of scene and intent, thus completing the construction of the five modal Perceiver-IO model under real scene conditions.

[0017] Preferred methods for constructing an anchor backbone consisting of three modalities: gesture, text, and scene include: Will The three modalities are each mapped to a unified latent space dimension through linear projection. get Subsequently, for gesture and text modalities, learnable temporal positional encodings are superimposed. With modal coding This forms a unified anchor token sequence. For scene modalities, only modal encoding is added without temporal position encoding to obtain the token sequence. The anchor token sequence is concatenated in chronological order to obtain the anchor input sequence: ; Input anchor points into the sequence Feed into the Perceiver encoder: This encoder first compresses the input information into a set of learnable latent arrays through cross-attention. In; specifically, with As a query, with As keys and values, calculate cross-attention: ; in To create a learnable projection matrix, key information from the input sequence is extracted and written into the latent array, achieving a compressed mapping from the high-dimensional input space to the low-dimensional latent space. Subsequently, multiple layers of self-attention and feedforward networks are stacked within the latent space to perform deep interaction and feature refinement on the latent representation. l The layer is calculated as follows: ; go through l After stacking the layers, the final latent representation is obtained. This completes the deep fusion and global modeling of cross-modal information; after processing by the encoder, a learnable output query vector is used. By using decoder cross-attention to read task-related information from the latent representation, the anchor backbone representation is obtained. : .

[0018] Preferably, after the anchor trunk is established, the method of using the IMU and audio support modes as residual supplementary branches and injecting auxiliary information in a residual manner includes: The IMU and audio modal are represented by anchor points through cross-attention readout modules. As a query, a sequence of tokens for each modality. As keys and values, extract supplementary features that are most relevant to the current main semantics: ; in, These are supplementary features extracted from the IMU and audio modal that are most relevant to the anchor backbone semantics, and are used for subsequent residual injection. The supplementary features are superimposed on the anchor backbone representation as residuals to form the final fused representation: ; in For the weighted residuals of the two supporting modes, , For dynamic gating values, , Hyperparameters for controlling the maximum injection intensity.

[0019] Preferred explicit missing modeling mechanisms include: anchor point random discarding and missing token mechanism, availability embedding mechanism, and selective element-by-element gating for IMU + Audio; The anchor point random discarding mechanism is as follows: on a modal basis, the two anchor point branches, gesture and text, are randomly retained or set to empty. The missing token mechanism is as follows: Let the original token representations of gesture and text be respectively... and If the input is missing, then the replacement is: ; in, and The learnable missing tokens for gesture and text, respectively; This represents the input when the gesture is missing. This indicates input when text is missing. The availability embedding mechanism is as follows: ;in, For the final input format, This indicates the state embedding when the modality is available. This represents the state embedding when the modality is missing. This is the projection feature matrix when the modality is available. A learnable matrix of missing tokens; Selective element-wise gating for IMU + Audio is as follows: Assume supported modal inputs are... Its gated output is represented as: ; Among them, the gate matrix The calculation method is as follows: ; In the formula For the Sigmoid function, For learnable weight matrix, For learnable bias, This is the learnable gating bias term for this mode.

[0020] The present invention also provides an augmented reality glasses multimodal user interaction intent recognition system based on modality missing modeling. The system is used to implement the aforementioned method and includes: a dataset construction module, a feature extraction module, a feature statistics module, a model construction module, and an intent recognition module. The dataset construction module is used to construct a multimodal interaction dataset for augmented reality glasses scenarios; The feature extraction module is used to extract features from the multimodal interaction dataset; The feature statistics module is used to perform statistical analysis on the extracted modal features to obtain multimodal samples; The model building module is used to build a multimodal user interaction intent recognition model; The intent recognition module is used to input multimodal samples into a trained multimodal user interaction intent recognition model to identify user interaction intents.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a robust modeling strategy for missing anchor modalities. Addressing the issue of incomplete key modalities caused by factors such as gesture occlusion, speech recognition failure, and sensor fluctuations, this invention, based on the anchor backbone model, explicitly models the missing states of key anchor modalities such as gestures and text through random anchor discarding, missing tokens, and availability embedding. This allows the model to learn discrimination methods under incomplete input conditions during the training phase. Simultaneously, considering that IMU and audio are more susceptible to noise interference in real-world scenarios, this invention introduces selective element-wise gating only for these two support branches, performing lightweight cleaning of support features without affecting the representation of the backbone modalities. This robustness scheme does not simply rely on random zeroing or uniform gating; instead, it addresses two specific issues: how to maintain core semantic representation when the backbone modalities are missing, and how to reduce noise interference when support modalities are unstable, thereby improving robustness in complex real-world environments. Attached Figure Description

[0022] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of gestures for six types of interactive intentions according to an embodiment of the present invention, wherein (a) is a menu; (b) is selection; (c) zoom in; (d) zoom out; (e) is a pen; and (f) cancel. Figure 2 This is a flowchart illustrating the data acquisition process in an embodiment of the present invention. Figure 3 This is a schematic diagram of the overall process for feature extraction of various modes in an embodiment of the present invention; Figure 4 This is a diagram illustrating the overall framework of multimodal recognition for augmented reality glasses according to an embodiment of the present invention. Figure 5 This is a schematic diagram of the model structure in an embodiment of the present invention; Figure 6 This is a block diagram of a multimodal user interaction intent recognition model based on the anchor-residual primary and secondary modal fusion mechanism in an embodiment of the present invention. Figure 7This is a schematic diagram of the extended part of the robustness model in an embodiment of the present invention; Figure 8 This is a schematic diagram of the model training process curve in an embodiment of the present invention; Figure 9 This is a schematic diagram of the model classification confusion matrix in an embodiment of the present invention; Figure 10 This is a schematic diagram of the robust model training process curve in an embodiment of the present invention; Figure 11 This is a schematic diagram of the classification confusion matrix of a robust model with partial modality loss according to an embodiment of the present invention; Figure 12 This is a schematic diagram of a method for recognizing multimodal user interaction intent for augmented reality glasses based on modality missing modeling, according to an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1 Existing methods for recognizing multimodal user interaction intentions in augmented reality (AR) glasses have achieved good joint recognition results by combining a gesture-text-scene anchor backbone with an IMU-speech residual branch. However, these models assume that key anchor modalities are stable and available during the inference phase, an assumption that does not always hold true in real AR glasses interaction scenarios. For example, the speech modality may fail due to environmental noise or errors in automatic speech recognition, and the gesture modality may be missing due to occlusion, insufficient lighting, or hands going out of bounds, resulting in incomplete backbone representations. To improve the stability of the model under incomplete input conditions, this invention provides an AR glasses multimodal user interaction intention recognition method based on modality loss modeling. This invention constructs a robust extension model oriented towards modality loss. This model employs mechanisms such as anchor dropout, missing tokens, availability embedding, and selective modality-specific gating for IMU and speech to learn decision-making methods under conditions of "partial failure of key modalities" during the training phase, thereby improving robustness in complex real-world environments. Figure 12 As shown, the method includes: Construct a multimodal interaction dataset for augmented reality glasses scenarios; Feature extraction is performed on the multimodal interaction dataset; Statistical analysis was performed on the extracted modal features to obtain multimodal samples; Construct a multimodal user interaction intent recognition model; Multimodal samples are input into a trained multimodal user interaction intent recognition model to identify user interaction intents.

[0027] In this embodiment, the process of constructing a multimodal interaction dataset for augmented reality glasses scenarios includes: To identify application scenarios for augmented reality glasses; Analyze user interaction behavior in augmented reality glasses application scenarios to obtain user interaction intent; Based on the user's interaction intent, HoloLens and an external fisheye camera were used to collect data of different modalities to generate multimodal interaction samples. Data preprocessing and time alignment are performed on multimodal interaction samples to obtain a multimodal interaction dataset for augmented reality glasses scenarios.

[0028] In this embodiment, the method for data preprocessing and time alignment of multimodal interaction samples includes: The audio stream from the video recorded by HoloLens is selected as the time alignment reference, and effective speech segments are extracted through Voice Activity Detection (VAD) to generate an approximate timestamp for each interaction sample. The approximate timestamp of each interaction sample is used as a unified time anchor point to align external fisheye camera video, audio, text, and IMU modalities.

[0029] In this embodiment, feature extraction of the multimodal interaction dataset includes: gesture feature extraction, audio feature extraction, text feature extraction, IMU feature extraction, and scene modality feature extraction; The gesture features are represented as follows: Audio features are represented as follows: , The MFCC feature frame number of the current audio window; the text modal features are represented as: The IMU modal characteristics are represented as follows: Scene modal features are represented as follows: .

[0030] In this embodiment, the method for constructing a multimodal user interaction intent recognition model includes: A five-modal Perceiver-IO model under real-world conditions is constructed, and an anchor-residual improvement model and a robust extension model are further introduced on the basis of the five-modal Perceiver-IO model. The methods for constructing the anchor-residual improvement model include: Construct an anchor trunk consisting of three modalities: gesture, text, and scene; After the anchor trunk is established, the two supporting modes, IMU and audio, are used as residual supplementary branches, and auxiliary information is injected in a residual manner. Methods for constructing robust expansion models include: An explicit missing modeling mechanism is introduced within the anchor trunk.

[0031] In this embodiment, the method for constructing a five-modal Perceiver-IO model under real-world scenario conditions includes: The five modalities are converted into fixed-length token sequences: for the four modalities of IMU, gesture, audio, and text, a temporal representation of length 10 is used; for the scene modality, a single global token is used. They are uniformly projected to the same latent space dimension using modality-specific linear mappings. : ; in, The latent space features after projection of each modality are preserved in the temporal mode. Step, scene modality is One token; It is a learnable linear transformation with no bias; After projection, time position encoding is added to the four time-series modes. At the same time, learnable modality codes are added to each modality. ; Only modal coding is added to the scene modalities This is to represent its semantic identity as a global context token: For timing modes At the t-th time step, t = 1, 2, …, 10: ;in, Let m be the original projected features of mode m at time step t. This is the final input token after encoding; For scene modalities, the encoded representation is as follows: ;in, This serves as the final input token for the scene modality; After projection and encoding are completed, the five modalities are concatenated along the token dimension to form a unified input sequence: ; The model input consists of a total of 41 tokens, including 40 time-series tokens and 1 scene token; After completing the input tokenization, Perceiver-IO is used to uniformly model the five modal inputs. The output of each sample is defined as a joint label of scene and intent, thus completing the construction of the five modal Perceiver-IO model under real scene conditions.

[0032] In this embodiment, the method for constructing an anchor backbone composed of three modalities: gesture, text, and scene includes: Will The three modalities are each mapped to a unified latent space dimension through linear projection. get Subsequently, for gesture and text modalities, learnable temporal positional encodings are superimposed. With modal coding This forms a unified anchor token sequence. For scene modalities, only modal encoding is added without temporal position encoding to obtain the token sequence. The anchor token sequence is concatenated in chronological order to obtain the anchor input sequence: ; Input anchor points into the sequence Feed into the Perceiver encoder: This encoder first compresses the input information into a set of learnable latent arrays through cross-attention. In; specifically, with As a query, with As keys and values, calculate cross-attention: ; in To create a learnable projection matrix, key information from the input sequence is extracted and written into the latent array, achieving a compressed mapping from the high-dimensional input space to the low-dimensional latent space. Subsequently, multiple layers of self-attention and feedforward networks are stacked within the latent space to perform deep interaction and feature refinement on the latent representation. l The layer is calculated as follows: ; go through l After stacking the layers, the final latent representation is obtained. This completes the deep fusion and global modeling of cross-modal information; after processing by the encoder, a learnable output query vector is used. By using decoder cross-attention to read task-related information from the latent representation, the anchor backbone representation is obtained. : .

[0033] In this embodiment, after the anchor trunk is established, the method of injecting auxiliary information by treating the IMU and audio support modes as residual supplementary branches and using a residual approach includes: The IMU and audio modal are represented by anchor points through cross-attention readout modules. As a query, a sequence of tokens for each modality. As keys and values, extract supplementary features that are most relevant to the current main semantics: ; in, These are supplementary features extracted from the IMU and audio modal that are most relevant to the anchor backbone semantics, and are used for subsequent residual injection. The supplementary features are superimposed on the anchor backbone representation as residuals to form the final fused representation: ;in, For the weighted residuals of the two supporting modes, , For dynamic gating values, , Hyperparameters for controlling the maximum injection intensity.

[0034] In this embodiment, the explicit missing modeling mechanism includes: anchor point random discarding and missing token mechanism, availability embedding mechanism, and selective element-by-element gating for IMU + Audio; The anchor point random discarding mechanism is as follows: on a modal basis, the two anchor point branches, gesture and text, are randomly retained or set to empty. The missing token mechanism is as follows: Let the original token representations of gesture and text be respectively... and If the input is missing, then the replacement is: ; in, and The learnable missing tokens for gesture and text, respectively; This represents the input when the gesture is missing. This indicates input when text is missing. The availability embedding mechanism is as follows: ;in, For the final input format, This indicates the state embedding when the modality is available. This represents the state embedding when the modality is missing. This is the projection feature matrix when the modality is available. A learnable matrix of missing tokens; Selective element-wise gating for IMU + Audio is as follows: Assume supported modal inputs are... Its gated output is represented as: Among them, the gate matrix The calculation method is as follows: In the formula For the Sigmoid function, For learnable weight matrix, For learnable bias, This is the learnable gating bias term for this mode.

[0035] Example 2 The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0036] Construction of a multimodal interaction dataset: This invention mainly introduces the process of constructing a multimodal interaction dataset for augmented reality glasses scenarios, including interaction scenario and intent design, data acquisition platform and equipment, data preprocessing and time alignment methods, and feature extraction process for each modality, providing a data foundation for subsequent model training and experimental verification.

[0037] Interaction scenarios and intent design: Application scenario selection: This invention selects two typical scenarios that are representative and suitable for the application of augmented reality glasses: museum artifact interpretation scenarios and office and study assistance scenarios.

[0038] Museum artifact interpretation scenario: When visiting a museum, users access supplementary information about artifacts, including text, images, and videos, through augmented reality glasses. They can interact with the artifacts via voice or gestures, such as zooming in to view details, switching between different narration sections, and saving information of interest. This scenario is characterized by changing ambient lighting, complex background noise, and diverse user actions, placing high demands on the robustness of multimodal intent recognition.

[0039] Office and study assistance scenario: Users in an office environment use augmented reality glasses to view task lists, document summaries, or perform simple content editing, completing operations such as information retrieval and task marking through natural interaction. This scenario is relatively quiet, but there may be situations such as rapid head movements and gesture occlusion, which also tests the stability of modal fusion.

[0040] Intent category definition: Based on the analysis of typical user interaction behaviors in the above scenarios, this invention defines six types of core interaction intents, as shown in Table 1.

[0041] Table 1. Definition of Scene Intent Categories Typical gesture examples corresponding to the six types of intentions are as follows: Figure 1 As shown in (a) to (f). Among them: (a) Menu gesture - palm open, fingers naturally spread; (b) Select gesture - index finger extended forward, other fingers clenched into a fist; (c) Enlarge gesture - two fingers spread, simulating an enlarging action; (d) Shrink gesture - two fingers pinched together, simulating a shrinking action; (e) Paintbrush gesture - index finger making a writing motion; (f) Cancel gesture - palm wave.

[0042] Data acquisition platform and equipment: Hardware platform: This invention employs a HoloLens-based multimodal acquisition scheme to construct an augmented reality glasses interaction dataset. Considering the joint modeling requirements of speech, motion state, and first-person perspective visual information for user interaction intent recognition tasks, this invention uses HoloLens as the core acquisition platform, utilizing its built-in sensors to acquire audio and IMU data, while simultaneously fixing a binocular fisheye camera externally to the HoloLens to acquire first-person perspective video data. This combined acquisition method allows for the simultaneous acquisition of three key types of information: speech, inertial sensing, and vision, providing a data foundation for subsequent multimodal intent recognition models.

[0043] HoloLens primarily collects audio and IMU data. The audio modality records the user's voice commands during interaction, reflecting the user's explicit operational intentions; the IMU modality records the user's head movements and posture changes, providing auxiliary information for recognizing dynamic features during interaction. Compared to solutions relying solely on vision or voice, IMU data can supplement information on head movement trends and operational rhythm to a certain extent, thereby enhancing the system's ability to perceive complex interaction states.

[0044] In terms of visual modality acquisition, this invention does not directly use the head-mounted display's native visual data as the primary input. Instead, a fisheye camera is attached to the outside of the HoloLens to obtain first-person perspective video more suitable for the tasks of this invention. The main reason for using a fisheye camera is that it has a large field of view, which can more completely cover the scene in front of the user and the area of ​​hand activity, thus making it more conducive to capturing gestures and scene information that occur during natural interactions. Through further processing of the acquired fisheye video, this invention can extract two types of visual modalities: gesture fisheye images and scene images, providing a foundation for the construction of anchor modalities in subsequent models.

[0045] Overall, this data acquisition platform establishes a multimodal data acquisition approach: "HoloLens captures audio and IMU data, while an external fisheye camera captures video data." This solution preserves the naturalness and mobility of the head-mounted device in real-world interactive environments, while also expanding the field of view through an external fisheye visual modality, thus better meeting the data acquisition needs for multi-channel user interaction intent recognition in augmented reality glasses scenarios.

[0046] Data collection process: Data collection is conducted in two real-world environments: museum galleries and office environments. The data collection process of this invention is as follows: Figure 2 As shown. The entire process includes equipment preparation and installation, participants performing preset interactive tasks, recording multimodal raw data, and post-processing steps such as timestamp extraction, cross-device time alignment, and sample construction.

[0047] like Figure 2 As shown, during the data acquisition phase, participants wear HoloLens to complete preset interactive tasks. HoloLens is responsible for acquiring audio and IMU data, while an external fisheye camera is responsible for acquiring first-person perspective video data. Both devices independently record raw data and their respective timestamps during acquisition, without hardware-level real-time synchronization. After acquisition, this invention extracts the timestamp information from both devices in the post-processing phase, establishes a unified timeline, and combines it with voice activity detection results to complete cross-device time alignment and sample segmentation, ultimately generating multimodal interaction samples for training and testing.

[0048] Data preprocessing and time alignment: Voice activity detection and timestamp generation: Since this invention uses HoloLens and an external fisheye camera to collect data of different modalities, and no hardware-level real-time synchronization is performed during the acquisition phase, a unified time reference needs to be established in the post-processing phase. Considering that voice commands usually have a strong consistency with the time of occurrence of user interaction intentions, this invention selects the audio stream in the video recorded by HoloLens as the time alignment benchmark, and extracts effective voice segments through Voice Activity Detection (VAD) to generate an approximate timestamp for each interaction sample.

[0049] Specifically, this invention first separates the audio from HoloLens video and saves it as a 16 kHz, 16-bit PCM mono waveform. Then, it uses the WebRTC VAD algorithm for frame-level speech detection, with VAD mode set to 2 and an analysis frame length of 30ms. To further suppress ambient white noise interference, this invention introduces a frame energy-based filtering strategy on top of VAD determination: first, the 70th percentile of the absolute amplitude of the entire audio segment is calculated as an energy reference; then, only speech frames that simultaneously meet the criteria of "VAD determination as speech" and "frame RMS higher than 0.6 times the reference value" are retained.

[0050] After obtaining consecutive valid voice frames, this invention merges them into voice segments, further retaining segments with a duration between 0.3s and 5.0s to remove short noise and abnormally long segments. To cover user voice commands as completely as possible, this invention extends the buffer by 0.4s before and after the start and end positions of each segment. Finally, the midpoint of each valid voice segment is taken as the approximate occurrence time of the interaction, and combined with the start time parsed from the HoloLens video file name, a millisecond-level ISO format timestamp is generated, denoted as approx_timestamps. This result is saved to features_timestamp_ The .npy file is used for subsequent multimodal alignment.

[0051] Through the above processing, each HoloLens recording file can obtain a set of approximate timestamps corresponding to the interaction behavior. These timestamps not only provide a unified benchmark for subsequent multimodal window capture, but also lay the foundation for cross-device time alignment and sample construction.

[0052] Cross-device time alignment and unified window construction: After obtaining the approx_timestamps, this invention further uses them as a unified time anchor point to align external fisheye camera video, audio, text, and IMU modalities. For external fisheye camera video, this invention first establishes a correspondence between fisheye video and HoloLens video based on the mapping table AVI_TO_MP4_MAP. This mapping table records the correspondence between each fisheye video file Video_i.avi and its corresponding HoloLens video file interaction_j.mp4, which can be represented as: ,in, Indicates the first A fisheye camera video file, This indicates the corresponding HoloLens video file. This represents the total number of videos. (Based on the mapping relationship.) This ensures that data from different devices belong to the same user interaction process during subsequent processing.

[0053] After completing the file mapping, the recording start time is parsed from the fisheye video filename and converted to UTC time by subtracting 8 hours. The video filenames saved by the fisheye camera contain local recording time information, for example: Video_20260306_152340690.avi. Here, 20260306_152340690 represents the year, month, day, hour, minute, second, and millisecond, i.e., the local time (UTC+8). Subsequently, the millisecond offset of the target interaction time relative to the fisheye video's start time is calculated; if this offset is within the effective video duration, the sample is considered usable for visual feature extraction. Specifically, using the fisheye video's start time as a unified reference point, let the... The approximate timestamp of each interaction sample is The starting UTC timestamp of the fisheye video is Then the time offset of this interaction moment relative to the start moment of the fisheye video can be expressed as: .

[0054] For the construction of samples for each modality, this invention uniformly adopts a time window centered at approx_timestamps, with 0.75s before and after it, for a total length of 1.5s. This window setting is used simultaneously for the extraction of gesture video, audio, text, and IMU data, as well as subsequent feature extraction, ensuring consistency across different modalities in the temporal dimension. Ultimately, each approximate timestamp corresponds to a multimodal sample, providing a unified input unit for subsequent feature extraction and model training.

[0055] To more intuitively demonstrate the overall processing flow of this invention, from raw multimodal data to aligned feature representation, Figure 3 The complete process of timestamp generation, cross-modal alignment, and feature extraction for each modality is presented.

[0056] Modal feature extraction: After completing multimodal time alignment, this invention further extracts features from four modalities: gesture, audio, text, and IMU, to construct input representations that can be uniformly processed by subsequent models. Since different modalities differ significantly in data format, sampling frequency, and semantic level, this invention employs different feature construction methods for each modality to preserve as much effective information as possible and enhance the complementarity between modalities.

[0057] Gesture Feature Extraction: The gesture modal input is a first-view video captured by an external fisheye camera. For each timestamp, this invention extracts a 0.75-second time window before and after it, and evenly samples 10 time points within this window. Image frames are then read from the corresponding video to form a 10-second gesture time sequence segment. This processing method can cover a relatively complete gesture action process while balancing the preservation of temporal information and the control of computational overhead.

[0058] Gesture region extraction relies on keypoint detection results from the MediaPipe Hands model. After constructing bounding boxes based on the detected hand keypoints, this invention extends the margins of the bounding boxes by an additional 40% to preserve local contextual information related to the gesture as much as possible, and uniformly adjusts the cropping results to 224×224 pixels. If a stable hand region is not detected in individual frames, the original image is directly and uniformly scaled to ensure the integrity of the temporal sequence.

[0059] Visual features are extracted by a pre-trained CLIP visual encoder, and the first token in the output sequence is used as the global semantic representation of a single frame. The CLIP model, pre-trained on large-scale image and text data, possesses strong generalization and semantic representation capabilities, and can adapt well to gesture image feature extraction tasks in complex scenes. After the above processing, each sample yields a gesture temporal feature matrix of length 10 and a single-frame feature dimension of 768. Finally, the gesture features are represented as follows: .

[0060] Audio Feature Extraction: The audio modal input is a monophonic speech waveform acquired and separated by HoloLens. Around each approx_timestamp, this invention extracts 0.75 s audio segments before and after it, resulting in a 1.5 s speech window. This window setting is consistent with other modalities to ensure the correspondence between multimodal samples in the time dimension.

[0061] Audio features are characterized using Mel-frequency cepstral coefficients (MFCCs). This invention utilizes librosa to extract 13-dimensional MFCC features from each audio window, and further calculates their first and second differences. These three differences are then concatenated along the feature dimensions to obtain a 39-dimensional acoustic feature representation. This processing method not only preserves the static spectral information of speech but also supplements the dynamic features of speech over time, helping to enhance the ability to characterize the content and pronunciation changes of speech commands.

[0062] Because audio inherently possesses continuous temporal characteristics, this invention preserves its original temporal resolution without forcibly resampling it to a fixed length. Therefore, the number of audio frames corresponding to different samples may vary slightly, but each frame corresponds to a 39-dimensional acoustic vector. Ultimately, the audio features are represented as follows: .in, This represents the number of MFCC feature frames in the current audio window, and its value depends on the window length, frame length, and frame shift. Different samples... There may be slight differences.

[0063] Text Feature Extraction: The text modality is generated from the speech content within the audio window through automatic speech recognition. For each 1.5-second audio segment, this invention uses Whisper's small model for Chinese speech recognition to obtain the text result corresponding to the current interaction moment. The text modality can directly reflect the user's operational goals at the linguistic semantic level, and therefore plays an important role in multimodal intent recognition.

[0064] To mitigate semantic ambiguity caused by homophones in Chinese, the identified Chinese text is further converted into pinyin strings with tones. Based on this, the present invention uses a pre-trained Sentence-BERT model all-MiniLM-L6-v2 to encode the pinyin text, obtaining a 384-dimensional semantic vector. Compared with shallow word frequency representation, this method can better preserve the contextual semantic information in the text.

[0065] Considering that semantic content within the same 1.5-second time window is generally relatively stable, this invention replicates a single 384-dimensional text vector 10 times along the time dimension to construct a temporal representation with the same length as the gesture modality, so as to facilitate subsequent multimodal fusion within a unified framework. Finally, the text modality features are represented as follows: .

[0066] IMU Feature Extraction: The IMU modal input is pose sequence data recorded by HoloLens. Before feature construction, the raw data is cleaned, sorted, and missing items are removed, and then reorganized according to timestamp order. The rotation quaternions in the pose information are converted into Euler angle representations, namely roll, pitch, and yaw, making pose changes easier to analyze and model later.

[0067] Building upon this, the present invention further calculates linear velocity and linear acceleration information by combining the position coordinate sequence to describe the dynamic changes during head movement. Around each approx_timestamp, IMU data windows of 0.75 s before and after are extracted and uniformly resampled into 10 time points using linear interpolation. Each time point contains 12 dimensions of features: 3D position, 3D Euler angles, 3D velocity, and 3D acceleration.

[0068] To reduce the impact of differences in the dimensions of different physical quantities on model training, this invention performs Z-score normalization on IMU features. After this step, the IMU modality retains both static information about head pose and dynamic information about motion changes, serving as an important supplement to speech and visual modalities. Finally, the IMU modality features are represented as follows: .

[0069] Scene Feature Extraction: The scene modality input consists of first-view video frames captured by an external fisheye camera, with timestamps strictly aligned with those of the gesture modality. Unlike the temporal sequence modeling of the gesture modality, the scene modality aims to provide global environmental context information at the current interaction moment; therefore, a single-frame image is used as input, rather than a temporal segment.

[0070] For each interaction sample, the single-frame image corresponding to the interaction time (approx_timestamps) is extracted from the fisheye video. Since the timelines of the fisheye camera and HoloLens are unified using UTC timestamps, the frame position in the fisheye video can be directly located based on the interaction time without the need for additional time window expansion.

[0071] The extracted fisheye images are first uniformly converted to RGB format and scaled to 224×224 pixels before being input into a pre-trained visual Transformer as a feature encoder. The first token in the model's output sequence is taken as the global semantic feature vector for that frame, with a dimension of 768. Finally, the scene modal features are represented as follows: .

[0072] Dataset Statistics and Analysis: After timestamp synchronization, VAD segmentation, invalid segment filtering, and multimodal feature alignment, this invention ultimately obtained 1923 valid five-modal samples. All samples have completed valid alignment of gesture, audio, text, IMU, and scene modalities and can be directly used for subsequent fusion modeling.

[0073] In terms of scene distribution, the dataset is relatively balanced overall, with 957 museum scene samples and 966 office scene samples. This relatively balanced scene distribution helps to reduce the bias of the model towards any single scene.

[0074] From the perspective of intent category distribution, there is a certain imbalance in the number of samples for the six types of interactive intents: menu (261 samples), select (433 samples), magnify (346 samples), narrow (258 samples), brush (286 samples), and cancel (339 samples). Among them, the select type has significantly more samples, while narrow and menu have relatively fewer.

[0075] Regarding data partitioning, this invention collected a total of 39 videos, of which 26 videos were used for training and validation, and 13 videos were used for testing. The dataset was partitioned at the video level to ensure that samples from the same video would not appear in both the training and testing sets simultaneously, thus avoiding data leakage. Under the current experimental setup, the training and validation portion contained 1180 samples, and the testing set contained 743 samples; after further partitioning, 944 training samples and 236 validation samples were obtained.

[0076] Table 2 Modal Feature Dimensions and Sample Statistics Overall, the dataset constructed in this invention has a relatively balanced distribution of scenarios, but there is a certain imbalance at the intent category level. This provides a foundation for the subsequent verification of model robustness and category balancing strategies. Table 2 further summarizes the input formats and feature dimensions of each modality.

[0077] Multimodal user interaction intent recognition method framework: Method Design Goals and Overall Framework: For the task of recognizing multi-channel user interaction intentions in augmented reality glasses, this invention aims to construct a multi-modal user interaction intention recognition model capable of uniformly processing heterogeneous modal inputs, preserving the dynamic temporal sequence of interactions, and fusing scene context information. Unlike traditional unimodal methods, the user intentions studied in this invention are simultaneously influenced by multiple factors such as semantic content, motion state, and environmental scene. Therefore, the model needs to possess three capabilities: first, to uniformly represent data from different sources and dimensions, such as gestures, audio, text, IMU, and scenes; second, to preserve the temporal structure of interactive behavior during the fusion process; and third, to explicitly utilize scene context to constrain intention discrimination.

[0078] Based on the above objectives, this invention employs Perceiver-IO as the multimodal fusion backbone. Perceiver-IO can organize inputs of varying lengths and heterogeneous sources into a unified token sequence, achieve information compression through cross-attention in the latent space, model global dependencies using self-attention in the latent space, and finally extract task-related information through learnable query vectors. Compared to directly performing global self-attention modeling on all input tokens, this structure offers better uniformity and more controllable computational complexity in multimodal scenarios. Accordingly, this invention first constructs a five-modal Perceiver-IO model under real-world scenario conditions, and further introduces anchor-based backbone fusion and robust modeling for missing modalities.

[0079] like Figure 4 As shown, the model input consists of five modalities. Gesture, audio, text, and IMU modalities retain temporal information, while the scene modality provides global contextual information. After being uniformly tokenized, the inputs from each modality are fed into the Perceiver-IO encoder for latent space modeling, ultimately outputting a scene-intent joint category. Within this framework, the model can uniformly fuse multimodal temporal features while preserving scene constraint information during the modeling process.

[0080] Real-world five-modal model design: To verify the basic applicability of Perceiver-IO to multimodal interaction intent recognition tasks under real-world input conditions, this invention first constructs a real-world five-modal model. This model takes five modalities—gesture, audio, text, IMU, and scene—as input, and scene-intent joint labels as the output target. The scene modality in the model is directly derived from real fisheye video frames, thus more realistically reflecting the impact of scene context on interaction intent recognition.

[0081] Structurally, this model follows the standard Perceiver-IO process, including multimodal tokenization, input encoding into the latent space, information exchange within the latent space, and query-based decoding output. Its core idea is to map five modalities, originally with different dimensions and semantics, into the same latent space, and then use the latent array to complete cross-modal fusion modeling. To clearly explain the model's construction, the following will introduce it from three aspects: input representation, multimodal fusion structure, and joint classification output method.

[0082] Input Representation and Multimodal Tokenization: To unify the input formats of different modalities, this invention first converts the five modalities into fixed-length token sequences. For the four modalities of IMU, gesture, audio, and text, a temporal representation of length 10 is used; for the scene modality, a single global token is used. Specifically, the gesture modality consists of 10 frames of visual features uniformly sampled within a fisheye video window; the IMU modality is obtained by interpolating and resampling the original sensor sequence within a time window to obtain a 10-step representation; the text modality is obtained by copying a single-window semantic vector along the time dimension to obtain a 10-step representation; and the audio modality is obtained by uniformly adjusting the original variable-length MFCC feature sequence to 10 steps through equal-interval sampling or zero-padding. The scene modality is obtained by extracting 768-dimensional global features from a single frame of a real fisheye scene corresponding to the sample timestamp through a visual encoder, thus corresponding to only one scene token.

[0083] Since different modalities have different original feature dimensions, this invention uses a modality-specific linear mapping to uniformly project them to the same latent space dimension. : in, The latent space features after projection of each modality are preserved in the temporal mode. Step, scene modality is One token; It is a learnable linear transformation with no bias.

[0084] After projection, time position encoding is added to the four time-series modes. This enables the model to perceive the sequential relationship between frames, while adding learnable modality encoding to each modality. This enables the model to distinguish information from different sensors; only modal encoding is added to the scene modalities. This is used to represent its semantic identity as a global context token.

[0085] For timing modes The t-th time step (t = 1, 2, …, 10): ,in, Let m be the original projected features of mode m at time step t. This is the final input token after encoding.

[0086] For scene modalities, since they only contain global features from a single frame and are not expanded along the time dimension, their encoded representation is as follows: ,in, This is the final input token for the scene modality.

[0087] After projection and encoding are completed, the five modalities are concatenated along the token dimension to form a unified input sequence: .

[0088] The model input consists of a total of 41 tokens, including 40 time-series tokens and 1 scene token.

[0089] Multimodal User Interaction Intent Recognition Model: After completing input tokenization, this invention uses Perceiver-IO to uniformly model the five modal inputs, and its overall structure is as follows: Figure 5 As shown, the model mainly consists of three stages: input encoding, latent space processing, and output decoding.

[0090] During the input encoding stage, the model first constructs a set of learnable latent vectors to form a latent array, which serves as an intermediate representation for information aggregation. In this invention, the model sets the number of latent vectors to 32 and the dimension of the latent vectors to 128; therefore, the size of the latent array can be expressed as: .

[0091] Subsequently, the input token sequence of length 41 is written into the latent array through cross-attention. This allows the latent vectors to read the most relevant multimodal features from the five-modal input, based on the user's interaction intent, through cross-attention. Here, the input sequence serves as the key and value, and the latent vector as the query. Attention weights are obtained by calculating the similarity between the query and the key, and then the values ​​are weighted and aggregated, thus completing the information compression mapping from the input space to the latent space. This process can be represented as: ,in, , , It is obtained from the latent vector and the input sequence through linear transformations, respectively. This is the scaling factor.

[0092] During the latent space processing phase, the model is stacked on the latent array. l Layered self-attention and feedforward networks are used to model the global dependencies between different modalities and time steps. Let the first layer be...l The potential representation of a layer is (common N =32 potential vectors, each vector dimension d= 128), the self-attention mechanism first obtains the query through a linear transformation. ,key Sum : in, For a learnable projection matrix, the first... l Layer input The mapping is defined as query, key, and value. The output of self-attention is: This operation enables each latent vector to interact with all other latent vectors, thereby capturing the dependencies between different latent variables. Since latent vectors do not explicitly correspond to a single modality or time step, this interaction is essentially a fusion of global information across modalities and time series. Subsequently, intermediate representations are obtained through residual connections and layer normalization (LayerNorm). Next, the feedforward network (FFN) independently performs a nonlinear transformation on each latent vector to further enhance its feature representation capabilities. An FFN typically consists of two linear layers and an activation function (Gaussian Error Linear Unit, GELU), and its output is: in Let be the weight matrix. This is the bias term. Finally, we obtain the... Layer output: Compared to repeatedly performing self-attention on the input token, Perceiver-IO concentrates the main computation in the latent space, thus effectively controlling computational complexity while maintaining modeling capabilities. l After the layer is updated, the model obtains the fused latent representation. , used to characterize the global multimodal semantics of the current sample.

[0093] In the output decoding stage, this invention introduces a learnable output query vector. Cross-attention from the decoder to the final latent representation It reads the information most relevant to the current classification task. Specifically, it reads... q As a query As keys and values, calculate cross-attention: ,in This is a learnable projection matrix. This computation enables the query vector to adaptively focus on the most valuable parts of the latent representation for intent classification, outputting an aggregate vector. This query-based reading method is one of the key features that distinguishes Perceiver-IO from ordinary encoders, enabling the model to "read latent representations based on the task." Finally, the globally fused representation is fed into the classification head to output the joint label prediction result.

[0094] Overall, the model implements a complete Perceiver-IO process from multimodal token input and latent space interaction to task-related decoding output. It retains the core idea of ​​"encoding-processing-decoding" in the original architecture and provides a standard reference for subsequent anchor-backbone improvement models.

[0095] Joint Classification Labels for Scene and Intent: This invention does not model the task as a separate 6-class intent classification problem. Instead, it adopts a joint modeling approach for scene and intent, defining the output of each sample as a joint label for scene and intent. Specifically, the scene categories include two classes: museum and office, and the intent categories include six classes: menu, select, magnify, narrow, brush, and cancel. Therefore, the final joint label space contains a total of 12 categories.

[0096] The reason for adopting a joint classification approach is that the interaction intent of augmented reality glasses is often not completely independent of the context. The same voice command or similar gesture may have different semantic interpretations or functional constraints in different scenarios. For example, in a museum setting, the intent to "zoom in" usually refers to observing the details of an artifact, while in an office setting it may refer to zooming in or out of document content. If only independent intent classification is performed, the model may easily ignore contextual information; however, by using joint labels for scene and intent, the model can learn the coupling relationship between scene features and intent features during training, thereby enhancing its overall discriminative ability.

[0097] In implementation, the model ultimately outputs a joint prediction result for 12 classes through a classification head. During the testing phase, based on the correspondence between the scene and intent components in the joint labels, the prediction results are further split into scene classification results and intent classification results, thereby generating separate scene classification reports and intent classification reports. In this way, the joint labels serve as the primary supervision target during model training, while the performance on both scene classification and intent classification dimensions can be observed separately during experimental analysis. Experimental results show that this joint modeling approach can better characterize the impact of scene context on intent recognition and provides a clear foundation for subsequent model improvement design.

[0098] A robust multimodal user interaction intent recognition method: Based on a real-world five-modal baseline model, this invention treats gesture, text, and scene as anchor modalities that provide a stable semantic skeleton, and imu and audio as supporting supplementary modalities, injecting auxiliary information through a residual approach. The goal of this design is to reduce the interference of weak or noisy modalities on the main discrimination path, while enhancing the model's ability to jointly model fine-grained action intent and scene context.

[0099] Gesture + Text + Scene Anchor Point Backbone Structure: Based on the above analysis, this invention first constructs an anchor backbone composed of three modalities: gesture, text, and scene. The core idea of ​​this design is that gesture provides action details and directional information, text provides stable semantic cues, and scene provides environmental context constraints. The combination of these three forms a relatively stable backbone representation. Unlike the baseline model where all 41 tokens are fed into the same perceiver encoding path, the improved model only feeds these three dominant modalities into the anchor backbone encoder. The gesture consists of 10 visual tokens, the text consists of 10 semantic tokens, and the scene consists of 1 global token. Therefore, the total length of the anchor input is 21 tokens.

[0100] In terms of specific implementation, there are three modes. Each is mapped to a unified latent space dimension via linear projection. get Subsequently, for gesture and text modalities, learnable temporal positional encodings are superimposed. With modal coding This forms a unified anchor token sequence. For scene modalities, since they are originally single-frame features, only modal encoding is added without temporal position encoding, resulting in a token sequence. Concatenate the above tokens in chronological order to obtain the anchor input sequence: .

[0101] Subsequently, The input information is fed into the Perceiver encoder. This encoder first compresses the input information into a learnable array of latent information through cross-attention. In China. Specifically, with As a query, with As keys and values, calculate cross-attention: .

[0102] in This is a learnable projection matrix. This operation extracts key information from the input sequence and writes it into the latent array, achieving a compressed mapping from the high-dimensional input space to the low-dimensional latent space. Subsequently, multiple layers of self-attention and feedforward networks are stacked within the latent space to perform deep interaction and feature refinement on the latent representation. l The layer is calculated as follows: ; go through l After stacking the layers, the final latent representation is obtained. This completes the deep fusion and global modeling of cross-modal information. After processing by the encoder, the model uses a learnable output query vector. By using decoder cross-attention to read task-related information from the latent representation, the anchor backbone representation is obtained. : .

[0103] This representation can be considered the core semantic skeleton of the current sample, and all subsequent supplementary information from auxiliary modalities will revolve around this backbone. The relationship between the backbone and auxiliary structures of the improved model is as follows: Figure 6 As shown.

[0104] Compared to the baseline model, the direct change brought about by this design is that the model first explicitly establishes a stable semantic skeleton composed of the dominant modality, rather than having all modalities compete for attention resources at the same level. This preserves the key roles of gesture, text, and scene in interactive intent recognition, while also providing a clear reference benchmark for the residual injection of subsequent auxiliary modalities.

[0105] IMU + Audio residual supplementation branch: After establishing the anchor backbone, this invention further introduces two support modalities, IMU and audio, as residual supplementary branches. The reason for this design is that although the overall contribution of these two modalities is weaker than gesture, text, and scene, they still provide additional motion variations and acoustic cues, and in some samples, they have supplementary value for the discrimination boundary. Completely discarding these two modalities would reduce the model's utilization of user dynamic states and speech acoustic information, while still inputting them with equal weight to the main modality easily introduces noise. Therefore, this invention adopts a residual supplementation method, which, while retaining their effective information, reduces the direct perturbation to the backbone representation.

[0106] Specifically, the IMU and audio modal do not directly enter the anchor-point backbone encoder, but are represented by anchor points through cross-attention readout modules. As a query, a sequence of tokens for each modality. As keys and values, extract supplementary features that are most relevant to the current main semantics: in These are supplementary features extracted from the IMU and audio modality that are most relevant to the anchor backbone semantics, used for subsequent residual injection. The specific calculation process across the attention readout module is as follows: in It is a learnable projection matrix.

[0107] These auxiliary features do not independently perform the classification task, but are superimposed on the anchor backbone representation as residuals to form the final fused representation: ,in For the weighted residuals of the two supporting modes, , For dynamic gating values, , Hyperparameters for controlling the maximum injection intensity.

[0108] From a structural perspective, this design essentially embodies the fusion principle of "the backbone makes decisions first, and the auxiliary modalities make corrections later." It avoids excessive competition between the supporting modalities and the backbone modalities in the early stages and is more in line with the modal role distribution patterns observed in experiments: the anchor backbone provides a stable semantic skeleton, and the residual supplementation only fine-tunes local details, thereby improving information utilization while suppressing noise interference.

[0109] Analysis of the problem of missing anchor modes: The core of the anchor-residual improved model lies in constructing a stable backbone representation using three modalities: gesture, text, and scene. The gesture provides fine-grained action information, the text provides the semantic framework, and the scene provides environmental context constraints. This structure achieves high performance under complete input conditions, but it also introduces a new dependency—the model becomes more sensitive to the completeness of the anchor modality. When the gesture or text is missing, the semantic completeness of the backbone representation decreases significantly, causing the fused representation to fail to accurately reflect the relationship between the current interaction action and the semantic intent.

[0110] The absence of gestures directly weakens the model's ability to distinguish fine-grained action categories such as drawing, zooming in, and zooming out. The absence of text weakens the stable support of semantic commands for intent recognition, making the model more reliant on shallow associations between actions or scenes. In contrast, scene, as a relatively stable global context modality, remains largely usable in real-world data acquisition environments. Therefore, the robustness extension model primarily models for the absence of gesture and text anchor modalities. The key to robustness is not that "all modalities will randomly fail," but rather how to maintain the availability and consistency of the backbone representation when key anchor modalities in the backbone are incomplete.

[0111] From a methodological design perspective, the model should not only learn the "optimal fusion method" under the condition of complete anchors, but also must have explicitly seen scenarios with incomplete anchors during the training phase and learn how to construct a reasonable backbone representation even when information is missing. Therefore, this invention introduces an explicit missing information modeling mechanism within the anchor backbone, rather than simply performing random discarding on supporting modalities. The following sections will elaborate on random anchor discarding, learnable missing tokens, usability embedding, and selective element-wise gating for auxiliary modalities (these four mechanisms constitute the explicit missing information modeling mechanism, which is a robust model based on modality missing information).

[0112] Anchor point random discarding and missing token mechanism: To enable the model to adapt to incomplete input of key anchor modalities during the training phase, this invention first introduces a random anchor discarding mechanism in the anchor backbone. Unlike conventional discarding, random anchor discarding does not randomly deactivate on a feature dimension basis, but rather on a modal basis, randomly retaining or setting the gesture and text anchor branches to zero. Specifically, in each training iteration, the probability of discarding the gesture modality and the text modality is independently determined with probabilities of 0.10 and 0.08, respectively. If a modality is determined to be discarded, all token values ​​at the corresponding time step of that modality are set to zero; if it is determined to be retained, the original tokens remain unchanged. This operation actively simulates scenarios where "gestures are unavailable" or "text is unavailable" during training, forcing the model to construct reasonable intermediate representations even when some key modalities are missing.

[0113] However, simply setting all tokens for a particular modality to zero is insufficient for the model to distinguish between two different situations: "the modality itself has a very small value" and "the modality is actually missing". Therefore, this invention further introduces a missing token mechanism, setting learnable missing tokens for both the gesture and the text. and When an anchor modality is detected as unavailable, or is actively discarded by the anchor random discarding mechanism during training, the model no longer inputs the original modality token, but instead replaces the entire modality sequence with the corresponding missing token. Let the original token representations of gesture and text be respectively... and If the input is missing, then the replacement is: .

[0114] The core of this design lies in the fact that the model will not misclassify a missing modality as a "normal modality with very small feature values," but rather will explicitly perceive that "a key input path is missing here." When the Perceiver encoder processes the anchor token sequence, the missing token participates in the attention calculation as an explicit state, enabling the model to construct a reasonable semantic representation even with incomplete information. The extended structure of the robust model is as follows: Figure 7 As shown, the missing modeling of gesture and text is uniformly incorporated into the anchor point backbone generation process.

[0115] Usability Embedding Mechanism Design: Building upon the missing token, this invention further designs a learnable embedding mechanism for explicitly encoding the "available" and "missing" states of anchor modalities. The rationale behind this mechanism is that even if two inputs have the same dimension after uniform projection at the numerical level, the model still needs an additional identifier to distinguish between "truly available gesture / text tokens" and "gesture / text tokens replaced by missing tokens."

[0116] For the two anchor modalities, gesture and text, two sets of learnable embeddings are set: one set represents the state embedding when the modality is available. Another set represents the state embedding when this mode is missing. When a modality is available, the model input is "original projection token + available state embedding"; when a modality is missing, the input is "missing token + missing state embedding". Therefore, for a given anchor modality... ,set up This is the projection feature matrix when the modality is available. If the missing token matrix is ​​learnable, then its final input form can be represented as: .

[0117] The role of learnable embedding mechanisms is to provide the encoder with explicit state cues, enabling the model to know not only "the numerical content of the current token" but also "the available state of the current modality." Without this mechanism, the model can only passively infer the modal state from the numerical distribution. With the addition of learnable embedding mechanisms, the model can more stably distinguish between normal and missing samples during the latent space modeling stage, thereby improving its adaptability to incomplete inputs.

[0118] Structurally, the random dropping of anchors provides a missing state simulation during the training phase, the missing token provides a placeholder representation for missing inputs, and the learnable embedding mechanism supplements the explicit modeling of the "missing state" itself. Together, these three constitute the missing state modeling framework of the anchor backbone in the robust expansion model.

[0119] Selective element-wise gating for IMU + Audio: Besides the lack of anchor modalities, the IMU and audio support modalities are also susceptible to jitter, sensor fluctuations, environmental noise, and changes in speech quality under actual acquisition conditions. If these support modalities are indiscriminately injected into the main representation when their quality is poor, additional noise can be introduced, weakening the discriminative power of the main representation. Therefore, this invention further introduces element-wise gating to the IMU and audio support branches in the robust extension model to achieve fine-grained front-end feature cleaning.

[0120] This approach doesn't apply gating uniformly to all modalities; instead, it uses the mechanism only for the IMU and audio support modalities. Applying the same front-end cleaning gating to core modalities like gesture, text, and scene might actually disrupt the expression of key action and semantic information. IMU and audio, as support modalities, are inherently more prone to noise and are better suited for lightweight cleaning before residual readout. Let the support modal input be... Its gated output can be expressed as: , Among them, the gate matrix The calculation method is as follows: In the formula For the Sigmoid function, For learnable weight matrix, This is a learnable bias (initialized to a negative value to suppress the initial bias of the gating). This is a learnable gating bias term for this modality. Through this mechanism, the model can adaptively suppress unreliable components along the feature dimension, retaining more discriminative support features. The cleaned IMU and audio token then enter the cross-attention readout module of the support branch, supplementing the anchor backbone representation as residuals.

[0121] This design, which "gates only the supporting modalities and not the backbone modalities," balances the stability of the backbone representation with the availability of supporting information. On the one hand, the dominant semantics of gesture, text, and scene are not disturbed by additional gating; on the other hand, the IMU and audio branches, which are more prone to noise, undergo feature-level cleaning before entering the fusion process, thereby improving the overall stability under modal noise and local failure conditions.

[0122] Model training objectives and optimization strategies: To ensure that the baseline model, anchor-residual improved model, and robust extension model are trainable and comparable within a unified framework, this invention adopts a unified design for their training objectives and optimization strategies. Overall, the main objective is scene-intention joint classification, and scene-assisted supervision, intention-assisted supervision, and gesture refinement supervision are introduced based on differences in model structure, enabling the model to learn stable representations at the joint label, scene label, and intention label levels. Simultaneously, a validation set-based model selection and early stopping mechanism is employed to achieve a good balance between joint classification performance and sub-task performance.

[0123] Training objective definition: The core task of this invention is to perform joint scene-intent recognition on user interaction samples. Let the scene label set be... The intent tag set is The joint tag space can then be represented as: ,in Represents a single scene label. Represents a single intent label.

[0124] In this invention, the scenarios include two categories: museums and offices; the intentions include six categories: menus, selections, zoom in, zoom out, brush strokes, and cancel; and the joint labels total 12 categories. The main objective of model training is to learn to classify these 12 joint labels.

[0125] In the baseline model, joint label prediction is the sole core supervision signal, and the model models the coupling relationship between scene and intent by directly learning the joint category. For the anchor-residual improved model, since the scene branch, intent branch, and gesture refinement branch are explicitly distinguished internally, additional auxiliary supervision is needed in addition to joint label supervision to stabilize the representations of each sub-branch, enabling them to assume more explicit semantic roles. The robust extension model maintains the same training objective but enhances its generalization ability in scenarios with missing modalities by using a missing modality modeling mechanism to allow the model to receive supervision from both complete and incomplete inputs.

[0126] Multi-task loss function design: To accommodate the structural characteristics of different models, this invention adopts a "consistent main task, progressive auxiliary task" approach in the loss function design. For the baseline model, only the joint classification loss is used: ,in, For joint classification logits, For genuine joint tags, Cross-entropy loss is used. Considering that the number of samples in different classes may differ, this invention introduces class weights during training to mitigate the bias caused by imbalanced distribution.

[0127] For the anchor-residual improved model, intention-assisted loss, basic intention loss, scene-assisted loss, and gesture-assisted intention loss are further introduced on the basis of joint supervision. Let the intention logits, basic intention logits, scene logits, and gesture-assisted intention logits output by the model be respectively... , , , The corresponding losses can be written as: The total loss is: ,in represents the weight coefficients of each auxiliary loss. This design ensures that the joint loss guarantees the direction of the main task, the scene loss constrains the semantic branches of the environment, the intent loss stabilizes the main graph path, and the gesture-assisted loss specifically enhances the ability to recognize fine-grained action categories, effectively alleviating the shortcut learning problem that may occur when only joint label supervision is used.

[0128] The robustness extension model maintains the same multi-task loss form, but due to the introduction of AnchorDropout, Missing Token, and Availability Embedding during training, the loss function actually imposes joint constraints on the model under more complex input conditions, enabling the model to learn the recognition ability when the complete input is complete and when key modal parts are missing.

[0129] Optimization strategies and model selection principles: In terms of optimization strategy, this invention uniformly adopts a parameter update method based on adaptive gradients to adapt to the problem of inconsistent parameter distributions in different branches of multimodal deep models. Compared with the traditional fixed learning rate strategy, this method has better stability in multi-task joint training and can smoothly complete the collaborative optimization between different loss terms. To reduce overfitting, strategies such as weight decay, label smoothing, and gradient clipping are combined during training to ensure that the model can still maintain stable convergence in the later stages of training.

[0130] Regarding model selection, this invention does not solely rely on minimizing training set loss to determine the final model, but rather adopts a model selection principle based on validation set performance. For the baseline model, the joint classification performance on the validation set is the primary criterion for model selection; for the improved model and the robust extension model, in addition to joint classification performance, the two dimensions of scene classification and intent classification are further considered, ensuring that the final retained model not only performs well on the joint scene-intent task but also has a good balance between the scene and intent subtasks. If the validation set performance no longer improves within several rounds, training is stopped early, and the model parameters with the best validation set performance are retained.

[0131] This model selection method considers the hierarchical structure of the task itself. Using joint accuracy alone may result in models with high scene accuracy but weak intent recognition; conversely, focusing solely on intent accuracy may weaken the effect of scene constraints. Therefore, this invention adopts the principle of "joint goal priority, sub-task balance" in model selection.

[0132] Example 3 To verify the basic effectiveness of the Perceiver-IO baseline model in the multimodal interaction intent recognition task for augmented reality glasses, this invention first evaluates its joint classification, intent classification, and scene classification performance on a test set. The model takes five modalities—gesture, audio, text, IMU, and real-world scene—as input and uses the scene_intent joint label as the prediction target. This reflects the basic performance level achievable by the multimodal fusion model without introducing modality hierarchy modeling and missing robustness mechanisms.

[0133] Overall performance of the baseline model: Experimental results show that the baseline model can perform the scene-intent joint recognition task well on the test set, and the specific results are shown in Table 3.

[0134] Table 3 Model Test Results Overall, the model achieved high accuracy across all three evaluation dimensions, demonstrating that the five-modal fusion framework based on real-world scene input can effectively model semantic, action, motion, and environmental context information in user interactions.

[0135] From the perspective of the indicator relationships, scene classification has the highest accuracy, followed by intent classification, while joint classification has a slightly lower accuracy than the other two. This phenomenon is consistent with the characteristics of the task: scene classification only needs to determine whether the sample belongs to a museum or an office, and the two types of scenes are significantly different in the overall visual context, making them relatively easy to identify; although intent classification has more categories, text and gesture modalities provide strong support, so its performance is still relatively high; joint classification requires correctly identifying both the scene and intent dimensions simultaneously, which is the most difficult, and a slight decrease in accuracy is a reasonable result.

[0136] Overall, the real-world five-modal baseline model validates the feasibility of the task setting in this invention. The introduction of real-world frames as scene tokens allows the model to directly utilize the environmental context in the deployment environment, rather than relying on proxy scenes or manually constructed alternative information. Simultaneously, gestures, text, audio, and IMU modalities collectively provide multi-source complementary information for intent recognition. These results demonstrate that the Perceiver-IO structure possesses good fundamental applicability in real-world augmented reality glasses multimodal recognition tasks.

[0137] Training convergence analysis of the baseline model: In addition to test results, the loss and accuracy curves during training also reflect the optimization characteristics of the baseline model. For example... Figure 8As shown, the model loss decreased rapidly in the early stages of training, indicating that the Perceiver-IO structure can quickly complete the joint modeling of multimodal inputs. The training loss dropped rapidly from 1.2457 in the first round to around 0.0016 in the later stages, while the training accuracy increased from 79.56% to nearly 100%, indicating that the model has a strong fitting ability on the training set.

[0138] The validation set curves show that the model achieved high performance in an early stage. The validation loss decreased to a minimum of 0.0723 in the second epoch, corresponding to a joint validation accuracy of 99.58%. Afterward, although the training loss continued to decrease and the training accuracy remained close to 100%, the validation loss began to fluctuate slightly, while the validation accuracy remained largely within the range of [98.73%, 99.15%] without significant improvement. This indicates that the model had essentially completed learning the main discriminative boundaries in the early stages of training, and subsequent training was more about further fitting the training set, showing a certain tendency towards overfitting.

[0139] The optimal checkpoint was found in the second epoch, corresponding to the highest joint validation accuracy, indicating that the validation set selection and early stopping mechanism employed in this invention are reasonable and effective. Although the validation loss decreased further in subsequent epochs, the validation accuracy did not improve significantly. This does not imply insufficient model training, but rather that the model had already reached the optimal generalization state on the validation set under the current data scale and feature representation conditions. If we simply rely on increasing the number of training epochs for further iterations, although the performance on the training set will improve further, the generalization gain on the validation and test sets will not be significant. Therefore, the loss curve of the baseline model verifies the trainability of the multimodal baseline structure and also indicates that in the current task, subsequent performance improvements should rely more on improvements to the fusion structure itself, rather than simply increasing the number of training epochs.

[0140] Error mode analysis of the baseline model: Although the baseline model performs well overall, the classification report and confusion matrix results show that the model still has relatively concentrated errors in a few fine-grained categories. For example... Figure 9As shown, errors in joint classification are mainly concentrated between categories with similar action boundaries. Among them, `museum_brush` and `museum_narrow` are two relatively weak classes, with the former having a recall of 71.43% and the latter 84.78%. The most typical error is that `museum_brush` is misclassified as `museum_menu`, occurring in 14 samples; in addition, `museum_narrow` is also frequently misclassified as `office_narrow`, occurring in 7 samples. This indicates that while the baseline model can distinguish most joint categories relatively well, it is still susceptible to interference from scene background information or intermodal redundancy in some fine-grained action categories.

[0141] From the intent classification results, the model's overall intent recognition accuracy reached 97.17%, indicating that it has a strong ability to discriminate most interactive commands. However, the recall rate for the "brush" category was only 86.09%, significantly lower than that for categories such as "select," "magnify," and "narrow," while the precision for the "menu" category was also relatively low. This is consistent with the results in the joint confusion matrix, indicating a significant misclassification between "brush" and "menu." Under the unified and equal-weighted fusion framework, although the model can utilize multimodal information to complete overall discrimination, the discrimination boundary for categories with similar local action patterns remains unclear.

[0142] In contrast, the scene classification results are relatively stable, with F1-scores of approximately 0.985 for both the office and museum scenes, indicating that scene modalities can provide reliable environmental context information for the model. Therefore, the main bottleneck of the current baseline model is not the difficulty in identifying the scenes themselves, but rather the inadequacy in distinguishing some fine-grained intent categories, even when the scenes are generally separable. The baseline model can answer questions like "What scene is the user in?" and "What type of interaction did the user roughly perform?" quite well, but there is still room for improvement in recognizing specific actions with similar boundaries.

[0143] In summary, Figure 9 The confusion matrix shown indicates that the baseline model has validated the basic effectiveness of Perceiver-IO in this task, but the unified equal-weighted fusion has not fully reflected the primary and secondary relationships of different modalities in the joint recognition task. Especially in fine-grained categories such as brush, menu, and narrow, the model still suffers from confusion caused by modal redundancy and unclear dominant information paths. Therefore, the next section will further investigate the marginal role of each modality in the current task through modal contribution analysis and ablation experiments, providing a basis for subsequent model improvement design.

[0144] Performance verification of the anchor-residual improved model: Based on the analysis of modal contribution and ablation experiments, this invention no longer adopts the five-modal unified equal-weight fusion approach. Instead, it constructs an anchor-residual improved model with Gesture, Text, and Scene as anchor backbones and IMU and Audio as residual aids. The core idea of ​​this model is that a stable core semantic representation is jointly established by the three dominant modalities, and then dynamic information is supplemented by the two supporting modalities in a residual manner, thereby reducing the interference of weak and noisy modalities on the main discrimination path. To verify the effectiveness of this structural design, this section analyzes it from three aspects: overall performance, training convergence characteristics, and error patterns.

[0145] Overall performance of the anchor-residual improvement model: Experimental results show that the anchor-residual improved model achieves superior recognition performance compared to the baseline model under full modal input conditions. As shown in Table 4, the baseline model achieves a joint classification accuracy of 95.69%, an intent classification accuracy of 97.17%, and a scene classification accuracy of 98.52%; while the anchor-residual improved model achieves a joint classification accuracy of 98.79%, an intent classification accuracy of 98.79%, and a scene classification accuracy of 100.00%. Compared to the baseline, the joint classification accuracy improved by 3.1 percentage points, the intent classification accuracy improved by 1.62 percentage points, and the scene classification accuracy improved by 1.48 percentage points. This result indicates that by constructing the anchor backbone based on the modal hierarchy, the model can more effectively utilize core semantic information and significantly improve the joint scene-intent recognition performance.

[0146] Table 4 Performance Comparison of Baseline Model and Anchor-Residual Improved Model The improved model not only enhanced the overall joint classification accuracy but also improved the performance of intent classification and scene classification. This indicates that the anchor-residual structure does not merely enhance a single subtask but improves the overall quality of multimodal fusion. In particular, the scene classification accuracy reached 100.00%, demonstrating a more stable and clear representation of the scene context within the backbone composed of Gesture, Text, and Scene. Simultaneously, the improved intent classification accuracy also indicates that incorporating Gesture into the backbone strengthens the modeling capability for fine-grained action categories.

[0147] Robustness experiment for missing modes: This invention has verified that the improved model, anchored by Gesture, Text, and Scene, and supplemented by IMU and Audio as residuals, achieves optimal recognition performance under full modality conditions. However, in real-world augmented reality glasses interaction scenarios, the stable availability of all modalities cannot always be guaranteed. For example, gestures may be missing due to occlusion or hands going out of bounds, text may be incomplete due to speech recognition failure, and audio and IMU may be affected by environmental noise or sensor jitter. Therefore, verifying performance only under full modality conditions is insufficient to demonstrate the model's stability in real-world applications. This invention further constructs a robust extended model for modality loss and evaluates its generalization ability under incomplete input conditions through partial loss tests.

[0148] Missing modality robustness experimental setup: The robustness experiments of this invention do not involve redefining the task. Instead, based on the anchor-residual master model, an explicit modeling mechanism for missing key anchor modalities is introduced, and partial modality-missing scenarios are constructed for evaluation during the testing phase. During the training phase, anchors in the Gesture and Text sections of the anchor backbone are randomly dropped to simulate modality-missing sections with a low probability. Simultaneously, missing tokens and availability embeddings are combined to enable the model to explicitly distinguish between "modality available" and "modality missing" states. For the two supported modalities, IMU and Audio, selective element-wise gating is further introduced to suppress unreliable feature components before they enter the residual readout module.

[0149] During the testing phase, this invention retains two evaluation conditions. The first is a clean test, where all modalities are fully available, used to measure the robust model's performance retention under standard input conditions. The second is a partial-missing test, where some modalities are randomly simulated to be missing on the test set with a set probability, used to measure the model's stability under incomplete input conditions. Since IMU and Audio, as supporting modalities, are more susceptible to sensor fluctuations and environmental noise in real-world use, a relatively high missing probability is set. Gesture and Text, as anchor backbone modalities, are more crucial for backbone representation construction, but may become temporarily unavailable during interaction due to hand occlusion, out-of-frame errors, or speech recognition failures; therefore, a lower but non-zero missing probability is set. The Scene modal maintains stable input in the current experimental setup, and its missing probability is set to 0. This invention uses IMU=0.35, Audio=0.20, Gesture=0.10, and Text=0.08 as the default missing modality setting. Actual sampling results show that 52.49% of the samples in the test set are missing at least one modality, 15.61% are missing one of the anchor modalities, and only 0.27% of the samples are missing both Gesture and Text. This setting can effectively simulate the application scenario of "some modalities being unstable, but not completely ineffective" in real-world interactions. To further observe the changes in model performance under different missing modalities, this invention adds two control experiments: weak missing and strong missing. Weak missing corresponds to IMU=0.20, Audio=0.10, Gesture=0.05, and Text=0.04, while strong missing corresponds to IMU=0.50, Audio=0.30, Gesture=0.15, and Text=0.12. This hierarchical setting allows for a more systematic comparison of the stability differences between the complete modality master model and the robust model under different missing modalities.

[0150] Overall performance and convergence analysis of the robust model: Experimental results show that the robust model maintains performance similar to the main improved model under full modality conditions, while exhibiting better stability under partial modality missing conditions. As shown in Table 5, under full modality conditions, the robust model achieves an intent classification accuracy of 98.65% and a scene classification accuracy of 100.00%. Compared to the main model, the intent classification accuracy drops by only 0.14 percentage points, indicating that the model suffers almost no significant performance loss on the standard test set after introducing missing modeling and selective gating.

[0151] Table 5 Performance Comparison of Master Model and Robust Model Under partial modality loss conditions, the anchor-residual master model exhibited a significant performance decline, with joint accuracy dropping to 88.83%, intent accuracy to 92.46%, and scene accuracy to 96.23%. In contrast, the robust model maintained a joint accuracy of 94.21%, an intent classification accuracy of 94.21%, and a scene classification accuracy of 100.00%. Compared to the master model, the robust model improved joint classification by 5.38% and intent classification by 1.75%, while scene classification remained at a perfect score. When more than half of the samples had at least one missing modality, the unrobust master model was significantly affected. While the robust model showed a decrease in accuracy compared to the complete modality model, it effectively mitigated the performance degradation caused by modality loss through four improvement mechanisms, demonstrating stronger fault tolerance.

[0152] To further analyze the performance changes under different missing strengths, this invention compared the anchor-model and the robust model under three sets of conditions: weak missing, medium missing, and strong missing. The results are shown in Table 6.

[0153] Table 6. Performance Comparison of Main Model and Robust Model under Different Missing Levels As shown in Table 6, the performance of both models declines as the missing data intensity increases, but at different rates. The joint classification accuracy of the complete modality master model drops from 94.48% to 82.77%, a decrease of 11.71%; while the robust model drops from 96.23% to 90.71%, a decrease of 5.52%, which is significantly more gradual. This difference is even more pronounced in scene classification: the complete modality master model drops from 98.25% to 92.73%, while the robust model remains close to perfect. This indicates that as the missing data intensity increases, the advantage of the robust model over the master model further expands, and its value is particularly evident under conditions of strong missing data.

[0154] like Figure 10 During the training process, the optimal checkpoint for the robust model appeared at the 11th epoch, corresponding to a joint classification accuracy of 100.00% on the validation set. Compared to the full modality master model, the robust model converged slightly slower. During training, this model not only had to learn the joint classification task itself but also simultaneously adapt to random missing anchor modalities, support modal noise cleaning, and multi-task auxiliary supervision, making optimization more challenging. The validation curve trend shows that the overall training process of the robust model was relatively stable. The current missing modeling strategy did not compromise the trainability of the original anchor-residual structure but rather enhanced the model's adaptability to incomplete inputs in a relatively gentle way.

[0155] Error Mode and Gating Analysis under Missing Mode Conditions: According to the classification report, the robust model's prediction results under full modality conditions retain only a small number of errors in a few fine-grained categories, and are generally close to the full modality master model. Robust training did not significantly compromise the model's discrimination boundary under normal input conditions.

[0156] Under conditions of partial modality loss, the main performance degradation of the model remains concentrated in fine-grained action categories and a few categories with similar boundaries. For example... Figure 11 As shown, under this condition, the confusion matrix remains largely concentrated on the diagonal, but the number of misclassifications for some categories increases. For example, the recall rate for `museum_cancel` drops to 85.71%, `museum_narrow` to 89.13%, and `office_narrow` to 89.71%. When anchor or support modalities are missing, the model is most vulnerable to categories with finer action boundaries and stronger dependencies on local gestures and semantics. Compared to directly using the full modality master model under missing conditions, the current robust model still maintains better overall stability.

[0157] From the gating statistics, the selective element-wise gating in the robust model mainly operates on the IMU and Audio support modalities. Its average feature gating strength remains stable at around 0.44 in both the complete and partially missing modalities, while the gating strength for the three anchor modalities (Gesture, Text, and Scene) remains at 1. This indicates that the current gating mechanism does not interfere with the core modalities, but only performs lightweight feature cleaning on the more easily noisy support modalities. Meanwhile, in the partially missing modalities, anchor availability statistics show that the average availability rate for Gesture is 0.9273, and for Text it is 0.9139, consistent with the actual missing proportions in the test set. This indicates that the model can correctly perceive the availability status of key anchor modalities during the inference phase and adjust the fusion method of the core representation and residual supplementation branches accordingly.

[0158] Overall, the impact of missing modalities is mainly concentrated on fine-grained action categories. Nevertheless, the robust model still achieved high joint recognition accuracy under missing conditions, and scene discrimination remained stable. This indicates that when some inputs are temporarily unavailable, the model can rely on the remaining modalities to achieve relatively reliable intent recognition.

[0159] Key robust mechanism ablation experiment analysis: To further determine the actual function of each component mechanism in the robust model, this invention removes anchor core random discarding, missing tokens, availability embedding, and selective element-by-element gating from the complete robust model, and compares them under the same partial modality missing test conditions. The experimental results are shown in Table 7.

[0160] Table 7. Experimental Results of Key Robust Mechanism Ablation As shown in Table 7, the models do not differ significantly under the condition of complete modality input, indicating that these robust designs do not significantly impair the main model's recognition ability under normal input. The real difference lies in the partial modality missing test. The anchor-residual main model maintains the highest joint classification accuracy under complete modality, but when directly used on the missing test set, the joint classification accuracy drops to 88.83%. In contrast, the complete robust model shows only a small performance loss under complete modality, while still maintaining a joint classification accuracy of 94.21% under partial modality missing conditions, demonstrating the effectiveness of the overall missing modeling approach.

[0161] Removing the anchor backbone random drop mechanism resulted in the most significant performance drop, with the joint classification accuracy falling to 88.96%, almost returning to the level when the main model directly faced the missing test set. Scene classification accuracy also dropped from 100.00% to 96.77%. This result demonstrates that the robust model can adapt to incomplete anchor modalities, the key being that such inputs have been explicitly encountered during the training phase. Without this step, although the model still performs well under complete modal conditions, the backbone representation struggles to maintain stability under missing conditions.

[0162] Removing the missing token and availability embedding both reduced model performance, but the decrease was relatively mild. The joint classification accuracy of the former was 93.81%, and that of the latter was 93.67%. The missing token provides a placeholder for the missing information, and the availability embedding further encodes this state into the input representation. Removing either one alone will not cause the model to fail immediately, but it will indeed weaken the stability under missing conditions.

[0163] The effect of selective element-wise gating is also quite significant; removing this gating mechanism reduced the joint classification accuracy to 92.06%. When some modalities are missing, it's not advisable to simply add as many IMU and Audio support modalities as possible. Instead, they need to undergo a certain degree of feature cleaning before being added to the backbone representation as residuals. Without this step, noise in the support modalities is more easily propagated into subsequent decisions.

[0164] These four ablation experiments demonstrate that the performance improvement of the robust model is not due to a single technique, but rather supported by multiple mechanisms. Among them, random anchor dropping contributes the most to the overall robustness, followed by selective element-wise gating, while missing tokens and availability embedding further refine the representation under missing conditions.

[0165] Example 4 AR Augmented Reality Glasses User Interaction Intent Recognition Implementation Case.

[0166] Experimental setup: To verify the effectiveness of the proposed model in real-world augmented reality glasses multimodal interaction intent recognition tasks, this invention conducts experiments based on a baseline model and an improved anchor-residual model. The experiments adhere to a unified data partitioning protocol, evaluation metric system, and training configuration to ensure comparability between different models. Unless otherwise specified, all experiments in this invention are conducted using the same dataset, the same label definition, and the same training and testing partitioning conditions.

[0167] Dataset partitioning and evaluation metrics: The dataset used in the experiment consists of five modalities: gesture, audio, text, IMU, and scene. After time synchronization, invalid sample filtering, and multimodal alignment, a total of 1923 valid samples were obtained. The dataset includes two real-world scene types: museum and office, and the intent categories include six types: menu, selection, zoom in, zoom out, pen, and return.

[0168] Because samples within the same video are temporally continuous and highly context-dependent, random sample-level partitioning could easily lead to potential data leakage between the training and test sets. Therefore, this invention employs a video-level partitioning approach to construct the training and test sets, ensuring that all samples from the same video appear in only one set. The dataset contains 39 videos, with 26 used for training and validation, and 13 for testing. According to this partitioning, the training videos correspond to 1180 samples, and the test videos to 743 samples. Subsequently, a validation set is further partitioned within the samples corresponding to the training videos for model selection and early stopping control, ultimately resulting in an experimental setup with 944 samples in the training set, 236 samples in the validation set, and 743 samples in the test set. This partitioning method ensures the independence of the test set evaluation and provides a reliable basis for hyperparameter tuning and optimal epoch selection during model training.

[0169] Regarding evaluation metrics, this invention focuses on scene-intent joint classification as the primary task, and uses joint classification accuracy as the core evaluation metric. Assuming the test set contains N samples, the number of samples with correctly predicted joint labels is... The joint classification accuracy is defined as: ; The joint label essentially contains both scene information and intent information. This invention further separates the joint prediction result into independent scene prediction and intent prediction, and calculates the intent classification accuracy and scene classification accuracy separately. , ,in, and These represent the number of samples that were correctly predicted for the intent label and the scene label, respectively.

[0170] In addition to the accuracy metrics mentioned above, this invention also incorporates precision, recall, and F1-score from the classification report to conduct fine-grained analysis of the performance of each category, and observes the error patterns of the model in easily confused categories through a confusion matrix. For the baseline and improved models, the focus is on the improvement of joint classification performance and the recognition of fine-grained action categories such as pen, zoom in, and zoom out; for robustness experiments, additional attention is paid to the changing trends of joint accuracy, intent accuracy, and scene accuracy under partial modality loss conditions.

[0171] Experimental environment and training parameters: All models in this invention were trained and tested in a unified software environment. Experiments were implemented using the Python deep learning framework and run on the GPU-accelerated PyTorch platform. During model training, a CUDA environment was used for tensor computation to improve the efficiency of multimodal input processing and Perceiver-IO structure training. Since this invention focuses on comparative analysis between different model structures, the basic training configuration was shared as much as possible among the models, with only necessary adjustments made to parts closely related to the model structure.

[0172] In the baseline model, the five modal features are first mapped to a unified latent space dimension d. model =128, the default size of the Perceiver latent array is set to 32 latent vectors, the number of multi-head attention heads is 4, the encoding depth is 2 layers, and the dropout coefficient is set to 0.1. Mini-batch iteration is used for optimization during training, with a batch size of 64 and a maximum training epoch of 100. The AdamW optimizer is used uniformly to improve the stability of multi-branch parameter updates; weight decay is also used to reduce overfitting. For the baseline model, a relatively high learning rate is set to accelerate the convergence of the joint classification task.

[0173] For the anchor-residual improved model and the robust extension model, to adapt to the training complexity brought about by multi-task loss and multi-branch structure, this invention adopts a more conservative optimization strategy, including a smaller initial learning rate, shorter patience epochs, and label smoothing and gradient pruning mechanisms. This is because the improved model simultaneously contains joint classification branches, scene branches, intent branches, and gesture assistance branches. If the same aggressive learning rate as the baseline is used, it can easily lead to training oscillations or imbalances between scene and intent branches in later stages. The robust model further introduces modality loss modeling on this basis, thus requiring even higher training stability.

[0174] Furthermore, all models underwent modal feature standardization before training, and the standardized parameters were fitted only on the training set before being uniformly applied to the validation and test sets to avoid leakage of test information. During model training, parameter selection was based on validation set performance. Training was stopped early when validation set performance ceased to improve within several rounds, and the model parameters that best performed on the validation set were restored. For ease of comparison in subsequent experiments, the main training parameters are summarized in Table 8.

[0175] Table 8 Main Experimental Parameter Settings Comparison Model Explanation: To fully verify the effectiveness of the proposed method, multiple comparative models, ranging from simple to complex, were set up in the experiment. First, the five-modal Perceiver-IO model was used as the formal baseline model. This model unifies the five modalities—IMU, gesture, audio, text, and scene—into a token sequence, and completes encoding, latent space fusion, and joint classification output through the standard Perceiver-IO structure. This baseline model reflects the basic level achievable in multimodal joint recognition tasks without introducing primary / secondary modality differentiation and missing modeling mechanisms.

[0176] Building upon the baseline model, this invention further constructs an improved anchor-residual model. This model no longer employs a five-modal equal-weight fusion strategy. Instead, based on modal contribution analysis results, it uses gesture, text, and scene as anchor backbones to construct stable core semantic representations; IMU and audio are used as supporting modalities, supplementing dynamic information as residuals, and adjusting the contribution of auxiliary residuals through a lightweight gating mechanism. This model is used to verify whether "explicitly distinguishing the primary and secondary relationships of modalities" is superior to unified equal-weight fusion.

[0177] Furthermore, considering the common problems encountered in real-world augmented reality glasses interaction, such as speech recognition failures, gesture occlusion, or instability of some sensors, this invention constructs a robust extended model for modality loss based on the main improved model. This model introduces Anchor Dropout, Missing Token, and AvailabilityEmbedding within the anchor core to explicitly model the loss of key anchor modalities such as gesture and text. Simultaneously, it introduces a selective ModalitySpecificGate only for the two supported modalities, IMU and audio, for lightweight cleaning of auxiliary features under incomplete input and noisy conditions. This model is used to verify the stability and generalization ability of the method in this invention under partial modality loss conditions.

[0178] In addition to the main model described above, this invention also conducted exploratory experiments on several intermediate structures or additional mechanisms, such as different latent vector sizes, different gating mechanisms, and different modality combinations. These experiments are not core components of the main model chain, but are mainly used to help illustrate the influence boundaries of different structural designs on joint classification, intent classification, and scene classification. Through the above hierarchical comparison, this invention can systematically answer the following questions: What is the performance upper limit of the five-modal baseline in real-world scenarios? Are the roles of different modalities balanced in the task? Is the anchor-residual structure effective? And does the model still have good stability under modality loss conditions?

[0179] Example 5 The present invention also provides an augmented reality glasses multimodal user interaction intent recognition system based on modality missing modeling. The system is used to implement the method described in Embodiment 1. The system includes: a dataset construction module, a feature extraction module, a feature statistics module, a model construction module, and an intent recognition module. The dataset building module is used to build multimodal interactive datasets for augmented reality glasses scenarios; The feature extraction module is used to extract features from the multimodal interaction dataset; The feature statistics module is used to perform statistical analysis on the extracted modal features to obtain multimodal samples; The model building module is used to build multimodal user interaction intent recognition models; The intent recognition module is used to input multimodal samples into a trained multimodal user interaction intent recognition model to identify user interaction intents.

[0180] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for recognizing multimodal user interaction intent in augmented reality glasses based on modality missing modeling, characterized in that, The method includes: Construct a multimodal interaction dataset for augmented reality glasses scenarios; Feature extraction is performed on the multimodal interaction dataset; Statistical analysis was performed on the extracted modal features to obtain multimodal samples; Construct a multimodal user interaction intent recognition model; Multimodal samples are input into a trained multimodal user interaction intent recognition model to identify user interaction intents; Methods for constructing multimodal user interaction intent recognition models include: A five-modal Perceiver-IO model under real-world conditions is constructed, and an anchor-residual improvement model and a robust extension model are further introduced on the basis of the five-modal Perceiver-IO model. The methods for constructing the anchor-residual improvement model include: Construct an anchor trunk consisting of three modalities: gesture, text, and scene; After the anchor trunk is established, the two supporting modes, IMU and audio, are used as residual supplementary branches, and auxiliary information is injected in a residual manner. Methods for constructing robust expansion models include: Introduce an explicit missing modeling mechanism within the anchor point backbone; Explicit missing modeling mechanisms include: anchor point random discarding and missing token mechanism, availability embedding mechanism, and selective element-wise gating for IMU+Audio; The anchor point random discarding mechanism is as follows: on a modal basis, the two anchor point branches, gesture and text, are randomly retained or set to empty. The missing token mechanism is as follows: Let the original token representations of gesture and text be respectively... and If the input is missing, then the replacement is: ； in, and Learnable missing tokens for gesture and text, respectively; This represents the input when the gesture is missing. This indicates input when text is missing. The availability embedding mechanism is as follows: ;in, For the final input format, This indicates the state embedding when the modality is available. This represents the state embedding when the modality is missing. This is the projection feature matrix when the modality is available. A learnable matrix of missing tokens; Selective element-wise gating for IMU + Audio is as follows: Assume supported modal inputs are... Its gated output is represented as: ； Among them, the gate matrix The calculation method is as follows: ； In the formula For the Sigmoid function, For learnable weight matrix, For learnable bias, This is the learnable gating bias term for this mode.

2. The method according to claim 1, characterized in that, The process of constructing a multimodal interaction dataset for augmented reality glasses scenarios includes: To identify application scenarios for augmented reality glasses; Analyze user interaction behavior in augmented reality glasses application scenarios to obtain user interaction intent; Based on the user's interaction intent, HoloLens and an external fisheye camera were used to collect data of different modalities to generate multimodal interaction samples. Data preprocessing and time alignment are performed on multimodal interaction samples to obtain a multimodal interaction dataset for augmented reality glasses scenarios.

3. The method according to claim 2, characterized in that, Methods for data preprocessing and time alignment of multimodal interaction samples include: The audio stream from the video recorded by HoloLens is selected as the time alignment reference, and effective speech segments are extracted through Voice Activity Detection (VAD) to generate an approximate timestamp for each interaction sample. The approximate timestamp of each interaction sample is used as a unified time anchor point to align external fisheye camera video, audio, text, and IMU modalities.

4. The method according to claim 3, characterized in that, Feature extraction of the multimodal interaction dataset includes: gesture feature extraction, audio feature extraction, text feature extraction, IMU feature extraction, and scene modality feature extraction; The gesture features are represented as follows: Audio features are represented as follows: , The MFCC feature frame number of the current audio window; the text modal features are represented as: The IMU modal characteristics are represented as follows: Scene modal features are represented as follows: .

5. The method according to claim 1, characterized in that, Methods for constructing a five-modal Perceiver-IO model under real-world conditions include: The five modalities are converted into fixed-length token sequences: for the four modalities of IMU, gesture, audio, and text, a temporal representation of length 10 is used; for the scene modality, a single global token is used. They are uniformly projected to the same latent space dimension using modality-specific linear mappings. : ； in, The latent space features after projection of each modality are preserved in the temporal mode. Step, scene modality is One token; It is a learnable linear transformation with no bias; After projection, time position encoding is added to the four time-series modes. At the same time, learnable modality codes are added to each modality. ; Only modal coding is added to the scene modalities This is to represent its semantic identity as a global context token: For time-series modes At the t-th time step, t = 1, 2, …, 10: ； in, Let m be the original projected features of mode m at time step t. This is the final input token after encoding; For scene modalities, the encoded representation is as follows: ； in, This serves as the final input token for the scene modality; After projection and encoding are completed, the five modalities are concatenated along the token dimension to form a unified input sequence: ； The model input consists of a total of 41 tokens, including 40 time-series tokens and 1 scene token; After completing the input tokenization, Perceiver-IO is used to uniformly model the five modal inputs. The output of each sample is defined as a joint label of scene and intent, thus completing the construction of the five modal Perceiver-IO model under real scene conditions.

6. The method according to claim 1, characterized in that, Methods for constructing an anchor backbone consisting of three modalities: gesture, text, and scene include: Will The three modalities are each mapped to a unified latent space dimension through linear projection. get Subsequently, for gesture and text modalities, learnable temporal positional encodings are superimposed. With modal coding This forms a unified anchor token sequence. For scene modalities, only modal encoding is added without temporal position encoding to obtain the token sequence. The anchor token sequence is concatenated in chronological order to obtain the anchor input sequence: ； Input anchor points into the sequence Feed into the Perceiver encoder: This encoder first compresses the input information into a set of learnable latent arrays through cross-attention. In; specifically, with As a query, with As keys and values, calculate cross-attention: ； in To create a learnable projection matrix, key information from the input sequence is extracted and written into the latent array, achieving a compressed mapping from the high-dimensional input space to the low-dimensional latent space. Subsequently, multiple layers of self-attention and feedforward networks are stacked within the latent space to perform deep interaction and feature refinement on the latent representation. l The layer is calculated as follows: ； go through l After stacking the layers, the final latent representation is obtained. This completes the deep fusion and global modeling of cross-modal information; after processing by the encoder, a learnable output query vector is used. By using decoder cross-attention to read task-related information from the latent representation, the anchor backbone representation is obtained. : 。 7. The method according to claim 1, characterized in that, After the anchor trunk is established, the IMU and audio support modes are used as residual supplementary branches, and auxiliary information is injected in a residual manner. The methods include: The IMU and audio modal are represented by anchor points through cross-attention readout modules. As a query, a sequence of tokens for each modality. As keys and values, extract supplementary features that are most relevant to the current main semantics: ； in, These are supplementary features extracted from the IMU and audio modal that are most relevant to the anchor backbone semantics, and are used for subsequent residual injection; The supplementary features are superimposed on the anchor backbone representation as residuals to form the final fused representation: ； in For the weighted residuals of the two supporting modes, , For dynamic gating values, , Hyperparameters for controlling the maximum injection intensity.

8. A multimodal user interaction intent recognition system for augmented reality glasses based on modality missing modeling, the system being used to implement the method described in any one of claims 1-7, characterized in that, The system includes: a dataset construction module, a feature extraction module, a feature statistics module, a model construction module, and an intent recognition module; The dataset construction module is used to construct a multimodal interactive dataset for augmented reality glasses scenarios; The feature extraction module is used to extract features from the multimodal interaction dataset; The feature statistics module is used to perform statistical analysis on the extracted modal features to obtain multimodal samples; The model building module is used to build a multimodal user interaction intent recognition model; The intent recognition module is used to input multimodal samples into a trained multimodal user interaction intent recognition model to identify user interaction intents.

Images