A multimodal, sensory, digital art installation with immersive interaction and methods for its implementation.

Abstract

The present invention discloses a multimodal sensor fusion device for immersive interactive digital art and its implementation method. Through the transformer architecture design within the multimodal data fusion submodule, this invention achieves precise recognition of the user's interaction intent. The system constructs different feature encoding paths for three modalities: motion, eye tracking, and speech. The motion modality extracts temporal features of human joint positions from successive images; the eye-tracking modality encodes simultaneous gaze focus and fixation duration sequences; and the speech modality converts recognized text into semantic embedding vectors. The concatenated features from all three modalities are fed into the transformer encoder.The multi-head self-attention mechanism captures latent cross-modality correlations, while a weighted fusion formula dynamically distributes attentional weights across modalities. This allows the system to accurately correlate semantic information across different modalities, thus precisely understanding the user's true intent in complex interaction scenarios and avoiding misjudgments due to biases from data originating from a single modality.