Mixed reality multi-modal dynamic identification method and system based on cross-modal space-time Transform

Abstract

The invention relates to the technical field of mixed reality multi-modal recognition, in particular to a mixed reality multi-modal dynamic recognition method based on cross-modal space-time Transform, which comprises the following steps of: constructing a multi-modal video data set and preprocessing the multi-modal video data set, constructing a cross-modal space-time Transform model architecture comprising a feature extractor, a cross-frame tube Transform module and a multi-frame Transform fusion module, and constructing a multi-modal video data set; and designing a diversity-consistency loss function, training a model in combination with identity loss and triple center loss, and deploying the trained model to perform dynamic recognition and virtual-real mapping output on multi-modal video data in a mixed reality scene. According to the method, multi-mode global dependency modeling and sequence consistency constraint are established in the space-time dimension, dynamic alignment and semantic fusion among multi-source visual modes are achieved, and the accurate capturing capacity for global space-time features and long-range time sequence dependency is improved.

Description

Technical Field

[0001] This invention relates to the field of mixed reality multimodal recognition technology, and in particular to a mixed reality multimodal dynamic recognition method and system based on cross-modal spatiotemporal Transformer. Background Technology

[0002] Multimodal visual recognition technology in mixed reality is a core supporting technology for mixed reality interactive systems. It is mainly used to achieve dynamic recognition, semantic alignment and interactive response of targets in virtual and real scenes.

[0003] Existing mixed reality multimodal recognition technologies mainly fall into two categories. The first category is multimodal recognition methods based on the combination of convolutional neural networks and recurrent neural networks. These methods extract static spatial features in visible light, infrared, and other modalities, and utilize recurrent structures to model inter-frame dependencies in the temporal dimension. While this type of method can capture short-term dynamic information within some modalities, it is limited by the local receptive field of the convolutional structure, making it difficult to fully describe the spatiotemporal global dependencies of multimodalities. Furthermore, recurrent networks suffer from gradient vanishing and feature memory decay problems in long sequence scenes, making it difficult to handle complex interactions and rapidly changing multimodal fusion scenarios.

[0004] The second category is mixed reality recognition methods based on temporal attention or graph convolution structures. These methods achieve cross-modal semantic association and feature fusion by performing graph modeling or attention weighting on multimodal sequences. While these methods improve intermodal consistency to some extent, they often employ shallow fusion strategies, which are insufficient in characterizing the global correlation of spatiotemporal features and struggle to cope with dynamic instability issues caused by changes in lighting, occlusion interference, and reflection errors in mixed reality environments.

[0005] Meanwhile, some methods rely on high-precision depth cameras or multimodal sensor arrays, resulting in high hardware costs and making them unsuitable for large-scale application in industrial mixed reality equipment. In addition, most existing multimodal fusion algorithms only perform modal alignment at the static frame level, lacking joint modeling of long-term dependencies in the temporal dimension, leading to insufficient cross-frame feature consistency and difficulty in ensuring semantic continuity and interaction stability in virtual and real spaces. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a mixed reality multimodal dynamic recognition method and system based on cross-modal spatiotemporal Transformer.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer, characterized by comprising the following steps: Construct a multimodal video dataset and perform preprocessing; A cross-modal spatiotemporal Transformer model architecture is built, including a feature extractor, a cross-frame Transformer module, and a multi-frame Transformer fusion module; Design a diversity-consistency loss function, combine identity loss and triple center loss to form the total loss function, and train the model; Deploy the trained model to dynamically identify and map virtual and real data from multimodal video data in mixed reality scenarios.

[0008] Preferably, the implementation steps of step 1 are as follows: The HITSZ-VCM dataset was selected, which contains 251,452 RGB images and 211,807 infrared images, corresponding to 927 unique identities. Captured by 12 non-overlapping cameras, it generates a large number of RGB and infrared trajectories, covering both training and testing sets, and includes both visible-infrared and infrared-visible evaluation scenarios. Preprocessing was performed on the data, adjusting the input images to 288×144 pixels and employing data augmentation techniques such as random cropping, horizontal flipping, and channel enhancement to improve the model's generalization ability. The video sequence was divided into segments with a fixed number of frames, each segment containing a continuous sequence of frames, preparing for subsequent feature extraction and spatiotemporal modeling.

[0009] Preferably, the implementation steps of step 2 are as follows: Feature Extractor Setup A ResNet50-based two-stream network is used as the backbone network, which is pre-trained on ImageNet. The five stages of ResNet50 are divided into a modality-specific extractor and a modality-sharing extractor. The modality-specific extractor captures the discriminative information inherent in each modality, while the modality-sharing extractor integrates the features of different modalities into a common space to reduce cross-modal differences. Through this feature extractor, modal feature vectors are extracted from video clips of visible and near-infrared modalities, respectively.

[0010] Cross-frame management Transformer module setup The feature vectors of the visible and near-infrared modes are divided into multiple time slices along the time dimension. Each time slice is reshaped into a series of 3D tube structures, each of which encapsulates the motion trajectory information of the image in the time dimension. The number of tubes is determined by the size of the feature map and the preset tube size. A class label ([CLS]) is added at the beginning of the tube sequence to capture global features, and a learnable position embedding is introduced for each tube to incorporate spatial information. The processed tube sequence is input into the Transformer module, and through a multi-head self-attention mechanism, layer normalization, and a feedforward network, global contextual relationships between tubes are established to capture the interdependence between cross-frame local features and global spatial information, thereby obtaining the global spatiotemporal features of each modality.

[0011] Building a multi-frame Transformer fusion module For the global spatiotemporal features output by the cross-frame Transformer module, a linear transformation is performed on the class label of each time slice to generate a message token representing the information of that time slice. Through the inter-frame interaction attention module, the message token is input into three linear projection layers to generate queries, keys, and values. The attention mechanism enables information interaction between different time slices, compensating for the lack of long-term information in a single time slice. The resulting interactive message token is concatenated with the patch features of the corresponding time slice and input into the intra-frame fusion attention module to establish the spatiotemporal relationships within the time slices, obtaining a fused feature set. The class label is concatenated with this feature set and input into the temporal fusion module. This module uses a standard Transformer structure, employing a multi-head self-attention mechanism, layer normalization, and a feedforward network to achieve information exchange between all time slices, ultimately obtaining a video-level feature representation. Finally, the SENet attention mechanism is used to aggregate the convolutional neural network and the temporal fusion module, resulting in a final video-level feature representation that fuses local and global information.

[0012] Preferably, step 3 is implemented as follows: Loss function design The total loss function includes identity loss (learning discriminative features), triple center loss (ensuring intra-class compactness), and diversity-consistency loss (including invariance constraints to reduce modality differences, variance constraints to improve feature diversity, and covariance constraints to reduce information redundancy). The expression is: Total Loss = Identity Loss + Triple Center Loss + Diversity-Consistency Loss Model training The SGD optimizer was used with an initial learning rate of 0.1 (dynamically adjusted according to a preset strategy); the batch size was set to 8 (containing 4 identities, with 2 video clips per identity); and training was performed for 300 epochs until the loss converged.

[0013] Preferably, step 4 is implemented as follows: The trained cross-modal spatiotemporal Transformer model is deployed into a mixed reality multimodal recognition system. The system includes visible light and infrared cameras to acquire video sequences of different modalities. After the system starts, the cameras acquire video data in the scene in real time and process the video segments according to the preprocessing method in step 1 to obtain multimodal temporal samples that meet the model input requirements. The preprocessed video is input into the cross-modal spatiotemporal Transformer model. Through feature extraction, cross-frame management Transformer module and multi-frame fusion Transformer module, the fused multimodal spatiotemporal feature representation is obtained. Based on this feature representation, the system calculates the dynamic consistency and semantic similarity between different modal video frames in the mixed reality environment to realize the recognition, state judgment and interactive response of targets in the virtual and real scenes. The model output includes frame-level recognition results and temporal consistency index to evaluate the dynamic recognition accuracy and stability of the model in the mixed reality scene. The system generates virtual-real mapping feedback according to the semantic consistency distribution of the fused features to realize the dynamic recognition and virtual-real fusion of multimodal visual information in the mixed reality environment.

[0014] A Mixed Reality Multimodal Dynamic Recognition System Based on Cross-Modal Spatiotemporal Transformer Its features include: Data preprocessing module: Constructs the HITSZ-VCM multimodal video dataset, performs resizing and data augmentation on the input images, divides the video sequence into continuous frame segments, and provides standardized samples for subsequent modeling; Feature extraction module: It adopts a dual-stream ResNet50 staged architecture, extracting feature vectors of visible mode and near-infrared mode respectively through modality-specific extractor and modality-shared extractor, balancing modality uniqueness and cross-modality consistency; Cross-frame spatiotemporal modeling module: namely the cross-frame tube Transformer module, which captures the cross-frame spatial relationship of local features through 3D tube tokenization and global association with Transformer, and generates global spatiotemporal features of each modality; Multi-frame fusion module: also known as multi-frame Transformer fusion module, which models long-range temporal dependencies and generates video-level feature representations that fuse local and global information through message token generation, dual attention mechanism (inter-frame interaction attention + intra-frame fusion attention) and temporal fusion module; Model training module: Based on the diversity-consistency loss function, identity loss and triple center loss to form the total loss function, the SGD optimizer and preset training strategy are used to complete model training and convergence; Recognition Output Module: Deploys the trained model, processes multimodal video data acquired by the camera in real time, calculates dynamic consistency and semantic similarity, and outputs frame-level recognition results, temporal consistency indicators, and virtual-real mapping feedback.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention addresses multimodal dynamic recognition scenarios in mixed reality by proposing a multimodal feature modeling technique based on a cross-modal spatiotemporal Transformer. By introducing a spatiotemporal attention mechanism and a multimodal interaction structure into the Transformer architecture, global modeling and dynamic fusion of features across multiple frames are achieved, simultaneously capturing the temporal trend and semantic relationships between modalities.

[0016] Compared with existing technologies, the method of this invention does not rely on additional depth sensors or complex hardware. It only requires conventional visible light and infrared camera equipment to complete multimodal spatiotemporal feature learning, resulting in lower system deployment costs and good engineering feasibility. Through joint modeling of the cross-frame management Transformer module and the multimodal fusion module, the system can maintain the stability of recognition results and the consistency of virtual-real mapping even in complex lighting, occlusion, and dynamic interaction environments. Attached Figure Description

[0017] Figure 1 is a flowchart of the overall process of the present invention.

[0018] Figure 2 is a visualization of the distributed nature of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0020] Please refer to Figure 1. This invention provides a technical solution: a mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer, comprising the following steps: Step 1: Construct a multimodal video dataset and perform preprocessing. The HITSZ-VCM dataset was selected. This dataset contains 251,452 RGB images and 211,807 infrared images, corresponding to 927 unique identities. It was captured by 12 non-overlapping cameras, generating a large number of RGB and infrared trajectories, covering both the training and test sets, and including two evaluation scenarios: visible-infrared and infrared-visible.

[0021] The data is preprocessed by adjusting the input image to a fixed size of 288×144, expanding the sample distribution range through random cropping, simulating the target shape from different perspectives by horizontal flipping, and enhancing the channel to improve the model's adaptability to changes in features within the modality, thereby improving the model's generalization ability.

[0022] The video sequence is divided into video segments with a fixed number of frames. Each segment contains a continuous sequence of frames, ensuring the integrity of the temporal information and providing structured input for subsequent feature extraction and spatiotemporal modeling.

[0023] Step 2: Build a cross-modal spatiotemporal Transformer model architecture Feature extractor implementation A two-stream network based on ResNet50 is used as the backbone network. This network has been pre-trained on the ImageNet dataset and has strong basic feature extraction capabilities.

[0024] The five stages of ResNet50 are divided into modality-specific extractors (s0 and s1 stages) and modality-sharing extractors (s2-s4 stages). The modality-specific extractors focus on capturing the discriminative information inherent in each modality, preserving modality uniqueness; while the modality-sharing extractors integrate the features of different modalities into a common space, effectively reducing the fusion difficulty caused by cross-modal differences.

[0025] This feature extractor extracts dimensionally uniform modal feature vectors from video clips in both visible and near-infrared modes, laying the foundation for subsequent spatiotemporal modeling.

[0026] Cross-frame management Transformer module implementation The feature vectors of the visible and near-infrared modes are divided into multiple time slices according to the time dimension, and each time slice corresponds to a continuous temporal feature.

[0027] Each time slice is reshaped into a series of 3D tube structures. Each 3D tube encapsulates the motion trajectory information of the image in the time dimension. The number of tubes is determined by the size of the feature map and the preset tube size to ensure complete encapsulation of trajectory information.

[0028] A class marker ([CLS]) is added at the beginning of the tube sequence to capture global feature information; at the same time, a learnable position embedding is introduced for each tube to incorporate spatial position information into the feature representation.

[0029] The processed tube sequence is input into the Transformer module. A global context relationship between tubes is established through a multi-head self-attention mechanism. Feature optimization is performed by combining layer normalization and a feedforward network. Finally, the interdependence between local features across frames and global spatial information are captured to obtain the global spatiotemporal features of each modality.

[0030] Multi-frame Transformer fusion module implementation For the global spatiotemporal features output by the cross-frame management Transformer module, a linear transformation is performed on the class label of each time slice to generate a message token that can represent the core information of the time slice.

[0031] The inter-frame interaction attention module inputs message tokens into three linear projection layers to generate queries, keys, and values, respectively. The attention mechanism is used to calculate the association weights between message tokens in different time slices, enabling cross-slice information interaction and compensating for the lack of long-term information in a single time slice.

[0032] The obtained post-interaction message token is concatenated with the patch features of the corresponding time slice and input into the intra-frame fusion attention module to establish the spatiotemporal relationship within the time slice, strengthen the correlation between local features and global features, and obtain the fused feature set.

[0033] The class labels are concatenated with the feature set and input into the temporal fusion module (standard Transformer structure). Through multi-head self-attention mechanism, layer normalization and feedforward network, deep information exchange between all temporal slices is realized, and finally video-level feature representation is obtained.

[0034] By leveraging the SENet attention mechanism, the output features of the convolutional neural network and the output features of the temporal fusion module are aggregated to highlight the weights of key features, resulting in a final video-level feature representation that integrates local and global information.

[0035] Step 3: Design the loss function and train the model Loss function construction The total loss function consists of three parts: identity loss (used to learn discriminative feature representations), triple center loss (ensuring the compactness of similar features), and diversity-consistency loss (including invariance constraints to reduce modality differences, variance constraints to improve feature diversity, and covariance constraints to reduce information redundancy). The specific expression is: Total Loss = Identity Loss + Triple Center Loss + Diversity-Consistency Loss Model training process The SGD optimizer is used to update the model parameters. The initial learning rate is set to 0.1 and dynamically adjusted according to a preset strategy to ensure the stability and convergence speed of the training process.

[0036] Set the batch size to 8, with each batch containing 4 identities and each identity corresponding to 2 video clips to ensure a balanced sample distribution.

[0037] Continue training for 300 rounds until the loss function value converges, completing the model training.

[0038] Step 4: Model Deployment and Dynamic Recognition Output The trained cross-modal spatiotemporal Transformer model is deployed into a mixed reality multimodal recognition system that integrates visible light and infrared cameras to acquire video sequences of different modalities in real time.

[0039] Once the system is started, the camera acquires video data from the scene in real time. The video segments are standardized according to the preprocessing methods in step 1 (size adjustment, data augmentation) to obtain multimodal temporal samples that meet the model input requirements.

[0040] The preprocessed video is input into the cross-modal spatiotemporal Transformer model, and then processed sequentially through feature extraction, cross-frame management Transformer module, and multi-frame fusion Transformer module to obtain the fused multimodal spatiotemporal feature representation.

[0041] Based on this feature representation, the system calculates the dynamic consistency and semantic similarity between different modal video frames in a mixed reality environment, enabling target recognition, state judgment, and interactive response in virtual and real scenes.

[0042] The model outputs frame-level recognition results and temporal consistency metrics to evaluate the model's dynamic recognition accuracy and stability in mixed reality scenarios.

[0043] The system generates virtual-real mapping feedback based on the semantic consistency distribution of fused features, ultimately achieving dynamic recognition and virtual-real fusion of multimodal visual information in a mixed reality environment.

[0044] Alternative solutions In the cross-modal spatiotemporal Transformer technology described in this invention, multiple modules and core algorithms can also be implemented through other equivalent solutions. The following description of alternative solutions is only for expanding technical paths and is not a limitation of this invention.

[0045] In step 2, the 3D tube feature modeling of the cross-frame tube Transformer module is based on the Transformer architecture to achieve global association. However, other networks that support global spatiotemporal feature modeling (such as Video SwinTransformer and Deformable Video Transformer) can also achieve cross-frame global feature capture by adjusting the input token format (such as replacing the 3D tube with a spatiotemporal patch), and therefore can be used as an alternative.

[0046] In step 2, for the long-range temporal fusion of the multi-frame Transformer fusion module, this invention uses an inter-frame interactive attention and temporal fusion module. However, other temporal modeling networks (such as the improved version of Long Short-Term Memory and GatedRecurrent Unit) can also achieve multi-frame information fusion by stacking multiple layers to enhance long-range dependency capture capabilities, and therefore can be used as alternatives.

[0047] In step 3, the diversity-consistency loss function is used. In this invention, features are optimized through invariance, variance, and covariance constraints. However, if other losses that take into account both cross-modal consistency and feature diversity are used (such as the improved version of Contrastive Loss and VICReg Loss), feature representation collapse can also be avoided by adjusting the weights of the constraint terms to adapt to cross-modal scenarios. Therefore, it can be used as an alternative.

[0048] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer, characterized in that, Includes the following steps: Step 1: Construct a multimodal video dataset and perform preprocessing; Step 2: Build a cross-modal spatiotemporal Transformer model architecture, including a feature extractor, a cross-frame Transformer module, and a multi-frame Transformer fusion module; Step 3: Design a diversity-consistency loss function, combine identity loss and triple center loss to form the total loss function, and train the model; Step 4: Deploy the trained model to perform dynamic recognition and virtual-real mapping output on multimodal video data in mixed reality scenarios.

2. The mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer as described in claim 1, characterized in that, In step 1, the HITSZ-VCM dataset (containing 927 identities, RGB / infrared data and trajectories from 12 cameras) is selected, and preprocessing is performed using "288×144 fixed size + random cropping / horizontal flipping / channel enhancement" to provide realistic and diverse samples for cross-modal modeling.

3. The mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer as described in claim 1, characterized in that, In step 2, the feature extractor adopts a "dual-stream ResNet50 staged design", with modality-specific (s0, s1) and shared (s2-s4) extractors to balance modality uniqueness and cross-modality consistency. The cross-frame tube Transformer module captures the cross-frame spatial relationship of local features through "3D tube tokenization + Transformer global association", breaking through the local limitations of convolutional neural networks; the multi-frame Transformer fusion module models long-term temporal dependencies with "message tokens + dual attention + temporal fusion module", solving the problem of forgetting long sequence information in long short-term memory networks.

4. The mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer as described in claim 1, characterized in that, In step 3, the innovation diversity-consistency loss function avoids cross-modal feature collapse through triple constraints of "invariance + variance + covariance"; the total loss is calculated according to... Optimize by combining dynamic learning rate and fixed batch strategy to train for 300 rounds.

5. The mixed reality multimodal dynamic recognition method based on cross-modal spatiotemporal Transformer as described in claim 1, characterized in that, In step 4, an end-to-end process of "real-time preprocessing - feature extraction - similarity matching" is constructed, and Rank-k and mean precision are used for evaluation to ensure accurate results. The performance on the HITSZ-VCM dataset is better than existing methods.

6. A system for a mixed reality multimodal visual fusion and recognition method based on homogeneous-heterogeneous relationship graphs according to any one of claims 1-5, characterized in that: The data preprocessing module constructs the HITSZ-VCM multimodal video dataset, performs data augmentation such as 288×144 size adjustment and random cropping on the input images, and divides the video sequence into continuous frame segments. The feature extraction module adopts a dual-stream ResNet50 staged architecture and extracts visible and near-infrared modal feature vectors through modality-specific and shared extractors. The cross-frame spatiotemporal modeling module divides the feature vector into time slices and reshapes them into a 3D tube structure, adds class labels and position embeddings, and captures global spatiotemporal features through the Transformer module; The multi-frame fusion module generates message tokens, and through a dual attention mechanism and a temporal fusion module, combined with SENet attention aggregation, outputs video-level features; The model training module uses the SGD optimizer to train the model for 300 rounds, based on identity loss, triple center loss and diversity-consistency loss to form the total loss. The recognition output module processes camera-acquired data in real time, calculates dynamic consistency and semantic similarity, and outputs frame-level recognition results, temporal consistency indicators, and virtual-real mapping feedback.

Images