A multi-modal remote collaboration interaction system fusing artificial intelligence and virtual reality

Abstract

The present application relates to human-computer interaction, artificial intelligence and virtual reality technical field, specifically to a kind of multi-modal remote collaboration interactive system of fusion artificial intelligence and virtual reality. Including data acquisition module, for real-time acquisition user's original physiological data stream and interactive behavior data stream;Load calculation module, for preset physiological data weight coefficient and interactive behavior data weight coefficient;Spike detection module, for calculating instantaneous load relative deviation;And determine whether cognitive spike event occurs and output cognitive spike event flag bit;Dynamic weight variation module, for solving instantaneous automation level;Interactive execution module, for the fusion arbitration of AI auxiliary instruction and user original operation instruction, and output final weight variation instruction. The present system provides more comprehensive, more objective quantitative basis, just as the communicative performance evaluation of foreign language learner in cross-cultural environment needs to consider language and non-language factors to ensure the accuracy of state evaluation.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, artificial intelligence and virtual reality, specifically a multimodal remote collaborative interaction system that integrates artificial intelligence and virtual reality. Background Technology

[0002] In a multimodal remote collaborative interaction system that integrates artificial intelligence and virtual reality, the goal is to improve the efficiency of human-computer collaboration through AI assistance. This is similar to an intelligent foreign language learning system that attempts to improve learners' expression efficiency through grammar and vocabulary assistance. The system optimizes the user's original operation commands through a built-in AI-assisted model, such as performing trajectory prediction or anti-shake smoothing, thereby outputting more stable and efficient commands.

[0003] Existing technologies face a technical problem when applied to high-risk collaborative scenarios: these AI-assisted systems typically lack the ability to perceive the user's real-time cognitive state. Under normal circumstances, AI assistance can improve efficiency; just as moderate error correction can help students improve in regular English teaching. However, when users face sudden crises and their cognitive load increases sharply, such as in a high-risk cross-cultural business negotiation, their rapid, even grammatically jarring and non-standard, operational reactions in response to the crisis are precisely the most genuine reflection of their intentions.

[0004] At this point, AI assistance designed for normal situations, such as automatic grammar correction or cultural behavior smoothing algorithms, may conflict with the user's true crisis management intentions. This could lead to the incorrect correction of the user's critical operations, which is equivalent to interrupting the learner's speech in an emergency and forcibly correcting minor language errors while ignoring the primary communicative intent. This could turn assistance into a fatal intervention. This conflict between human and machine intentions at critical moments makes it impossible for the system to ensure that it can fully and instantly return control to the user in a crisis, posing a serious safety hazard.

[0005] The information disclosed in the background section above is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention discloses a multimodal remote collaborative interaction system integrating artificial intelligence and virtual reality. Specifically, the technical solution of this invention includes:

[0007] The data acquisition module is used to collect users' raw physiological data stream and interactive behavior data stream in real time. The raw physiological data stream includes heart rate variability indicators and skin conductance response indicators, while the interactive behavior data stream includes kinematic vectors, gaze point data, and operation frequency.

[0008] The load calculation module is used to calculate the standardized instantaneous cognitive load index based on the raw physiological data stream and interactive behavior data stream, and combined with the preset physiological data weight coefficient and interactive behavior data weight coefficient, through an additive weight fusion model.

[0009] The peak detection module is used to calculate the relative deviation of instantaneous cognitive load based on the instantaneous cognitive load index and the calibrated user cognitive load reference baseline; and based on the comparison result of the relative deviation of instantaneous cognitive load with the preset disaster trigger threshold, to determine whether a cognitive peak event has occurred and output the cognitive peak event flag bit.

[0010] The dynamic contingency module is used to calculate the instantaneous automation level based on the cognitive peak event flag and the preset default automation level;

[0011] The interactive execution module is used to fuse and arbitrate AI-assisted instructions and original user operation instructions based on the instantaneous automation level, and output the final contingency instruction.

[0012] Preferably, the load calculation module calculates the standardized instantaneous cognitive load index, including:

[0013] The raw physiological data stream is processed to obtain normalized physiological state feature values;

[0014] The interaction behavior data stream is processed to obtain normalized abnormal interaction behavior feature values;

[0015] Invoke preset physiological data weighting coefficients and interactive behavior data weighting coefficients;

[0016] Based on physiological state characteristic values, abnormal interactive behavior characteristic values, physiological data weight coefficients, and interactive behavior data weight coefficients, the instantaneous cognitive load index is calculated using an additive weighted fusion model.

[0017] Preferably, the peak detection module is further used for:

[0018] In system calibration mode, the instantaneous cognitive load index data stream is continuously collected during the user's execution of standard tasks;

[0019] Based on the standard definition of the arithmetic mean, calculate the statistical average of the instantaneous cognitive load index data stream within the calibration period;

[0020] Store the statistical average as a reference baseline for user cognitive load.

[0021] Preferably, the spike detection module determines whether a cognitive spike event has occurred and outputs a cognitive spike event flag, including:

[0022] In collaborative mode, the current instantaneous cognitive load index is obtained in real time, and the stored user cognitive load reference baseline is invoked;

[0023] The relative deviation of instantaneous cognitive load is calculated in real time based on the current instantaneous cognitive load index and the user cognitive load reference baseline.

[0024] The instantaneous load relative deviation is compared with a preset disaster trigger threshold;

[0025] A cognitive spike event is determined to have occurred and the cognitive spike event flag is set to 1 if and only if the instantaneous relative load deviation exceeds the disaster trigger threshold.

[0026] Otherwise, set the cognitive peak event flag to 0.

[0027] Preferably, the dynamic contingency module calculates the instantaneous automation level, including:

[0028] Receive the binary cognitive spike event flag from the spike detection module and call the preset default automation level;

[0029] Use the cognitive spike event flag as a switching factor to suppress or enable the default automation level;

[0030] The instantaneous automation level is determined by multiplying the switching factor by the default automation level.

[0031] Preferably, the interactive execution module outputs the final contingency instruction, including:

[0032] It receives raw operation commands from the user's VR controller in real time, as well as AI-assisted commands from the system's built-in AI-assisted model.

[0033] Invoke the instantaneous automation level from the dynamic contingency module;

[0034] Based on linear interpolation logic, the instantaneous automation level is used as a mixing coefficient to weight and mix the AI-assisted instructions and the original operation instructions to generate and output the final contingency instructions.

[0035] Preferably, the interactive execution module:

[0036] When the cognitive peak event flag is 0, the instantaneous automation level is equal to the default automation level, and the final contingency instruction is a weighted mixture of AI-assisted instructions and original operation instructions;

[0037] When the cognitive peak event flag is set to 1, the instantaneous automation level is forcibly reset to zero, and the final contingency instruction is instantly collapsed to be equal to the original operation instruction.

[0038] Preferably, the data acquisition module is further used for:

[0039] Deploy a non-invasive physiological sensor array to collect raw physiological data streams in real time;

[0040] The system tracks interactive behavior data streams synchronously through VR controller tracking and head-mounted display tracking.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. This system, through its data acquisition module, utilizes non-invasive physiological sensors and VR tracking devices to simultaneously capture physiological data streams such as heart rate variability and skin conductance, as well as interactive behavioral data streams such as kinematics and gaze points. Compared to traditional methods relying on single indicators or subjective reports, this multimodal data fusion provides a more comprehensive and objective quantitative basis for quantifying users' cognitive load. Just as an English teacher cannot rely solely on test scores but also needs to consider classroom performance and communication anxiety levels to comprehensively assess students' intercultural communication abilities, this system ensures the accuracy of state assessment from the outset.

[0043] 2. The load calculation module used in this system employs an additive weighted fusion model with non-compensatory characteristics to perform weighted calculations on normalized physiological and behavioral feature values. This model effectively amplifies concurrent physiological and behavioral risk signals, rather than averaging them, thus providing a more sensitive input signal for the subsequent peak detection module and enhancing the system's early warning capability for potential crisis moments.

[0044] 3. The peak detection module of this system introduces a personalized calibration mechanism, which calculates and stores a unique cognitive load reference baseline by collecting user data under standard tasks. During collaboration, the system calculates the relative deviation between the instantaneous load and this baseline in real time to determine peak events. This design effectively overcomes the baseline drift problem caused by individual differences and avoids the high false alarm or false negative rates caused by using universal fixed thresholds.

[0045] 4. This system constructs a rapid response chain from detection to arbitration. The dynamic contingency module uses spike event flags as switching factors to suppress the default automation level, and the interactive execution module performs linear interpolation on human-machine instructions based on this result. This mechanism ensures that AI assistance automatically disables protection the moment a crisis is detected, completely returning control to the user. This enables the system or teacher to instantly stop auxiliary error correction when foreign language learners encounter serious cultural conflicts or communication barriers, and instead fully respect their original expressive intent, achieving absolute intent fidelity and eliminating the defect that AI may turn into interference under pressure. Attached Figure Description

[0046] The present invention will be further explained below with reference to the accompanying drawings and embodiments:

[0047] Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0049] Example 1:

[0050] Please see Figure 1 A multimodal remote collaborative interaction system integrating artificial intelligence and virtual reality includes:

[0051] The data acquisition module is used to collect users' raw physiological data stream and interactive behavior data stream in real time. The raw physiological data stream includes heart rate variability indicators and skin conductance response indicators, while the interactive behavior data stream includes kinematic vectors, gaze point data, and operation frequency.

[0052] The load calculation module is used to calculate the standardized instantaneous cognitive load index based on the raw physiological data stream and interactive behavior data stream, and combined with the preset physiological data weight coefficient and interactive behavior data weight coefficient, through an additive weight fusion model.

[0053] The peak detection module is used to calculate the relative deviation of instantaneous cognitive load based on the instantaneous cognitive load index and the calibrated user cognitive load reference baseline; and based on the comparison result of the relative deviation of instantaneous cognitive load with the preset disaster trigger threshold, to determine whether a cognitive peak event has occurred and output the cognitive peak event flag bit.

[0054] The dynamic contingency module is used to calculate the instantaneous automation level based on the cognitive peak event flag and the preset default automation level;

[0055] The interactive execution module is used to fuse and arbitrate AI-assisted instructions and original user operation instructions based on the instantaneous automation level, and output the final contingency instruction.

[0056] A multimodal remote collaborative interaction system integrating artificial intelligence and virtual reality aims to construct a closed-loop, cognitively adaptive human-computer interaction framework. In this embodiment, the system includes five core modules that work together to achieve a complete logical flow from data acquisition to contingent execution.

[0057] The data acquisition module aims to capture multi-dimensional data reflecting the user's cognitive state in real time and synchronously. This is analogous to an experienced cross-cultural communication teacher keenly observing students' nonverbal signals. In this embodiment, the module acquires the user's raw physiological data stream in real time by deploying a non-invasive physiological sensor array. This data stream explicitly includes: heart rate variability indicators reflecting the balance of the autonomic nervous system, such as those acquired through photoplethysmography (PPG) sensors, and skin conductance indicators reflecting the level of sympathetic nerve excitation. These physiological indicators are like the learner's real stress response when facing culture shock.

[0058] Meanwhile, the module collects interactive behavior data streams in real time through the VR tracking system. These data streams explicitly include: kinematic vectors reflecting the dynamics of user operations, such as the displacement velocity, angular velocity, and acceleration vectors of the VR controller; gaze point data reflecting the user's focus of attention, obtained through eye tracking of the HMD head-mounted display; and operation frequency reflecting task execution efficiency. These behavioral data also indirectly reflect the learner's degree of hesitation or focus of attention during communication.

[0059] The load calculation module aims to fuse heterogeneous, multimodal raw data streams and quantify them into a unified standard metric. This module receives data from the data acquisition module and, based on the raw physiological data stream and interactive behavior data stream, combines two key preset parameters: physiological data weighting coefficients. Weighting coefficients of interaction behavior data The standardized instantaneous cognitive load index is calculated using a specific additive weighted fusion model. This index comprehensively reflects the psychological stress and cognitive overload levels of users, such as a learner undergoing immersive foreign language training.

[0060] Peak detection module: Its purpose is to identify the abrupt change in a user's cognitive state from normal fluctuations to acute overload; similar to assessing the critical point at which an English learner transitions from normal academic stress to acute culture shock in an overseas environment; this module receives continuous data from the load calculation module. The data stream is based on the instantaneous cognitive load index and the system-predefined user cognitive load reference baseline. Real-time calculation of the instantaneous relative load deviation between the two The module is based on the relative deviation of instantaneous load. Compared with the preset disaster trigger threshold The comparison results determine whether a cognitive spike event has occurred; finally, the module outputs a binary cognitive spike event flag. Its value is 0 or 1;

[0061] The dynamic contingency module aims to convert detected binary spike events into continuous control signals for the degree of intervention in the AI ​​system; this module receives cognitive spike event flags from the spike detection module. It then calls a system-preset constant, namely the default automation level. ; This represents the level of AI assistance allowed by the system under normal conditions; similar to the ratio of assistance to error correction in teaching. This module utilizes... As a switching factor, for Suppress or activate to calculate the instantaneous level of automation. ;

[0062] Interactive Execution Module: Its purpose is to serve as the final arbitration outlet for human-machine commands, determining the operation command to be sent to the remote system based on the instantaneous automation level; this module is based on the instantaneous automation level output by the dynamic contingency module. For two input commands—that is, the user's AI-assisted commands. For example, automatic syntax highlighting commands and user-defined commands. For example, learners' unmodified raw foreign language expressions are processed through a fusion arbitration, and the module outputs the final contingency instruction. ;

[0063] This system provides a complete technological closed loop, solving the problem that traditional AI-assisted systems may conflict with user intentions in crisis situations. It objectively quantifies cognitive load by integrating physiological and behavioral data, rather than relying on subjective reports or single indicators. It establishes a rapid response link from cognitive peak detection to instantaneous zeroing of the automation level. This enables the system to provide AI assistance under normal conditions, but automatically returns control to the user the moment it detects that the user has entered an extreme stress state. Thus, while ensuring safety redundancy, it achieves a dynamic balance between human-machine collaboration efficiency and safety under extreme conditions.

[0064] Example 2:

[0065] The load calculation module calculates the standardized instantaneous cognitive load index, including:

[0066] The raw physiological data stream is processed to obtain normalized physiological state feature values;

[0067] The interaction behavior data stream is processed to obtain normalized abnormal interaction behavior feature values;

[0068] Invoke preset physiological data weighting coefficients and interactive behavior data weighting coefficients;

[0069] Based on physiological state characteristic values, abnormal interactive behavior characteristic values, physiological data weight coefficients, and interactive behavior data weight coefficients, the instantaneous cognitive load index is calculated through an additive weighted fusion model.

[0070] Based on Example 1, this embodiment describes the specific technical process by which the load calculation module calculates the standardized instantaneous cognitive load index. The core of this process is to use a non-compensatory additive weighted fusion model to ensure the superposition effect of multi-source risk signals.

[0071] This module preprocesses and normalizes the raw physiological data stream, for example, using Z-Score or Min-Max scaling relative to the baseline, to obtain normalized physiological state feature values, which are defined in this embodiment as... ;Should It is a dimensionless scalar that characterizes the level of physiological stress. To eliminate changes in physiological indicators caused purely by physical activities rather than mental stress, motion artifact removal algorithms can be introduced in the data preprocessing stage. For example, using synchronously acquired accelerometer data as a reference input, the effects of motion interference on heart rate variability can be separated and suppressed through adaptive filtering and other techniques.

[0072] Simultaneously, this module processes the interactive behavior data stream, kinematic vectors, gaze point data, etc., and calculates the degree to which it deviates from the normal operation mode, such as jitter amplitude, decreased gaze entropy, or abnormal operation frequency, to obtain normalized abnormal interactive behavior feature values, which are defined in this embodiment as ;Should Similarly, it is a dimensionless scalar that represents the level of abnormal behavior. For multidimensional interactive behavior data, the Mahalanobis distance of each indicator from its normal pattern can be calculated, or they can be normalized separately and then weighted and summed according to their importance to cognitive load through preset sub-weights to form a single abnormal interactive behavior feature value.

[0073] This module calls the preset physiological data weighting coefficients ( ) and interaction behavior data weighting coefficients ( These two coefficients were obtained from a calibrated dataset containing known levels of cognitive load, from which corresponding physiological characteristic samples were collected. and behavioral characteristic samples Key parameters obtained by performing multivariate regression analysis, such as least squares fitting, to optimize the model's prediction accuracy for known load levels; for example, after calibration, the weights of physiological indicators in a certain task can be determined as follows. The weights of behavioral indicators can be determined as follows: ;

[0074] This module is based on physiological state characteristic values ​​( ), Abnormal interaction behavior feature values ​​( ), physiological data weighting coefficients ( ) and interaction behavior data weighting coefficients ( The instantaneous cognitive load index was calculated using an additive weighted fusion model. This model is an additive utility model in multi-attribute decision theory, and its specific calculation formula is as follows:

[0075]

[0076] in, Instantaneous cognitive load index, a dimensionless scalar, is the final output of this module;

[0077] : Physiological data weighting coefficients, dimensionless;

[0078] : Normalized physiological state characteristic value, dimensionless, is the calculation result of step 1 in this module;

[0079] : Interactive behavior data weighting coefficient, dimensionless, is a preset calibration value;

[0080] : Normalized abnormal interaction behavior feature value, dimensionless, is the calculation result of step 2 in this module;

[0081] This invention is a non-compensatory model, that is and The sum does not need to equal 1; this differs from the traditional weighted average method; its technical effect lies in achieving risk superposition and amplification: when physiological and behavioral abnormalities occur simultaneously, for example... and Total load index It can exceed 1, rather than being averaged; this design is particularly important for capturing the dual pressures faced by foreign language learners due to concurrent language barriers and cultural misunderstandings; making the integration index... It can effectively amplify the risk signals of concurrency, providing a more sensitive and discriminative input signal for subsequent spike detection;

[0082] This embodiment employs a linear additive model to achieve computational simplicity and efficiency, making it suitable for systems with high real-time requirements. In applications requiring higher model fidelity, this additive weighted fusion model can be replaced with a nonlinear model, such as introducing quadratic or cross terms for each feature value, or using an end-to-end model based on neural networks to accurately characterize the complex nonlinear relationship between cognitive load and multimodal data.

[0083] Example 3:

[0084] The spike detection module is also used for:

[0085] In system calibration mode, the instantaneous cognitive load index data stream is continuously collected during the user's execution of standard tasks;

[0086] Based on the standard definition of the arithmetic mean, calculate the statistical average of the instantaneous cognitive load index data stream within the calibration period;

[0087] Store the statistical average as a reference baseline for user cognitive load;

[0088] Based on Example 1, this embodiment describes how the peak detection module generates a personalized user cognitive load reference baseline. This is the key basis for subsequent peak determination;

[0089] This module runs in system calibration mode before formal system collaboration; in this mode:

[0090] The system guides users to perform a series of routine, low-risk standard tasks in the VR environment;

[0091] During this period, the system continuously collects instantaneous cognitive load index data streams from the user during the execution of standard tasks, i.e., a series of data streams from the load calculation module. value, The sample number;

[0092] The calibration period includes a total of After each sample is completed, the module calculates the statistical average of the data stream over the entire calibration period according to the standard arithmetic mean definition; the calculation formula is as follows:

[0093]

[0094] in, User cognitive load reference baseline, a dimensionless scalar;

[0095] The first data collected within the calibration period One load index sample, load calculation module;

[0096] : Total number of samples within the calibration period, a dimensionless integer;

[0097] After the calculation is completed, the system will use the statistical average ( Stored as a reference baseline for user cognitive load;

[0098] This embodiment addresses the issue of significant differences in physiological and behavioral baselines among different users by introducing an initial personalized calibration model. This is particularly important for cross-cultural adaptation, as individuals from different cultural backgrounds, such as foreign language learners, exhibit vastly different stress baselines and expression styles. It avoids using a fixed, universally applicable load threshold, instead generating a load level for each user that reflects their specific load level under normal conditions. This improves the accuracy and reliability of subsequent peak detection, ensuring that the system only reacts to significant deviations relative to the user's own baseline, effectively reducing false alarms or missed alarms caused by individual differences.

[0099] Example 4:

[0100] The spike detection module determines whether a cognitive spike event has occurred and outputs a cognitive spike event flag, including:

[0101] In collaborative mode, the current instantaneous cognitive load index is obtained in real time, and the stored user cognitive load reference baseline is invoked;

[0102] The relative deviation of instantaneous cognitive load is calculated in real time based on the current instantaneous cognitive load index and the user cognitive load reference baseline.

[0103] The instantaneous load relative deviation is compared with a preset disaster trigger threshold;

[0104] A cognitive spike event is determined to have occurred and the cognitive spike event flag is set to 1 if and only if the instantaneous relative load deviation exceeds the disaster trigger threshold.

[0105] Otherwise, set the cognitive peak event flag to 0;

[0106] Based on Example 3, this embodiment describes the specific logic by which the peak detection module determines whether a cognitive peak event has occurred and outputs the cognitive peak event flag after entering the collaborative mode; this logic includes the threshold determination method in the statistical process control.

[0107] In collaborative mode, the module obtains the current instantaneous cognitive load index in real time. ), and invoke the stored user cognitive load reference baseline ( );

[0108] The module is based on the current instantaneous cognitive load index ( ) and user cognitive load reference baseline ( Real-time calculation of instantaneous load relative deviation ( The computational logic is defined as follows:

[0109]

[0110] in, It is a dimensionless value that quantifies the percentage deviation of the current load from the user's normal baseline; its required parameters From the load calculation module, It is a value stored in this module;

[0111] The module will measure the relative deviation of the instantaneous load. Compared with the preset disaster trigger threshold Comparison; disaster trigger threshold It is a key, dimensionless preset parameter; the determination of this threshold has a clear technical basis, for example: by collecting multiple sets of user data in simulated high-risk scenarios and statistically analyzing their performance before task failure. The distribution is determined, and the statistically significant quantile of this distribution is taken, for example, the 95th percentile. The value, for example, calibrated as This means that it is triggered when a learner's transient cognitive load, such as exceeding 180% of their normal baseline during a failed cross-cultural communication session;

[0112] Execute the decision logic based on the comparison result:

[0113] If and only if the instantaneous relative load deviation ( Exceeding the disaster trigger threshold ( When, that is The module determines that a cognitive spike event has occurred and sets the cognitive spike event flag ( Set it to 1;

[0114] Otherwise, that is The module determines that no spike event has occurred and sets the cognitive spike event flag ( Set to 0;

[0115] This embodiment provides a robust and statistically significant peak event determination mechanism; by calculating the relative deviation ( Instead of comparing absolute values, it incorporates personalized baselines. ) and based on statistics, such as the 95th percentile setting of the disaster threshold ( This scheme ensures that the system will only trigger the spike flag when a statistically significant cognitive overload event occurs, which is extremely unlikely to happen under normal conditions. This achieves high sensitivity in capturing real cognitive crises while maintaining high specificity for normal task stress fluctuations, i.e., a low false alarm rate.

[0116] Example 5:

[0117] The dynamic contingency module calculates the instantaneous automation level, including:

[0118] Receive the binary cognitive spike event flag from the spike detection module and call the preset default automation level;

[0119] Use the cognitive spike event flag as a switching factor to suppress or enable the default automation level;

[0120] The instantaneous automation level is determined by multiplying the switching factor by the default automation level;

[0121] This embodiment, based on embodiment 4, calculates the instantaneous automation level using a dynamic contingency module. The specific process of ); this module acts as a logical control gate based on cognitive state;

[0122] This module receives binary cognitive spike event flags from the spike detection module. Its value must be 0 or 1, and the preset default automation level will be invoked. ); Default automation level ( ) is a dimensionless control coefficient ranging between 0 and 1, for example It defines the weight of AI-assisted instructions under normal conditions;

[0123] The core logic of this module is to utilize cognitive spike event flags ( As a switching factor, it suppresses or enables the default automation level. In this embodiment, the switching factor is specifically defined as follows: ;

[0124] The module uses this switching factor ( ) and the default level of automation ( Multiply by , to determine the instantaneous level of automation. The calculation follows a custom binary control logic referred to in this invention as the suppression gate, and its calculation formula is as follows:

[0125]

[0126] in, Instantaneous automation level, dimensionless control coefficient, output to interactive execution module;

[0127] : Cognitive peak event flag, i.e., 0 or 1;

[0128] : The default level of automation, dimensionless coefficient;

[0129] The formula The effect is bistate:

[0130] when Time, i.e., normal state, switching factor The value is 1; at this point, the formula becomes ,Right now The automation level remains at the preset default level.

[0131] when At that time, a cognitive peak was detected, and the switching factor was activated. It is 0; at this time It is instantly forced to zero; this symbolizes that in a cross-cultural crisis, the system abandons all decorative interventions and instead adopts a mode of 100% listening and acceptance.

[0132] This embodiment provides a control switching mechanism that is computationally extremely lightweight, has an instantaneous response, and is logically well-defined; by using The flag acts as a hard-switching factor rather than a smooth transition factor to directly suppress the default level. This ensures that the automation level drops to zero immediately and definitively the moment a crisis is detected, avoiding any delayed or ambiguous transition states and providing deterministic control input for subsequent execution modules to achieve absolute intent fidelity.

[0133] Example 6:

[0134] The interactive execution module outputs the final contingency instructions, including:

[0135] It receives raw operation commands from the user's VR controller in real time, as well as AI-assisted commands from the system's built-in AI-assisted model;

[0136] Invoke the instantaneous automation level from the dynamic contingency module;

[0137] Based on linear interpolation logic, the instantaneous automation level is used as the mixing coefficient to weight and mix the AI-assisted instructions and the original operation instructions to generate and output the final contingency instructions.

[0138] This embodiment, based on embodiment 5, describes how the interactive execution module uses dynamic computation. The value is output as the final contingency instruction. This module is the final fusion arbiter for human-machine commands;

[0139] This module receives two instruction inputs in real time that must have the same data structure and physical dimensions, for example, both being six-degree-of-freedom vectors:

[0140] The user's original operation commands for the VR controller are defined as follows: ;

[0141] The system has built-in AI-assisted models, such as trajectory prediction or anti-shake smoothing models that generate AI-assisted commands, defined as follows: ;

[0142] This module invokes the instantaneous automation level from the dynamic contingency module ( );

[0143] This module is based on linear interpolation logic and uses an instantaneous level of automation. As a mixing coefficient, for and Perform weighted mixing to generate and output the final contingency instruction. This logic innovatively applies the Alpha blending formula from computer graphics to human-computer control arbitration, and the formula it executes is as follows:

[0144]

[0145] in, The final output contingency instruction, such as a vector, is sent to the remote collaboration system.

[0146] Instantaneous automation level, dimensionless scalar;

[0147] AI-assisted commands, such as vectors;

[0148] The weight of the user's original command, a dimensionless scalar;

[0149] User's original operation commands, such as vectors;

[0150] This embodiment employs linear interpolation logic and replaces the mixing coefficients (Alpha values) with those dynamically driven by passive physiological signals. This creates a streaming, adaptive instruction arbitration mechanism; the formula mathematically ensures that the output instruction is a smooth mixture of two valid input sources, enabling fully manual... This means allowing learners complete autonomy, even with incorrect expressions, and AI assistance. This means providing seamless switching between conventional teaching aids, with the sole basis for switching being the user's objective cognitive state.

[0151] Example 7:

[0152] Interactive execution module:

[0153] When the cognitive peak event flag is 0, the instantaneous automation level is equal to the default automation level, and the final contingency instruction is a weighted mixture of AI-assisted instructions and original operation instructions;

[0154] When the cognitive peak event flag is 1, the instantaneous automation level is forced to zero, and the final contingency instruction is instantly collapsed to be equal to the original operation instruction.

[0155] This embodiment further clarifies the two key operating states and their technical effects based on embodiment 6;

[0156] The final arbitration logic of the interactive execution module, driven by the dynamic contingency module, will inevitably present the following two states:

[0157] Normal state, i.e., AI-assisted hybrid:

[0158] When the cognitive peak event flag is 0, that is According to the logic of Example 5, the instantaneous automation level is equal to the default automation level, that is... ;

[0159] The arbitration formula then becomes: ;

[0160] The final contingency instruction is a weighted mixture of AI-assisted instructions and original operation instructions; in this state, the user's operation is assisted by the AI ​​model, such as smoothing or trajectory optimization, thus achieving efficiency gains in human-machine collaboration.

[0161] Cognitive peak state, i.e., absolute intent fidelity:

[0162] When the cognitive peak event flag is 1, that is According to the logic of Example 5, the instantaneous automation level is forcibly reset to zero, that is... ;

[0163] At this moment, the arbitration formula collapses instantaneously: ;

[0164] The formula simplifies to ;

[0165] The final contingency instruction instantly collapses to be equal to the original operation instruction; in this state, the weight of the AI-assisted instruction is forcibly reduced to zero, and any assistance or prediction behavior of the system is instantly cut off, ensuring that the output instruction 100% reproduces the user's original hardware input.

[0166] This bimodal collapse logic perfectly achieves the core technical objective of this invention; it solves the problem that AI assistance may become a fatal intervention in a crisis; it also solves the problem that teaching assistance, such as over-correction, may become a fatal intervention leading to complete communication failure in cross-cultural communication crises; by ensuring that cognitive peaks are detected, i.e. In that instant, the AI ​​assistance immediately fails its protection and collapses back to the pure, original user command, that is... The system enables an instantaneous switch from AI assistance to absolute intent fidelity, returning the highest level of control to the human operator when it is most needed.

[0167] It should be noted that the instantaneous zeroing strategy adopted in this embodiment aims to achieve the highest level of safety redundancy; in specific scenarios with higher requirements for control smoothness, the calculation logic of the dynamic contingency module can also be modified to a smooth transition function, for example, when After changing from 0 to 1, let Within a very short time window The speed is reduced to zero smoothly to ensure a rapid response while mitigating any perceived abrupt changes in control by the user.

[0168] Example 8:

[0169] The data acquisition module is also used for:

[0170] Deploy a non-invasive physiological sensor array to collect raw physiological data streams in real time;

[0171] Simultaneous monitoring of interactive behavior data streams is achieved through VR controller tracking and head-mounted display tracking.

[0172] This embodiment is a supplementary explanation of the specific hardware implementation of the data acquisition module in Embodiment 1;

[0173] To acquire raw physiological data streams, this module deploys a non-invasive physiological sensor array to collect raw physiological data streams in real time. This array is preferably integrated into a VR head-mounted display, wristband, or controller, and may include photoplethysmography (PPG) sensors or electrocardiogram (ECG) electrodes for acquiring HRV indicators, and skin conductance sensors for acquiring GSR indicators. The non-invasive approach ensures high availability of the system in actual operation and does not interfere with the user's main tasks.

[0174] To collect interactive behavior data streams, this module utilizes the hardware built into the VR system, tracking the data streams simultaneously through VR controllers, such as Inside-Out or Lighthouse tracking, head-mounted display (HMD) tracking, HMD position tracking, and the built-in eye-tracking module. Controller tracking provides the kinematic vectors of the operation handles, namely position, rotation, velocity, and acceleration, while HMD tracking provides head posture data and gaze point data.

[0175] This embodiment provides a highly integrated and feasible multimodal data acquisition solution by explicitly using a non-invasive physiological sensor array combined with VR controller / HMD tracking. It utilizes the standard tracking function of existing VR hardware to acquire behavioral data, while only adding lightweight non-invasive physiological sensors, thereby achieving synchronous and real-time acquisition of key data required for cognitive load assessment, namely physiological and behavioral data, with minimal hardware cost and user intrusion.

[0176] To ensure the robustness of the system, the data acquisition module also includes a data pre-validation step. In this step, the system checks the validity of the input data stream, such as whether the physiological signal values ​​are within a reasonable physiological range and whether the behavioral data is continuous. When sensor readings are lost or obvious outliers are detected, the system can use a strategy of interpolating nearby samples or temporarily ignoring the data channel for a short period of time to prevent incorrect input data from causing misjudgments in subsequent modules.

[0177] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal remote collaborative interaction system integrating artificial intelligence and virtual reality, characterized in that, include: The data acquisition module is used to collect users' raw physiological data stream and interactive behavior data stream in real time. The raw physiological data stream includes heart rate variability indicators and skin conductance response indicators, while the interactive behavior data stream includes kinematic vectors, gaze point data, and operation frequency. The load calculation module is used to calculate the standardized instantaneous cognitive load index based on the raw physiological data stream and interactive behavior data stream, and combined with the preset physiological data weight coefficient and interactive behavior data weight coefficient, through an additive weight fusion model. The peak detection module is used to calculate the relative deviation of instantaneous cognitive load based on the instantaneous cognitive load index and the calibrated user cognitive load reference baseline; and based on the comparison result of the relative deviation of instantaneous cognitive load with the preset disaster trigger threshold, to determine whether a cognitive peak event has occurred and output the cognitive peak event flag bit. The dynamic contingency module is used to calculate the instantaneous automation level based on the cognitive peak event flag and the preset default automation level; The interactive execution module is used to fuse and arbitrate AI-assisted instructions and original user operation instructions based on the instantaneous automation level, and output the final contingency instruction. The peak detection module is also used for: In system calibration mode, the instantaneous cognitive load index data stream is continuously collected during the user's execution of standard tasks; Based on the standard definition of the arithmetic mean, calculate the statistical average of the instantaneous cognitive load index data stream within the calibration period; Store the statistical average as a reference baseline for user cognitive load; The spike detection module determines whether a cognitive spike event has occurred and outputs a cognitive spike event flag, including: In collaborative mode, the current instantaneous cognitive load index is obtained in real time, and the stored user cognitive load reference baseline is invoked; The relative deviation of instantaneous cognitive load is calculated in real time based on the current instantaneous cognitive load index and the user cognitive load reference baseline. The instantaneous load relative deviation is compared with a preset disaster trigger threshold; A cognitive spike event is determined to have occurred and the cognitive spike event flag is set to 1 if and only if the instantaneous relative load deviation exceeds the disaster trigger threshold. Otherwise, set the cognitive peak event flag to 0; The dynamic contingency module calculates the instantaneous automation level, including: Receive the binary cognitive spike event flag from the spike detection module and call the preset default automation level; Use the cognitive spike event flag as a switching factor to suppress or enable the default automation level; The instantaneous automation level is determined by multiplying the switching factor by the default automation level; The interactive execution module outputs the final contingency instructions, including: It receives raw operation commands from the user's VR controller in real time, as well as AI-assisted commands from the system's built-in AI-assisted model. Invoke the instantaneous automation level from the dynamic contingency module; Based on linear interpolation logic, the instantaneous automation level is used as the mixing coefficient to weight and mix the AI-assisted instructions and the original operation instructions to generate and output the final contingency instructions. The interactive execution module: When the cognitive peak event flag is 0, the instantaneous automation level is equal to the default automation level, and the final contingency instruction is a weighted mixture of AI-assisted instructions and original operation instructions; When the cognitive peak event flag is set to 1, the instantaneous automation level is forcibly reset to zero, and the final contingency instruction is instantly collapsed to be equal to the original operation instruction.

2. The multimodal remote collaborative interaction system integrating artificial intelligence and virtual reality according to claim 1, characterized in that, The load calculation module calculates the standardized instantaneous cognitive load index, including: The raw physiological data stream is processed to obtain normalized physiological state feature values; The interaction behavior data stream is processed to obtain normalized abnormal interaction behavior feature values; Invoke preset physiological data weighting coefficients and interactive behavior data weighting coefficients; Based on physiological state characteristic values, abnormal interactive behavior characteristic values, physiological data weight coefficients, and interactive behavior data weight coefficients, the instantaneous cognitive load index is calculated using an additive weighted fusion model.

3. The multimodal remote collaborative interaction system integrating artificial intelligence and virtual reality according to claim 1, characterized in that, The data acquisition module is also used for: Deploy a non-invasive physiological sensor array to collect raw physiological data streams in real time; The system tracks interactive behavior data streams synchronously through VR controller tracking and head-mounted display tracking.

Images