A VR teaching immersive interaction instruction recognition method and system

Abstract

The application discloses a VR teaching immersive interactive instruction recognition method and system, relates to the field of VR teaching immersive interactive instruction recognition, and is used for enhancing the immersion and learning effect of VR teaching and comprises the following steps: acquiring hand action information, voice information and specific information of a current VR teaching scene of a user; processing the hand action information to obtain a probability distribution of a gesture instruction intention; processing the voice information to obtain a probability distribution of a voice instruction intention; adjusting the weight of the probability distribution of the gesture instruction intention and the weight of the probability distribution of the voice instruction intention according to the specific information; aligning and fusing the probability distribution of the gesture instruction intention and the probability distribution of the voice instruction intention based on the adjusted weights, calculating a comprehensive intention probability distribution, and determining an instruction intention according to the comprehensive intention probability distribution.

Description

Technical Field

[0001] This invention relates to the field of VR teaching immersive interactive command recognition, and more particularly to a VR teaching immersive interactive command recognition method and system. Background Technology

[0002] Traditional VR teaching immersive interactive command recognition methods often face challenges such as low recognition accuracy, slow system response, and difficulty in effectively integrating multimodal information when processing user gestures and voice commands. These problems are particularly prominent in complex VR teaching scenarios that require users to perform long periods of fine-grained operations and frequent body movements, seriously affecting the user's immersion and learning outcomes. Summary of the Invention

[0003] This application discloses a VR teaching immersive interactive command recognition method and system, which aims to solve the problems in existing VR teaching immersive interactive command recognition technologies, such as low command recognition accuracy, untimely system response, difficulty in effectively integrating multimodal information, and the easy interference of gestures and voice information when users perform long-term fine operations and frequent body movements, resulting in a decline in user immersion and learning effect.

[0004] In a first aspect, this application discloses a VR teaching immersive interactive command recognition method, comprising the following steps: Acquire user hand gestures, voice information, and specific information about the current VR teaching scenario; Hand movement information is processed to obtain the probability distribution of gesture command intent; The voice information is processed to obtain the probability distribution of the voice command intent; Adjust the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent based on specific information. Based on the adjusted weights, the probability distributions of gesture command intent and voice command intent are aligned and fused to calculate the comprehensive intent probability distribution, and the command intent is determined according to the comprehensive intent probability distribution.

[0005] Through this technical solution, this application can effectively integrate instruction information from both gesture and voice modalities, and dynamically adjust the weights of each modality according to the specific information of the VR teaching scenario, thereby improving the accuracy and robustness of instruction recognition, solving the problem of the difficulty in effectively integrating multimodal information in the prior art, and enhancing the immersion and interaction efficiency of VR teaching.

[0006] Furthermore, based on specific information, the weights of the probability distributions of gesture command intentions and voice command intentions are adjusted, including: Acquire minute motion data from the head-mounted display and ambient sound; The instantaneous vibration intensity is obtained by processing the minute motion data. The instantaneous humming sound intensity is obtained by processing the ambient sound. Predict future disturbance intensity based on instantaneous vibration intensity and instantaneous humming intensity; Based on the future interference intensity, the health of gesture information and voice information will be corrected. Based on the type of the current VR teaching task, obtain the preset modal weights; Based on the corrected gesture information health, the corrected voice information health, and the preset modal weights, the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent are adjusted.

[0007] Through this technical solution, this application can predict potential future interference by monitoring the subtle movements of the head-mounted display and ambient sounds in real time, and correct the health status of gestures and voice information accordingly. It can also dynamically adjust modal weights based on the type of teaching task, thereby more accurately reflecting the current interactive environment and user status, further optimizing the accuracy of weight adjustment, and effectively dealing with the complex and ever-changing interference factors in VR teaching.

[0008] Based on the above, this application further proposes to predict future interference intensity based on instantaneous vibration intensity and instantaneous humming intensity, including: Monitor the rate and direction of change of instantaneous vibration intensity and instantaneous humming intensity; Based on the rate and direction of change of instantaneous vibration intensity and instantaneous humming intensity, identify abnormal patterns in instantaneous vibration intensity and instantaneous humming intensity; Predict future interference intensity based on anomalous patterns.

[0009] Through this technical solution, this application can identify potential abnormal patterns by analyzing the dynamic changes in instantaneous vibration intensity and instantaneous humming intensity, thereby predicting possible future interferences earlier and more accurately, providing a forward-looking basis for subsequent health correction and weight adjustment, and effectively improving the system's predictive ability.

[0010] In some preferred embodiments, the health status of gesture information is corrected based on the future intensity of interference, including: Obtain the actual shaking amplitude of key hand points in the image; The comparison results are obtained by comparing the actual jitter amplitude with the future interference intensity; When the comparison results show a deviation within a preset range, a dynamic correction factor is calculated; Combine dynamic correction factors to correct the health status of gesture information.

[0011] Through this technical solution, this application can compare the predicted future interference intensity with the actual hand tremor amplitude, and dynamically calculate the correction factor based on the deviation, thereby more precisely correcting the health of the gesture information, effectively compensating for the inaccurate gesture recognition caused by factors such as head display displacement, and improving the recognition accuracy of gesture commands.

[0012] More specifically, in some implementation plans, the health of voice information is corrected, including: Obtain user physiological data; Based on user physiological data, extract physiological indicators that reflect user fatigue or stress status; Analyze the non-command speech features in user speech signals; Identify abnormal changes in speech patterns based on non-command speech characteristics; Based on abnormal changes in physiological indicators and speech patterns, an internal physiological state influence coefficient is generated. The health of voice information is corrected by combining the influence coefficient of internal physiological state and the intensity of future interference.

[0013] Through this technical solution, this application can comprehensively consider the user's physiological state and the non-command features in the voice signal to generate an internal physiological state influence coefficient. Combined with the intensity of external interference, it can more comprehensively and accurately correct the health of voice information, effectively distinguish between the user's real commands and non-command sounds caused by fatigue or stress, and significantly improve the recognition accuracy of voice commands.

[0014] Preferably, an internal physiological state influence coefficient is generated based on abnormal changes in physiological indicators and speech patterns, including: Identify the type of the current VR teaching task; To obtain the correlation patterns between abnormal changes in physiological indicators and speech patterns related to the current VR teaching task type in the user's historical interaction data; Based on the types and correlation patterns of current VR teaching tasks, determine the mapping relationship between physiological indicators and abnormal changes in speech patterns; Based on the mapping relationship, abnormal changes in physiological indicators and speech patterns are transformed; The internal physiological state influence coefficient is calculated based on the abnormal changes in the converted physiological indicators and the converted speech patterns.

[0015] Through this technical solution, this application can establish a mapping relationship between physiological indicators and abnormal changes in voice patterns by recognizing the current VR teaching task type and utilizing the correlation patterns in the user's historical interaction data. This allows for a more accurate calculation of the internal physiological state influence coefficient, making the assessment of the user's physiological state more personalized and contextualized.

[0016] Furthermore, the correlation patterns between abnormal changes in physiological indicators and speech patterns related to the current VR teaching task type are obtained from historical user interaction data, including: Obtain abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type from the user's historical interaction data; Identify teaching task types in user history interaction data that are similar to the current VR teaching task type; and extract supplementary physiological indicators and abnormal changes in voice patterns from the data of similar teaching task types. Consistency verification was performed on the supplemented physiological indicators and abnormal changes in speech patterns; After the consistency check is passed, the supplemented physiological indicators and abnormal changes in voice patterns will be integrated with the abnormal changes in physiological indicators and voice patterns in the user's historical interaction data that are related to the current VR teaching task type. The integrated data was analyzed to identify the correlation between physiological indicators and abnormal changes in speech patterns.

[0017] This technical solution effectively expands the dataset for analyzing correlation patterns by introducing supplementary data of similar teaching task types and performing consistency checks, thereby improving the generalization and accuracy of correlation patterns and solving the problem of insufficient historical data for a single user.

[0018] Based on the above, this application also proposes to perform consistency verification on the supplementary physiological indicators and abnormal changes in speech patterns, including: Obtain the distribution characteristics of each physiological indicator and abnormal change dimension of speech pattern in the supplementary physiological indicator and speech pattern abnormal change data; Based on the current VR teaching task types, determine the verification standards corresponding to the distribution characteristics; For each dimension of the supplementary physiological indicators and abnormal changes in speech patterns, the matching degree of distribution features is calculated according to the verification criteria. Based on the matching degree calculation results, the heterogeneity dimension in the supplemented physiological indicators and abnormal changes in speech patterns is identified and quantified; the heterogeneity dimension is the dimension with a matching degree lower than the matching degree threshold. Based on the quantification results of the heterogeneity dimension, the supplementary physiological indicators and abnormal changes in speech patterns are adjusted or filtered for consistency.

[0019] Through this technical solution, this application can identify and process heterogeneous dimensions by performing strict consistency checks on supplementary data, ensuring the quality of supplementary data and its consistency with the current task type, avoiding errors introduced by data mismatch, and further improving the reliability of correlation patterns.

[0020] Furthermore, the integrated data is analyzed to identify the correlation between physiological indicators and abnormal changes in speech patterns, including: Construct a multi-layered processing structure; Based on the integrated data and multi-layer processing structure, the correlation between physiological indicators and abnormal changes in speech patterns is identified. Each layer of the processing structure receives the output of the previous layer and performs feature transformation on the output. The last layer of the multi-layer processing structure outputs the correlation between physiological indicators and abnormal changes in speech patterns.

[0021] Through this technical solution, this application can utilize a multi-layer processing structure to perform in-depth analysis and feature transformation on the integrated data, thereby more effectively identifying the complex nonlinear correlation between physiological indicators and abnormal changes in speech patterns, and improving the depth and accuracy of correlation pattern mining.

[0022] Secondly, this application also discloses a VR teaching immersive interactive command recognition system, which includes: The information acquisition module is used to acquire the user's hand gestures, voice information, and specific information about the current VR teaching scene; The gesture processing module is used to process hand motion information to obtain the probability distribution of gesture command intent; The voice processing module is used to process voice information and obtain the probability distribution of the intent of the voice command. The weighting adjustment module is used to adjust the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent based on specific information. The intent determination module is used to align and fuse the probability distribution of gesture command intent with the probability distribution of voice command intent based on the adjusted weights, calculate the comprehensive intent probability distribution, and determine the command intent based on the comprehensive intent probability distribution.

[0023] This application provides a system for implementing the above-mentioned method through this technical solution. Through modular design, it clearly divides the functions of information acquisition, gesture processing, voice processing, weight adjustment and intent determination, making the system structure more reasonable, easier to implement and maintain, and able to work together effectively to achieve accurate recognition of VR teaching immersive interactive commands.

[0024] Beneficial effects This application discloses a VR teaching immersive interactive command recognition method. It acquires the user's hand gesture information, voice information, and specific information of the current VR teaching scene, and processes the gesture and voice information separately to obtain their respective command intent probability distributions. Based on this, the weights of the gesture and voice command intent probability distributions are dynamically adjusted according to the specific information of the VR teaching scene. The two modal probability distributions are then aligned and fused based on the adjusted weights, ultimately calculating the comprehensive intent probability distribution and determining the command intent. This method effectively solves the problems of low command recognition accuracy, untimely system response, and difficulty in effectively integrating multimodal information in existing VR teaching immersive interactive command recognition technologies. By dynamically adjusting modal weights, this application can flexibly allocate the priority of gesture and voice information according to the complexity of the actual teaching scene and changes in the user's state, thus maintaining a high command recognition accuracy even when gesture or voice information is interfered with. For example, when the user's hand movements become shaky due to head display displacement, the system can appropriately reduce the weight of gesture commands and rely more on voice commands; conversely, when environmental noise affects voice recognition, the weight of gesture commands can be increased. This adaptive weight adjustment mechanism significantly improves the robustness and recognition accuracy of the system in complex VR teaching environments, overcoming the limitations of single-modality or fixed-weight fusion methods in existing technologies. It enables users to still obtain a smooth and accurate interactive experience when performing fine operations and frequent body movements, thereby enhancing the immersion and learning effect of VR teaching. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a VR teaching immersive interactive command recognition method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of another VR teaching immersive interactive command recognition method provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a VR teaching immersive interactive command recognition system provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] First, let's introduce the terminology used in this application.

[0029] "Hand movement information" refers to data such as the posture, position, and movement trajectory of a user's hands captured by VR devices (such as cameras built into VR headsets or external hand trackers). This information is the basis for recognizing gesture commands.

[0030] "Voice information" refers to user voice data collected through the built-in microphone of the VR headset or other audio input devices, including command voice, non-command voice, and ambient sounds.

[0031] "Specific information of the VR teaching scenario" refers to the contextual information related to command recognition in the current VR teaching environment, such as the current teaching task type, scene complexity, user physiological state, and environmental interference. This information is crucial for dynamically adjusting the weight of gestures and voice commands.

[0032] "Probability distribution of gesture command intent" and "probability distribution of voice command intent" refer to the various possible command intents and their corresponding probability values ​​output by the recognition models of their respective modalities after making a preliminary judgment on the user's intent. For example, the gesture recognition model may output a probability of 0.7 for "grab", 0.2 for "click", and 0.1 for "move".

[0033] "Weight" refers to the degree of importance assigned to different modalities during the multimodal information fusion process. The higher the weight, the greater the influence of that modality in the final decision.

[0034] "Alignment and fusion" refers to matching the probability distributions of instruction intents from different modalities in terms of time and semantics, and generating a unified comprehensive intent probability distribution through a certain fusion algorithm (such as weighted average, Bayesian fusion, etc.).

[0035] The implementation environment of this application typically includes a VR headset, a hand tracking device (which can be built into or external to the headset), a microphone, and a computing unit (such as a PC or VR all-in-one machine) that runs VR teaching applications and command recognition algorithms.

[0036] The VR teaching immersive interactive command recognition method of this application is based on the intelligent processing and fusion of user hand gesture information, voice information and specific information of the current VR teaching scene in order to accurately determine the command intent.

[0037] The following specific embodiments will provide a detailed introduction and explanation of the VR teaching immersive interactive command recognition method provided in this application.

[0038] Reference Figure 1 This invention provides a VR teaching immersive interactive command recognition method, comprising the following steps: S1 acquires the user's hand gestures, voice information, and specific information about the current VR teaching scene.

[0039] Specifically, the process of "acquiring user hand movement information, voice information, and specific information about the current VR teaching scenario" can be achieved in several ways. For example, hand movement information can be captured in real time using an infrared camera array built into the VR headset. These cameras can track key points of the user's hands and generate three-dimensional coordinate data. Another method is to use an inertial measurement unit (IMU) sensor worn on the user's wrist or fingers to infer hand movements by sensing the acceleration and angular velocity of the hand. Voice information can be collected through a microphone array integrated into the VR headset, which can effectively suppress ambient noise and capture the user's voice commands. Alternatively, external high-sensitivity microphones can be used to obtain clearer voice signals. Specific information about the current VR teaching scenario, such as the type of teaching task and the complexity of the scenario, can be obtained directly from the VR teaching application programming interface (API) or inferred by analyzing the user's interaction history and current state in the VR scenario. For example, when a user enters a virtual laboratory to conduct a chemistry experiment, the system can automatically identify the current task as a "chemistry experiment" and acquire specific information related to that task.

[0040] S2. Process the hand movement information to obtain the probability distribution of the gesture command intention.

[0041] Specifically, a machine learning model-based approach can be used. For example, the acquired hand keypoint sequences or IMU data can be input into a pre-trained recurrent neural network (RNN) or convolutional neural network (CNN) model. This model, trained on a large amount of gesture data, can recognize different gesture patterns and output the probability distribution of each gesture's intended meaning. For instance, when a user makes a "grab" gesture, the model might output a probability of 0.8 for the "grab" intention, 0.15 for the "click" intention, and 0.05 for the "move" intention.

[0042] Another approach is to use rule-based matching. A series of gesture templates are predefined, and the probability of a gesture command intention is determined by comparing the similarity between real-time hand movement information and these templates. For example, when the position and trajectory of key hand points highly match the preset "grab" gesture template, a higher probability of "grab" intention is assigned.

[0043] S3. Process the voice information to obtain the probability distribution of the voice command intent.

[0044] Specifically, speech recognition technology can be used. For example, the acquired speech signal is preprocessed, including noise reduction and endpoint detection, and then input into an acoustic model and a language model. The acoustic model is responsible for converting the speech signal into a phoneme sequence, while the language model, based on the phoneme sequence and a pre-defined vocabulary, identifies possible command words or phrases and outputs the probability distribution of each speech command intent. For example, when a user says "grab," the model might output a probability of 0.9 for the "grab" intent, 0.08 for the "select" intent, and 0.02 for the "place" intent.

[0045] Another approach is to use keyword recognition technology, which involves pre-setting a set of core instruction keywords. When the voice signal contains these keywords, it is recognized as the corresponding instruction intent, and a corresponding probability is assigned based on the matching degree.

[0046] S4. Adjust the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent based on specific information.

[0047] Specifically, the weights of different modalities can be dynamically adjusted based on the context of the current VR teaching scenario. For example, when the system recognizes that the current teaching task is a virtual assembly task requiring fine hand manipulation, the weight of the probability distribution of gesture commands can be appropriately increased to ensure the priority of gesture commands. Conversely, when the teaching task is a virtual explanation task requiring a large amount of voice interaction, the weight of the probability distribution of voice commands can be increased. Specific information may also include the degree of environmental interference and user fatigue. For example, when there is a lot of environmental noise, the weight of voice commands can be reduced, and more reliance can be placed on gesture commands. The weight adjustment can be done through preset rules or through adaptive learning and adjustment based on historical data using machine learning models.

[0048] S5. Based on the adjusted weights, align and fuse the probability distribution of gesture command intent with the probability distribution of voice command intent, calculate the comprehensive intent probability distribution, and determine the command intent based on the comprehensive intent probability distribution.

[0049] A weighted average method can be used to sum the probability distributions of the adjusted gesture command intent and the voice command intent, resulting in a combined intent probability distribution. Specifically, if the gesture intent probability distribution is P_gesture, the voice intent probability distribution is P_voice, the adjusted gesture weight is W_gesture, and the voice weight is W_voice, then the combined intent probability distribution P_combined = W_gesture P_gesture + W_voice P_voice.

[0050] After calculating the overall intent probability distribution, the system selects the intent with the highest probability as the final instruction intent. For example, if the probability of "grab" is 0.75, the probability of "click" is 0.2, and the probability of "move" is 0.05 in the overall intent probability distribution, then the final instruction intent is determined to be "grab".

[0051] The VR teaching immersive interactive command recognition method of this application works by using intelligent perception, processing and dynamic fusion of multimodal information to overcome the limitations of single-modal recognition, thereby achieving high-accuracy command recognition in complex VR teaching environments.

[0052] Specifically, when a user interacts in a VR teaching scenario, the system simultaneously acquires the user's hand gestures, voice information, and specific information about the current VR teaching scenario. The hand gesture information is sent to the gesture processing module, which analyzes hand posture and movement trajectory using advanced algorithms (such as deep learning models) to output a probability distribution of the gesture command intent. Simultaneously, the voice information is sent to the voice processing module, which parses the user's voice commands using speech recognition technology and outputs a probability distribution of the voice command intent.

[0053] The key is that the system doesn't simply superimpose the recognition results of these two modalities. Instead, it dynamically adjusts the weights of the probability distributions for gesture and voice commands based on the specific information of the current VR teaching scenario. For example, in virtual experiments requiring fine hand movements, the weight of gesture commands might be increased; while in explanations with frequent voice interaction, the weight of voice commands might be increased. This dynamic weight adjustment mechanism allows the system to intelligently prioritize the more reliable and important interaction modality based on the real-time context, effectively addressing the problem of declining quality of single-modal information due to environmental interference or changes in user state.

[0054] After the weights are adjusted, the system aligns and fuses the probability distributions of gesture commands and voice commands based on these adjusted weights. Alignment ensures the temporal or semantic synchronization of information from different modalities, while fusion integrates these weighted probability distributions into a unified, comprehensive intent probability distribution. Finally, the system determines the user's most likely command intent based on this comprehensive intent probability distribution.

[0055] In this way, the method of this application can effectively solve the problems of low accuracy in command recognition, slow system response, and difficulty in effectively integrating multimodal information in traditional VR teaching. For example, when users are performing fine operations, even if the voice is interfered with by environmental noise, the system can ensure accurate recognition of gesture commands by increasing the weight of gesture commands; conversely, when hand movements are shaky due to head-mounted display displacement, the system can rely more on voice commands. This collaborative working mechanism allows various technical features to work together to form a robust and efficient command recognition system, greatly improving the immersion and learning efficiency of VR teaching.

[0056] Traditional command recognition methods often employ fixed weights or simple fusion strategies to process multimodal information. This leads to a significant impact on overall recognition performance in complex VR teaching environments when the quality of one modality deteriorates. For example, voice command recognition rates drop sharply in noisy environments; gesture command recognition also becomes inaccurate when hand tracking is unstable. These problems directly result in a poor user experience and reduced learning efficiency.

[0057] The core innovation of this application lies in the introduction of a dynamic weight adjustment mechanism that "adjusts the weights of the probability distributions of gesture command intentions and voice command intentions based on specific information." This mechanism enables the system to intelligently assess the reliability of different modalities based on contextual information of the current VR teaching scenario, such as the type of teaching task, the degree of environmental interference, and the user's physiological state, and adjust their influence in the fusion process in real time. For example, in virtual assembly tasks requiring fine hand manipulation, the system can automatically increase the weight of gesture commands, prioritizing gesture information even if voice commands are slightly interfered with. In virtual explanation sessions where voice interaction is the primary mode, the system increases the weight of voice commands to ensure accurate recognition of them.

[0058] This dynamic adjustment strategy enables the method of this application to adaptively cope with various complex and changing situations in VR teaching environments, significantly improving the accuracy and robustness of command recognition. By aligning and fusing the probability distribution of gesture command intent with the probability distribution of voice command intent, and calculating the comprehensive intent probability distribution based on the adjusted weights, this application can more accurately capture the user's true intent, thereby achieving a smoother and more natural interactive experience. This intelligent multimodal fusion method effectively solves the problem of effectively integrating multimodal information in existing technologies, providing a more immersive and efficient interactive method for VR teaching.

[0059] In some of the embodiments described above in this application, the weights of the probability distributions of gesture command intentions and voice command intentions are adjusted based on specific information from the VR teaching scenario. However, in actual implementation, relying solely on static or preset specific information for weight adjustment may not adequately address dynamically changing interference factors in the VR teaching environment, such as environmental noise, device vibration, or changes in the user's own physiological state. These factors may lead to a decrease in the quality of gesture or voice information, thereby affecting the accuracy and robustness of command recognition.

[0060] In response, one possible design is as follows: Figure 2 As shown, in order to adjust the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent based on specific information, this application may further include the following steps: S101: Acquire minute motion data of the head-mounted display and ambient sound.

[0061] Specifically, acquiring minute motion data from the headset refers to real-time collection of data on the headset's subtle movements, jitters, or vibrations in space using the inertial measurement unit (IMU) or other sensors inside the VR headset. This minute motion data can reflect vibrations in the external environment, the user's body movements, or the stability of the device itself. Ambient sound refers to non-command sounds such as background noise, human voices, and equipment operating sounds in the VR teaching scene, collected through the VR headset or external microphones.

[0062] S102. Based on the minute motion data, the instantaneous vibration intensity is obtained.

[0063] Specifically, the collected minute motion data from the head-mounted display can be filtered, denoised, and feature-extracted. For example, the root mean square values ​​of acceleration and angular velocity, or energy within a specific frequency range, can be calculated to quantify the vibration level at the current moment. Higher instantaneous vibration intensity usually indicates greater external interference or user instability.

[0064] S103. Based on the ambient sound, the instantaneous humming intensity is obtained.

[0065] Specifically, based on ambient sound, the instantaneous humming intensity is obtained. This can be understood as performing spectrum analysis, noise identification, and energy calculation on the collected ambient sound to quantify the intensity of background noise or specific interference sounds at the current moment. For example, it can identify and quantify the humming sound of a VR device fan, external construction noise, etc.

[0066] S104. Based on instantaneous vibration intensity and instantaneous humming intensity, predict future interference intensity.

[0067] Specifically, it can monitor the rate and direction of change of instantaneous vibration intensity and instantaneous humming intensity; identify abnormal patterns in instantaneous vibration intensity and instantaneous humming intensity based on the rate and direction of change of instantaneous vibration intensity and instantaneous humming intensity; and predict future interference intensity based on abnormal patterns.

[0068] The identification of abnormal patterns in instantaneous vibration and humming intensity based on their rate and direction of change refers to marking an anomaly when the monitored rate or direction of change significantly deviates from the normal or expected pattern. Abnormal patterns may include, but are not limited to: a sharp increase or decrease in intensity value within a very short period, a sustained exceedance of a preset safety threshold, or the presence of periodic but unexpected fluctuations. The identification process can be implemented using statistical methods (e.g., Z-score analysis, interquartile range (IQR) method), machine learning models (e.g., anomaly detection algorithms), or expert systems based on predefined rules.

[0069] In practical applications, predicting future interference intensity based on anomalous patterns means that once the system successfully identifies an anomalous pattern, it infers the potential level of interference intensity or its evolution trend over a future period based on the characteristics of that anomalous pattern and its correlation patterns in historical data. For example, if vibration intensity is detected to be rising at an abnormally high rate, it can be predicted that future interference intensity will increase significantly. Prediction can be achieved using time series analysis models (such as ARIMA models or Long Short-Term Memory networks, LSTM) or matching and inference methods based on anomalous pattern libraries.

[0070] Thus, by monitoring the rate and direction of change of instantaneous vibration intensity and instantaneous humming intensity, the dynamic evolution of interference signals can be captured, rather than relying solely on static values ​​at a single moment. It is precisely this in-depth analysis of these dynamic characteristics that enables the system to further identify potential abnormal patterns, such as sudden external impacts or continuous noise accumulation. By identifying these abnormal patterns, the system can predict the intensity of future interference earlier and more accurately, thereby providing a more reliable and forward-looking basis for subsequent corrections to the health of gesture and voice information. This predictive mechanism based on dynamic trends and abnormal patterns significantly improves the perception and prediction accuracy of changes in environmental interference.

[0071] In some preferred embodiments, a specific example is given below. Suppose that in a VR teaching scenario, a user is performing a detailed surgical simulation task. The system continuously monitors the minute motion data of the headset and ambient sound, and calculates the instantaneous vibration intensity and instantaneous humming sound intensity.

[0072] Specifically, the system records instantaneous vibration intensity and instantaneous hum intensity every 100 milliseconds. By comparing consecutive records, the system calculates the rate of change (e.g., the amount of increase or decrease per second) and direction of change (upward or downward) of these intensity values. For example, if the system detects a sharp increase in instantaneous vibration intensity from 0.1G to 0.5G within the past second, with the direction of change continuously upward, this will be identified as an anomalous pattern, indicating a possible external impact or severe shaking of the user's body. Similarly, if the instantaneous hum intensity of ambient sound increases from 40dB to 70dB within a short period, accompanied by a continuous upward trend, this may be identified as another anomalous pattern, indicating a sudden increase in ambient noise.

[0073] Once these abnormal patterns are identified, the system predicts that the interference intensity will reach a "high" level within the next 5 seconds based on preset rules or a trained model. For example, the system may know from historical data that when the vibration intensity continuously increases at a rate of 0.4 G / s, it will typically reach its peak within the next few seconds. Based on this prediction, the system will preemptively adjust the health status of gesture and voice information; for example, by lowering the health status of gesture information and appropriately lowering the health status of voice information, to avoid misinterpreting user commands when actual interference occurs. In this way, even before the interference fully manifests, the system can prepare in advance to ensure the accuracy of command recognition.

[0074] S105. Based on the future interference intensity, correct the health of gesture information and the health of voice information.

[0075] Gesture health can be understood as an assessment of the quality of hand movement information, such as the stability of hand keypoint tracking, the degree of tremor, and occlusion. Speech health can be understood as an assessment of the quality of speech information, such as signal-to-noise ratio, speech clarity, and speech rate stability. Higher future interference intensity generally indicates lower health for both gesture and speech information, requiring corresponding corrections, such as lowering their initial health scores.

[0076] S106. Obtain the preset modal weights based on the type of the current VR teaching task.

[0077] Specifically, the type of the current VR teaching task can be determined from the first mapping relationship, and the weights with the mapping relationship can be used as preset modal weights.

[0078] For example, gesture modality may be more important in fine motor tasks, while voice modality may be more important in oral reporting tasks. Preset modality weights are pre-configured weight values ​​that reflect the importance of different modalities based on the task type.

[0079] S107. Based on the corrected gesture information health, the corrected voice information health, and the preset modal weights, adjust the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent.

[0080] For example, when the health status of the corrected gesture information is less than the preset first health status threshold, the weight of the probability distribution of the gesture command intent is reduced by a preset step size, and the weight of the probability distribution of the voice command intent is increased by a preset step size.

[0081] When the corrected voice information health is less than the preset second health threshold, the weight of the probability distribution of the voice command intent is reduced by a preset step size, and the weight of the probability distribution of the gesture command intent is increased by a preset step size.

[0082] This mechanism avoids instruction recognition failures caused by a serious mismatch between preset weights and actual information quality. Especially in high-precision and high-time-sensitivity tasks, it ensures that the system can still accurately and quickly understand user intent in complex and ever-changing environments.

[0083] This application's solution dynamically captures external interference information in VR teaching scenarios by introducing real-time perception of the head-mounted display's minute motion data and ambient sound. By processing this data, instantaneous vibration and humming intensity are obtained, and future interference intensity is further predicted, providing a forward-looking basis for evaluating user input quality. Based on this predicted interference intensity, the current health status of gesture and voice information can be corrected, making the evaluation of input modality quality more accurate and real-time. Simultaneously, by combining the preset modal weights obtained from the type of the current VR teaching task, it is ensured that weight adjustments not only consider the real-time environment and user state but also take into account the characteristics and requirements of the task itself. This multi-dimensional, dynamic weight adjustment mechanism allows the weights of the probability distributions of gesture and voice command intentions to more accurately reflect the current situation, thus enabling a more reasonable allocation of the contribution of different modalities during subsequent alignment and fusion processes, effectively overcoming the limitations of relying solely on static, specific information for weight adjustment.

[0084] Through the above technical solution, this application enables adaptive and refined adjustment of modal weights in the VR teaching immersive interactive command recognition method. This solution, by real-time perception and prediction of environmental interference, combined with user input modality health assessment and task type preferences, allows the modal weights to dynamically adapt to complex VR teaching environments and changes in user state. This significantly improves the accuracy and robustness of command recognition, especially in the presence of external interference or poor user condition, effectively avoiding command misjudgments caused by deterioration in modal information quality, thereby enhancing the immersiveness of VR teaching and the user interaction experience.

[0085] In some preferred embodiments, it is assumed that a user is conducting a VR surgical simulation teaching task. In this task, the user needs to precisely operate virtual surgical instruments through gestures and issue commands via voice (such as "zoom in" or "switch tools").

[0086] During surgical simulation, if the VR headset detects continuous, high-frequency vibrations in minute motion data (e.g., slight vibrations in the user's real-world environment) and a continuous humming sound in the ambient noise (e.g., loud air conditioning noise), the system processes this data to obtain high instantaneous vibration and humming sound intensities. Based on these intensities, the system predicts that future interference intensity will increase.

[0087] At this point, the system will adjust the health of gesture information (e.g., reduce the confidence of gesture tracking) and the health of speech information (e.g., reduce the clarity score of speech recognition) based on the predicted future interference intensity. Simultaneously, since this is a surgical simulation task, the system will acquire preset modal weights, which may assign higher initial weights to the gesture modality because surgical procedures require more precise gestures.

[0088] Ultimately, the system dynamically adjusts the weights of the probability distributions for gesture and voice command intentions based on the corrected gesture health, corrected voice health, and preset modal weights. For example, if gesture health significantly decreases due to vibration, even if the task type is gesture-oriented, the system will appropriately reduce the weight of the gesture modality and may slightly increase the weight of the voice modality (if voice health is relatively good) to ensure optimized overall accuracy of command recognition under the current interference environment. In this way, even under adverse environmental conditions, user commands can be recognized more accurately, ensuring the effectiveness and safety of VR teaching.

[0089] In some embodiments described above, this application proposes correcting gesture information health based on predicted future interference intensity. However, in practical applications, relying solely on predicted future interference intensity may not accurately reflect the actual hand tremors of the user. For example, when a user's physiological state (such as slight fatigue) or operating habits cause slight hand tremors, these tremors may not be entirely caused by external environmental interference, but they can still affect the accuracy of gesture recognition. If this problem is not addressed, the weighting of the probability distribution of gesture command intent may not be precise enough, thus affecting the accuracy of VR teaching immersive interactive command recognition and user experience. Therefore, this application further proposes a more refined gesture information health correction method. By acquiring the actual tremor amplitude of key hand points in the image and comparing it with the future interference intensity, a correction factor is dynamically calculated to more accurately and robustly correct gesture information health.

[0090] In response, this application further proposes the above-mentioned method of correcting the health of gesture information based on future interference intensity, including: S201. Obtain the actual shaking amplitude of the hand key points in the image.

[0091] Specifically, acquiring the actual jitter amplitude of key hand points in an image refers to identifying and tracking key points of the user's hand (such as fingertips, knuckles, wrists, etc.) by capturing images or depth information of the user's hand through a VR headset or external sensors. By analyzing the positional changes of these key points in consecutive frames, the actual jitter amplitude in space or the image plane can be calculated. This jitter amplitude can be quantified using metrics such as root mean square error, maximum displacement difference, or jitter frequency within a specific time window.

[0092] S202. Compare the actual jitter amplitude with the future interference intensity to obtain the comparison results.

[0093] Specifically, this can be understood as comparing the actual hand tremor amplitude calculated in real-time or near real-time with the previously predicted future interference intensity. The future interference intensity can be a quantified value representing the degree of impact of expected environmental interference on gesture recognition. The comparison result can be a difference, ratio, or some composite index used to measure the difference between the actual tremor and the predicted interference. For example, the absolute difference or relative difference between the actual tremor amplitude and the future interference intensity can be calculated.

[0094] S203. When the comparison results have a deviation within a preset range, calculate the dynamic correction factor.

[0095] The preset range can be set based on factors such as the system's tolerance to jitter, the robustness of the gesture recognition algorithm, and user experience requirements. When the difference between the actual jitter and the predicted interference exceeds this preset range, it indicates that relying solely on the predicted interference for correction may be insufficient or inaccurate. In this case, a dynamic correction factor needs to be introduced. This dynamic correction factor can be a multiplicative or additive factor, and its magnitude and direction depend on the magnitude and nature of the deviation. For example, if the actual jitter is much greater than the predicted interference, the dynamic correction factor will increase to further reduce the health of the gesture information; conversely, if the actual jitter is much less than the predicted interference, the dynamic correction factor will decrease to avoid over-correction. The dynamic correction factor can be calculated using a preset lookup table, a function based on the deviation value (such as the sigmoid function or a linear function), or a machine learning model.

[0096] S204. Combine dynamic correction factors to correct the health status of gesture information.

[0097] Gesture information health is a comprehensive indicator reflecting the reliability and usability of current gesture information. After obtaining the dynamic correction factor, it is combined with the gesture information health initially corrected based on future interference intensity. This combination can be achieved by directly applying the dynamic correction factor to the gesture information health, such as through multiplication or addition, or by using it as a parameter to adjust the correction magnitude of the gesture information health. In this way, gesture information health can more accurately reflect the actual state of the user's hand movements, thus providing a more reliable basis for adjusting the weights of the probability distribution of subsequent gesture command intentions.

[0098] This application's solution addresses the limitations of relying solely on predicting future interference intensity to correct gesture health by introducing a real-time monitoring and comparison mechanism of the actual shaking amplitude of key hand points. Specifically, when the system predicts future interference intensity, it initially corrects the gesture health. However, to ensure the accuracy of the correction, this solution further acquires the actual shaking amplitude of key hand points in the image. By comparing the actual shaking amplitude with the predicted future interference intensity, the deviation between the two can be identified. This deviation may stem from imperfections in the prediction model or from minor hand tremors caused by the user's physiological or psychological state (such as fatigue or tension), factors that are difficult to fully capture with simple prediction of future interference intensity. When the comparison results show a deviation exceeding a preset range, the system calculates a dynamic correction factor. This dynamic correction factor can further adjust the initially corrected gesture health based on the difference between the actual shaking and the predicted interference, thereby enabling the gesture health to more accurately and precisely reflect the true reliability of the current gesture information. This ensures that gesture command recognition is more robust and accurate in complex or dynamically changing VR teaching environments.

[0099] Through the above technical solution, this application overcomes the inaccuracies of traditional methods that rely solely on predicting interference intensity to correct the health status of gesture information. By acquiring the actual tremor amplitude of key hand points in real time and comparing it with the predicted future interference intensity, this solution can more comprehensively assess the true health status of gesture information, especially when hand tremors are caused by user factors, enabling timely detection and compensation. The introduction of a dynamic correction factor makes the correction process for gesture information health status more adaptive and accurate, avoiding over-correction or under-correction. Therefore, it significantly improves the accuracy and robustness of VR teaching immersive interactive command recognition, providing users with a smoother and more natural interactive experience, effectively enhancing teaching effectiveness and user satisfaction.

[0100] In some preferred embodiments, a specific example is given below. Suppose that in a VR surgical simulation teaching scenario, students need to precisely operate virtual surgical instruments. The system first predicts that the future interference intensity is moderate based on the subtle motion data of the head-mounted display and ambient sound, and accordingly makes an initial correction to the health status of the gesture information. However, when students are performing delicate operations, due to their high concentration or slight tension, their hands may exhibit minute physiological tremors that are difficult to detect with the naked eye.

[0101] At this point, the solution acquires the actual jitter amplitude of key points on the student's hand (e.g., the tip of a virtual scalpel) in the image in real time. Assume the system detects that the actual jitter amplitude is slightly higher than the jitter level corresponding to the predicted future interference intensity. Through comparison, it is found that there is a deviation between the actual jitter amplitude and the future interference intensity that exceeds a preset tolerance range. For example, the jitter amplitude corresponding to the predicted interference intensity is 0.5 pixels, while the actual detected jitter amplitude is 0.8 pixels, a deviation of 0.3 pixels, exceeding the preset range of 0.1 pixels.

[0102] Based on this deviation, the system calculates a dynamic correction factor. For example, based on the magnitude of the deviation, a preset function might calculate a dynamic correction factor of 0.9. This dynamic correction factor is then incorporated into the correction process for gesture health. If the initial corrected gesture health is 0.7, then after incorporating the dynamic correction factor, the final gesture health may be further corrected to 0.7. 0.9 = 0.63. This lower gesture health score prompts the system to assign lower weights to gesture modalities when fusing the probability distribution of gesture command intent, or to use stricter matching thresholds when recognizing gesture commands, thereby avoiding misidentifying unstable gestures as valid commands. In this way, even when students' hands have physiological tremors, the system can more accurately assess the reliability of gesture information, ensuring that only stable and clear commands are recognized, thus guaranteeing the accuracy and safety of VR surgical simulations.

[0103] In some embodiments described above, when adjusting the weights of the probability distributions of gesture command intentions and voice command intentions, the health of the voice information is corrected based on the intensity of future interference. However, in actual VR teaching environments, the quality of a user's voice input is affected not only by external environmental interference but also by the user's internal physiological state (such as fatigue or stress). If only external interference is considered while ignoring the potential impact of the user's internal state on voice quality, the correction of the voice information health may not be comprehensive or accurate enough, thus affecting the robustness of command intention recognition.

[0104] In response, this application further proposes a method for correcting the health of voice information, which includes: S301. Obtain user physiological data.

[0105] Specifically, acquiring user physiological data refers to collecting various physiological signals from users in real time through wearable devices or VR headsets with built-in sensors, such as heart rate, heart rate variability, skin conductance, eye movement data, and brain waves.

[0106] S302. Based on user physiological data, extract physiological indicators that reflect the user's fatigue or stress state.

[0107] Specifically, indicators that can quantify a user's fatigue or stress level can be extracted from their physiological data. For example, an increased heart rate and decreased heart rate variability may indicate stress; increased blinking frequency and pupil dilation in eye movements may indicate fatigue; fluctuations in skin conductance may be related to emotional agitation. These physiological indicators provide objective evidence for assessing a user's internal state.

[0108] S303. Analyze the non-command speech features in the user's speech signal.

[0109] Specifically, acoustic analysis can be performed on the voice signals emitted by users during VR teaching, but the focus is not on the content of the instructions themselves, but rather on the accompanying characteristics of the voice. For example, the fundamental frequency, volume, speech rate, pauses, timbre, and emotional tone of the voice can be analyzed. These characteristics can reflect the user's emotions, clarity, or level of fatigue when expressing instructions.

[0110] S304. Identify abnormal changes in speech patterns based on non-command speech characteristics.

[0111] Specifically, the non-command speech features can be compared with the user's baseline speech pattern or speech pattern under normal conditions to detect whether there are significant deviations. For example, sudden increases or decreases in speech rate, abnormal increases or decreases in volume, excessive pauses or stuttering can all be identified as abnormal changes in speech patterns, indicating that the user may be in a state of fatigue, tension, or emotional instability.

[0112] S305. Generate an internal physiological state influence coefficient based on abnormal changes in physiological indicators and speech patterns.

[0113] Specifically, an internal physiological state influence coefficient can be generated based on the deviation of physiological indicators, non-command speech features, and preset mapping relationships.

[0114] The preset mapping relationship includes the mapping relationship between the deviation range of different physiological indicators and different non-command speech features, and the influence coefficient of different internal physiological states.

[0115] S306. Combining the internal physiological state influence coefficient and future interference intensity, the health of voice information is corrected.

[0116] Specifically, the internal physiological state influence coefficient generated above can be weighted or fused with the future interference intensity (predicted from external environmental factors) to obtain a more comprehensive and accurate voice information health score. For example, when the internal physiological state influence coefficient is high, even if the external interference intensity is low, the voice information health score may be appropriately lowered to reflect the decline in the reliability of the user's voice input.

[0117] This application's solution overcomes the shortcomings of relying solely on external environmental interference to correct voice information health by incorporating consideration of the user's internal physiological state. Specifically, by acquiring user physiological data and extracting physiological indicators, the user's fatigue or stress state can be objectively quantified. Simultaneously, by analyzing non-command speech features in the user's speech signal and identifying abnormal changes in speech patterns, the user's emotions and clarity of expression can be reflected in the speech itself. Integrating these two types of information generates an internal physiological state influence coefficient, which can more comprehensively assess the inherent reliability of the user's voice input. Therefore, when correcting voice information health, not only external environmental interference is considered, but also the user's own physiological and psychological factors are incorporated, making the assessment of voice information health more accurate and providing more reliable voice modal weights for subsequent command intent recognition.

[0118] Through the above technical solution, this application can more comprehensively and accurately assess the health of user voice input, effectively avoiding voice command recognition errors or efficiency decline caused by user fatigue, stress, or other internal physiological states. Compared with solutions that only consider external interference, this application can better adapt to individual user differences and real-time state changes, improving the robustness and accuracy of VR teaching immersive interactive command recognition, thereby providing users with a smoother and more natural interactive experience and enhancing teaching effectiveness.

[0119] In some preferred embodiments, it is assumed that a user is undergoing a highly complex VR surgical simulation training. During the simulation, the system acquires the user's blink frequency and pupil size data through the eye-tracking sensor built into the VR headset, and simultaneously acquires the user's heart rate and skin conductance data through a smart bracelet. These constitute the user's physiological data. Based on this data, the system extracts physiological indicators such as a continuously increasing heart rate, increased blink frequency, and slight pupil dilation, which are judged to reflect that the user is in a state of stress.

[0120] Meanwhile, the system continuously analyzes the user's voice signals when issuing commands. For example, when a user attempts to issue the command "cut the stitches," their speech rate noticeably increases, their pitch rises slightly, and there are brief pauses before and after the command. These non-command voice features are identified as abnormal changes in voice patterns.

[0121] Based on the abnormal changes in physiological indicators and speech patterns under stress, the system calculates a high internal physiological state influence coefficient. For example, this coefficient is quantified as 0.7 (range 0-1, higher values ​​indicate greater influence). Even if the future interference intensity from ambient sounds in the VR teaching scenario (such as operating room background noise) is low, the system will combine this high internal physiological state influence coefficient with the future interference intensity to correct the speech information health score. For example, if the original speech information health score is 0.9, the future interference intensity causes it to decrease by 0.1, and the internal physiological state influence coefficient causes it to decrease by an additional 0.2, then the final corrected speech information health score may become 0.6. This lower health score allows the system to appropriately reduce the weight of speech command intent when integrating gestures and speech commands, or prompt the user to confirm the command, thereby avoiding misrecognition caused by unclear speech expression under stress, ensuring the accuracy and safety of the teaching process.

[0122] In some of the embodiments described above in this application, a scheme for generating internal physiological state influence coefficients based on abnormal changes in physiological indicators and speech patterns was proposed. However, in practical applications, if coefficients are generated solely based on instantaneous or general data, it may not adequately consider the differences in the impact of different VR teaching task types on users' physiological and speech states, and may also ignore the cumulative effect of users' individual historical behavioral patterns. This results in insufficient accuracy and robustness of the generated internal physiological state influence coefficients, thereby affecting the correction effect on speech information health. To address this, this application further proposes a more refined and personalized method for generating internal physiological state influence coefficients. By introducing task type, historical interaction data, and correlation patterns, the accuracy and adaptability of coefficient generation are improved.

[0123] In this regard, this application further proposes that the steps for generating the internal physiological state influence coefficient based on the abnormal changes in physiological indicators and speech patterns include: S401. Identify the current VR teaching task type.

[0124] Specifically, identifying the current VR teaching task type means that the system automatically or through user input determines the category of the ongoing VR teaching activity, such as surgical simulation, language learning, historical scene roaming, or physics experiment. This step aims to provide important contextual information for subsequent data processing.

[0125] S402. Obtain the correlation patterns between abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type in the user's historical interaction data.

[0126] The process of identifying correlations between physiological indicators and abnormal changes in speech patterns related to the current VR teaching task type within historical user interaction data can be understood as the system extracting historical data similar to or identical to the current task type from the user's past VR teaching interaction records. This historical data includes abnormal changes in the user's physiological indicators (such as heart rate, skin conductance, and eye movement data) and speech patterns (such as speech rate, pitch, pauses, and non-command vocalizations) under similar tasks, and analyzing the inherent connections and patterns of change among these data. The aim is to establish a personalized user behavior baseline and an abnormal pattern recognition model.

[0127] S403. Based on the type and correlation patterns of the current VR teaching tasks, determine the mapping relationship between physiological indicators and abnormal changes in speech patterns.

[0128] In practical applications, determining the mapping relationship between physiological indicators and abnormal changes in speech patterns, based on the type and correlation patterns of the current VR teaching task, refers to constructing a function or model that transforms real-time physiological indicators and abnormal changes in speech patterns into internal physiological state influence coefficients, based on the identified task type and correlation patterns learned from historical data. For example, in a high-pressure simulation task, a slight increase in heart rate may be considered a normal response, while in a relaxed learning task, it may be considered a stress response. This mapping relationship is dynamic and personalized.

[0129] S404. Based on the mapping relationship, convert the abnormal change data of physiological indicators and voice patterns.

[0130] Furthermore, based on the mapping relationship, the abnormal change data of physiological indicators and voice patterns are transformed. This means that the real-time acquired user physiological data and abnormal change data of voice patterns are input into the above-determined mapping relationship and processed by standardization, normalization or feature extraction, so that they can be used for subsequent coefficient calculation.

[0131] S405. Calculate the internal physiological state influence coefficient based on the abnormal changes in the converted physiological indicators and the converted speech patterns.

[0132] Specifically, an internal physiological state influence coefficient can be generated based on the deviation of the converted physiological indicators, the converted non-command speech features, and the preset influence coefficient mapping relationship.

[0133] The preset influence coefficient mapping relationship includes the mapping relationship between the deviation of different physiological indicators and different non-command speech features and the influence coefficient of different internal physiological states.

[0134] This application's solution first identifies the current VR teaching task type, providing crucial contextual information for subsequent data analysis. Based on this, the system can accurately filter abnormal changes in physiological indicators and speech patterns related to the current task type from massive amounts of historical user interaction data, and learn and extract the correlation patterns between these data. It is precisely this personalized learning based on historical data and task type that enables the system to dynamically determine the mapping relationship between abnormal changes in physiological indicators and speech patterns according to the type of the current VR teaching task and the acquired correlation patterns. This mapping relationship is no longer static or universal, but adaptable to different task scenarios and individual user differences. Subsequently, by transforming real-time abnormal changes in physiological indicators and speech patterns according to this mapping relationship, the consistency and effectiveness of the input data and the model are ensured. Finally, based on these transformed data, the calculated internal physiological state influence coefficient can more accurately and precisely reflect the user's true physiological and psychological state in a specific VR teaching scenario, thus overcoming the potential problems of insufficient generalization and lack of personalization in traditional methods.

[0135] Through the aforementioned technical solution, this application significantly improves the accuracy and personalization of the generated internal physiological state influence coefficient. By fully considering the types of current VR teaching tasks and the correlation between physiological indicators and abnormal changes in voice patterns in users' historical interaction data, the generated internal physiological state influence coefficient can more accurately reflect the user's fatigue or stress state in specific situations. This refined coefficient generation mechanism avoids misjudgments caused by differences in task type or individual differences, thus making the correction of voice information health status more reasonable and effective. Ultimately, this helps the system more accurately identify the user's command intent, especially when the user's physiological or psychological state fluctuates, significantly improving the robustness of VR teaching immersive interactive command recognition and user experience.

[0136] In some preferred embodiments, a specific example is given below. Suppose a user is performing a challenging VR surgical simulation teaching task. The system first identifies the current task type as "surgical simulation." Subsequently, the system extracts physiological indicators such as heart rate, skin conductance, and eye movement trajectory, as well as abnormal changes in speech patterns such as speech rate, tone, pauses, and stuttering, from the user's historical interaction data of all past surgical simulation tasks. By analyzing this historical data, the system learns that in such high-pressure tasks, when users feel increased stress, their heart rate typically increases by 10-15 beats per minute, their speech rate increases by 15%, and they produce more non-command vocalizations such as "um" and "ah." Based on these correlations, the system establishes a mapping model from the user's abnormal changes in physiological indicators and speech patterns during surgical simulation tasks to the internal physiological state influence coefficient. When the user's real-time heart rate suddenly increases by 20 beats per minute, their speech rate increases by 25%, and non-command vocalizations significantly increase during the current simulated surgery, this real-time data is converted using the aforementioned mapping model. For example, an abnormally high heart rate is quantified as a high stress factor, and an increased speech rate and more non-command vocalizations are quantified as corresponding fatigue or stress factors. Ultimately, these transformed factors are combined to calculate a high internal physiological state influence coefficient, indicating that the user is currently in a state of high fatigue or stress. This coefficient is then used to correct the health of the voice information, allowing the system to process user voice commands more cautiously, such as reducing the confidence level for certain ambiguous voice commands or prompting the user to rest, thereby avoiding misrecognition of commands due to poor user condition.

[0137] In some existing implementations, when it is necessary to obtain the correlation patterns between abnormal changes in physiological indicators and speech patterns related to the current VR teaching task type in the user's historical interaction data, relying solely on historical data that perfectly matches the current VR teaching task type may result in insufficient data volume or weak data representativeness. This may lead to the identification of incomplete or inaccurate correlation patterns, thereby affecting the accuracy of generating the internal physiological state influence coefficient and ultimately reducing the accuracy of correcting the health status of speech information.

[0138] In response, this application further proposes a method for optimizing the correlation between abnormal changes in physiological indicators and speech patterns related to the current VR teaching task type in historical user interaction data, including: S501. Obtain abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type from the user's historical interaction data.

[0139] S502. Identify teaching task types in the user's historical interaction data that are similar to the current VR teaching task type; and extract supplementary physiological indicators and abnormal changes in voice patterns from the data of similar teaching task types.

[0140] Similarity can be assessed based on multiple dimensions, including task objectives, operational complexity, required cognitive load, and interaction patterns. For example, similar tasks can be identified by calculating the cosine similarity of feature vectors between different task types or by using clustering algorithms. Once similar teaching task types are identified, supplementary physiological indicators and abnormal changes in speech patterns are extracted from the data of these similar teaching task types. This supplementary data aims to expand the dataset and provide richer information for subsequent identification of correlation patterns.

[0141] S503. Perform consistency verification on the supplementary physiological indicators and abnormal changes in voice patterns.

[0142] The purpose is to ensure that these supplementary data are sufficiently consistent with the data of the current VR teaching task type in terms of statistical characteristics, distribution patterns or key features, and to avoid introducing noise or irrelevant data, thereby ensuring the effectiveness of the integrated data.

[0143] Consistency checks can include, but are not limited to, statistical tests (such as t-tests and chi-square tests), distribution matching analysis, or anomaly detection based on machine learning models.

[0144] S504. After the consistency check passes, the supplemented physiological indicators and abnormal changes in voice patterns are integrated with the abnormal changes in physiological indicators and voice patterns in the user's historical interaction data that are related to the current VR teaching task type.

[0145] Specifically, the integration method can be a simple merging of the two sets of data, or a more complex weighted fusion strategy, such as assigning different weights to data from different sources based on similarity or data quality. The integrated data will contain a wider range of samples and richer variability, laying the foundation for subsequent analysis.

[0146] S505. Analyze the integrated data to identify the correlation between physiological indicators and abnormal changes in speech patterns.

[0147] Specifically, a multi-layer processing structure can be constructed; based on the integrated data and the multi-layer processing structure, the correlation between physiological indicators and abnormal changes in speech patterns can be identified; each layer of the processing structure receives the output of the previous layer and performs feature transformation on the output; the last layer of the multi-layer processing structure outputs the correlation between physiological indicators and abnormal changes in speech patterns.

[0148] In this context, a multi-layered processing architecture can be understood as a deep learning model or a multi-stage data processing pipeline. Specifically, this architecture is designed to contain multiple interconnected processing layers, each configured to perform a specific operation or transformation on the input data. For example, a multi-layered processing architecture can be a multilayer perceptron (MLP), a recurrent neural network (RNN), a convolutional neural network (CNN), or a variant thereof, or it can be a hybrid model combining different types of processing units.

[0149] This layer-by-layer transmission and processing mechanism allows data to be progressively refined and transformed at an abstract level. For example, the first-layer processing structure can extract low-level features from the original integrated data, while the second-layer processing structure can build higher-level, more abstract feature representations based on these low-level features. Feature transformations can include, but are not limited to, linear transformations, non-linear activations, pooling, and attention mechanisms, with the aim of learning more discriminative or representative features from the data.

[0150] Ultimately, the final layer of the multi-layered processing structure is configured to output the correlation between physiological indicators and abnormal changes in speech patterns. This means that after multiple layers of feature extraction and transformation, the model can generate a definite output that directly reflects the intrinsic relationship or predictive model between physiological indicators and abnormal changes in speech patterns. For example, this output could be a numerical value representing correlation, a classification result (such as whether a certain correlation exists), or a regression prediction (such as the strength of the correlation).

[0151] Thus, by constructing a multi-layered processing structure, deep and multi-level feature learning and abstraction can be performed on the integrated data. Each layer further transforms and refines the features output by the previous layer, gradually revealing the complex correlation patterns hidden in the original data. This hierarchical processing mechanism allows the model to automatically learn feature representations at different levels of abstraction from abnormal changes in original physiological indicators and speech patterns, thereby more effectively capturing the non-linear and complex correlation patterns between them. In this way, even with large amounts of data, high feature dimensions, and unclear correlation patterns, the accuracy and robustness of correlation pattern identification can be improved.

[0152] The solution proposed in this application optimizes the acquisition of correlation patterns because it effectively expands the dataset used for analysis by introducing historical data similar to the current VR teaching task types. When historical data for a specific task type is sparse, data from similar task types can provide valuable supplementary information, thus overcoming the limitation of insufficient data for a single task type. Simultaneously, rigorous consistency checks on the supplementary data ensure the quality and relevance of the introduced data, avoiding analytical biases caused by data mismatches. It is precisely because of the increased data volume and guaranteed data quality that the correlation patterns between physiological indicators and abnormal changes in speech patterns can be identified more comprehensively and accurately, thus providing a more solid data foundation for the subsequent generation of internal physiological state influence coefficients.

[0153] Through the above technical solution, this application effectively addresses the problem of accurately identifying the correlation between physiological indicators and abnormal changes in speech patterns when historical data is insufficient. By integrating data from similar teaching task types and performing rigorous consistency checks, the amount of data used for model training and analysis is significantly increased, improving the generalization ability and robustness of the correlation patterns. Consequently, the generated internal physiological state influence coefficient will more accurately reflect the user's true physiological state, leading to more accurate correction of speech health information and ultimately improving the overall performance and user experience of VR teaching immersive interactive command recognition.

[0154] In some preferred embodiments, assuming the current VR teaching task is a "surgical simulation," it's necessary to identify the user's fatigue or stress state during the operation. The system first acquires historical physiological indicators (such as heart rate variability and skin conductance) and abnormal changes in speech patterns (such as speech rate, pitch, and pause frequency) from the user's experience in the "surgical simulation" task. However, if the historical data for "surgical simulation" is limited, the system further identifies similar teaching task types, such as "internal medicine diagnosis simulation" or "first aid drill simulation," which share similarities in cognitive load and operational stress. Subsequently, the system extracts supplementary physiological indicators and abnormal changes in speech patterns from the historical data of "internal medicine diagnosis simulation" and "first aid drill simulation." Before integrating this supplementary data, the system performs a consistency check. For example, it checks whether the heart rate variability distribution in the "internal medicine diagnosis simulation" is statistically insignificantly different from that in the "surgical simulation," or checks whether the characteristic dimensions of abnormal changes in speech patterns are consistent. If the check passes, this supplementary data is integrated into the original dataset of the "surgical simulation." Finally, the larger integrated dataset is analyzed, for example, by using deep learning models to learn the complex nonlinear correlation between physiological indicators and abnormal changes in speech patterns, thereby obtaining a more comprehensive and accurate correlation model for generating internal physiological state influence coefficients.

[0155] In some embodiments described above, this application proposes extracting supplementary physiological indicators and abnormal voice pattern change data from historical data similar to the current VR teaching task type to identify the correlation patterns between these changes. However, simply integrating this supplementary data may have potential limitations. Specifically, even if different teaching task types are similar, their abnormal physiological indicator and voice pattern change data may have subtle or significant differences. Directly integrating these potentially heterogeneous data without proper screening and processing may introduce noise or inconsistent information, affecting the accuracy of subsequent correlation pattern identification. This could lead to insufficiently accurate internal physiological state influence coefficients, ultimately impacting the overall performance of VR teaching immersive interactive command recognition.

[0156] In response, this application further proposes steps for verifying the consistency between the aforementioned supplementary physiological indicators and abnormal changes in speech patterns, specifically including: S601. Obtain the distribution characteristics of each physiological indicator and speech pattern abnormal change dimension in the supplementary physiological indicator and speech pattern abnormal change data.

[0157] Specifically, obtaining the distribution characteristics of abnormal changes in various physiological indicators and speech patterns in supplementary physiological indicators and speech patterns refers to extracting the statistical properties of the supplementary data across different dimensions of physiological indicators (e.g., heart rate, skin conductance, eye movement data, etc.) and speech patterns (e.g., speech rate, pitch, volume variation, pause frequency, etc.) through statistical analysis or data mining techniques. These distribution characteristics may include, but are not limited to, mean, variance, skewness, kurtosis, probability density function, or cumulative distribution function, with the aim of comprehensively characterizing the inherent patterns and manifestations of the supplementary data across various dimensions.

[0158] S602. Based on the current VR teaching task type, determine the verification standard corresponding to the distribution characteristics.

[0159] Specifically, a set of benchmarks can be established to evaluate the quality and relevance of supplementary data based on the specific requirements of the current VR teaching task and the expected user state. For example, for a VR surgical simulation task requiring high concentration, the validation criteria might require that the distribution characteristics of physiological indicators (such as heart rate variability) and speech patterns (such as speech rate stability) be within a specific range to reflect the user's highly focused state. These validation criteria can be defined based on expert knowledge, historical task data analysis, or preset rules, with the aim of providing an objective and quantifiable reference system for subsequent matching degree calculations.

[0160] S603. For each dimension of the supplementary physiological indicators and abnormal changes in speech patterns, calculate the matching degree of distribution features according to the verification standard.

[0161] Specifically, appropriate statistical or machine learning methods can be used to quantify the degree of conformity between the supplementary data and the preset verification standards in each physiological indicator and abnormal change dimension of speech pattern. For example, methods such as the Kolmogorov-Smirnov test, chi-square test, Kullback-Leibler divergence, or cosine similarity can be used to calculate the similarity or difference between the distribution characteristics of the supplementary data and the verification standards. The purpose is to identify data dimensions that are inconsistent with the current task type or have significant deviations.

[0162] S604. Based on the matching degree calculation results, identify and quantify the heterogeneity dimension in the supplemented physiological indicators and abnormal changes in speech patterns.

[0163] Among them, the heterogeneity dimension is the dimension with a matching degree lower than the matching degree threshold.

[0164] S605. Based on the quantification results of the heterogeneity dimension, make consistency adjustments or screenings for the supplementary physiological indicators and abnormal changes in speech patterns.

[0165] Specifically, consistency adjustment can include normalizing, standardizing, interpolating, resampling, or model-based corrections to data in heterogeneous dimensions, making their distribution characteristics more consistent with verification standards. Screening, on the other hand, means directly removing data with excessively high heterogeneity or that cannot be effectively adjusted, ensuring that the final integrated data has higher consistency and reliability. Its purpose is to improve the overall quality of the data, providing a cleaner and more effective data foundation for subsequent identification of the correlation between physiological indicators and abnormal changes in speech patterns.

[0166] This application's solution effectively addresses the heterogeneity issue that may arise when extracting supplementary data from similar teaching task types by introducing a consistency verification step. Specifically, by acquiring the distribution characteristics of the supplementary data and determining the verification criteria in conjunction with the current VR teaching task type, an objective basis for data quality assessment can be provided. Subsequently, by calculating the matching degree of data in each dimension, heterogeneous dimensions that do not match the current task can be accurately identified. It is precisely because these heterogeneous dimensions are quantified that subsequent targeted consistency adjustments or screenings can be performed, thereby ensuring that the quality and relevance of the supplementary data are effectively guaranteed before integration with users' historical interaction data. This preprocessing mechanism avoids biases in identifying correlation patterns caused by data inconsistencies, thereby improving the accuracy of the internal physiological state influence coefficient.

[0167] Through the above technical solutions, this application can significantly improve the quality and consistency of supplementary physiological indicators and abnormal changes in speech patterns. By identifying, quantifying, and processing the heterogeneous dimension, irrelevant or inaccurate data can be effectively avoided from being introduced into the process of identifying correlation patterns, thereby ensuring that the correlation patterns between the identified physiological indicators and abnormal changes in speech patterns are more accurate and reliable. This further improves the generation accuracy of the internal physiological state influence coefficient, enabling the VR teaching immersive interactive command recognition method to more accurately adapt to the user's real physiological and psychological state, optimize the adjustment effect of modal weights, and ultimately provide users with a smoother and more natural immersive interactive experience.

[0168] In some preferred embodiments, assuming the current VR teaching task is "high-altitude operation simulation training," this task typically induces a higher heart rate and a certain degree of stress response in the user. When acquiring supplementary physiological indicators and abnormal changes in voice patterns, some supplementary data may be extracted from historical data of "driving simulation training."

[0169] First, the system will acquire the distribution characteristics of these supplementary data, such as changes in heart rate, skin conductance, and speech rate. For example, the heart rate distribution in driving simulation training may be relatively stable, while the verification standard for high-altitude work simulation training expects the heart rate distribution to be significantly higher.

[0170] Next, the system will determine the corresponding verification standards based on the current VR teaching task type, "high-altitude operation simulation training." For example, it may set the heart rate to fluctuate within a certain high range and the skin conductance to show a significant upward trend.

[0171] Then, the matching degree is calculated for the supplementary data extracted from the driving simulation training. If the heart rate distribution in the supplementary data is found to have a low matching degree with the verification standard of the high-altitude operation simulation training, the heart rate dimension is identified as a heterogeneous dimension.

[0172] Finally, based on the quantification results of the heterogeneity dimension, the system can process the supplementary data. For example, for the heart rate dimension, adjustments can be made to make its distribution more closely resemble the characteristics of high-altitude work simulation training, or supplementary data with heart rate data that are seriously inconsistent with the high-altitude work simulation training scenario can be directly filtered out. In this way, the integrated data is ensured to be more targeted and accurate, thus providing a high-quality data foundation for subsequent identification of the correlation between physiological indicators and abnormal changes in speech patterns.

[0173] like Figure 3 As shown in the figure, this embodiment of the invention also provides a VR teaching immersive interactive command recognition system. The system includes: The information acquisition module is used to acquire the user's hand gestures, voice information, and specific information about the current VR teaching scene; The gesture processing module is used to process hand motion information to obtain the probability distribution of gesture command intent; The voice processing module is used to process voice information and obtain the probability distribution of the intent of the voice command. The weighting adjustment module is used to adjust the weights of the probability distribution of gesture command intent and the probability distribution of voice command intent based on specific information. The intent determination module is used to align and fuse the probability distribution of gesture command intent with the probability distribution of voice command intent based on the adjusted weights, calculate the comprehensive intent probability distribution, and determine the command intent based on the comprehensive intent probability distribution.

[0174] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by a computer program instructing related hardware. This program can be stored in the computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be an internal storage unit of the task execution device (including a data sending end and / or a data receiving end) of any of the foregoing embodiments, such as the hard disk or memory of the task execution device. The computer-readable storage medium can also be an external storage device of the terminal device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device. Further, the computer-readable storage medium can include both the internal storage unit of the task execution device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the task execution device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0175] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0176] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0177] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A method for recognizing immersive interactive commands in VR teaching, characterized in that, include: Acquire user hand gestures, voice information, and specific information about the current VR teaching scenario; The hand gesture information is processed to obtain the probability distribution of the gesture command intent; The voice information is processed to obtain the probability distribution of the voice command intent; Based on the specific information, adjust the weights of the probability distribution of the gesture command intent and the probability distribution of the voice command intent; Based on the adjusted weights, the probability distribution of the gesture command intent is aligned and fused with the probability distribution of the voice command intent to calculate the comprehensive intent probability distribution, and the command intent is determined according to the comprehensive intent probability distribution.

2. The VR teaching immersive interactive command recognition method according to claim 1, characterized in that, The step of adjusting the weights of the probability distribution of the gesture command intent and the probability distribution of the voice command intent based on the specific information includes: Acquire minute motion data from the head-mounted display and ambient sound; Based on the aforementioned minute motion data, the instantaneous vibration intensity is obtained; Based on the ambient sound, the instantaneous humming sound intensity is obtained; Based on the instantaneous vibration intensity and the instantaneous humming intensity, predict the future interference intensity; Based on the predicted future interference intensity, the health scores of gesture information and voice information are adjusted. Based on the type of the current VR teaching task, obtain the preset modal weights; Based on the corrected gesture information health score, the corrected voice information health score, and the preset modal weights, the weights of the probability distribution of the gesture command intent and the probability distribution of the voice command intent are adjusted.

3. The VR teaching immersive interactive command recognition method according to claim 2, characterized in that, The prediction of future interference intensity based on the instantaneous vibration intensity and the instantaneous humming intensity includes: Monitor the rate and direction of change of the instantaneous vibration intensity and the instantaneous humming sound intensity; Based on the rate and direction of change of the instantaneous vibration intensity and the instantaneous humming intensity, abnormal patterns in the instantaneous vibration intensity and the instantaneous humming intensity are identified; Based on the abnormal pattern, predict the future interference intensity.

4. The VR teaching immersive interactive command recognition method according to claim 2, characterized in that, The process of correcting the health status of gesture information based on the future interference intensity includes: Obtain the actual shaking amplitude of key hand points in the image; The comparison results are obtained by comparing the actual jitter amplitude with the future interference intensity; When the comparison results show a deviation within a preset range, a dynamic correction factor is calculated; The health status of the gesture information is corrected by combining the dynamic correction factor.

5. The VR teaching immersive interactive command recognition method according to claim 2, characterized in that, Correcting voice message health status, including: Obtain user physiological data; Based on the user's physiological data, extract physiological indicators that reflect the user's fatigue or stress state; Analyze the non-command speech features in user speech signals; Based on the aforementioned non-command speech features, identify abnormal changes in speech patterns; Based on the abnormal changes in the physiological indicators and the speech patterns, an internal physiological state influence coefficient is generated. The health status of the voice information is corrected by combining the internal physiological state influence coefficient and the future interference intensity.

6. The VR teaching immersive interactive command recognition method according to claim 5, characterized in that, The step of generating an internal physiological state influence coefficient based on the abnormal changes in the physiological indicators and the speech pattern includes: Identify the type of the current VR teaching task; Obtain the correlation patterns between abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type in the user's historical interaction data; Based on the type of the current VR teaching task and the correlation pattern, determine the mapping relationship between the physiological indicators and the abnormal change data of the voice pattern; Based on the mapping relationship, the abnormal change data of the physiological indicators and the voice pattern are converted; The internal physiological state influence coefficient is calculated based on the abnormal changes in the converted physiological indicators and the converted speech patterns.

7. The VR teaching immersive interactive command recognition method according to claim 6, characterized in that, The process of obtaining correlation patterns between abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type in the user's historical interaction data includes: Obtain abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type from the user's historical interaction data; Identify teaching task types in the user's historical interaction data that are similar to the current VR teaching task type; and extract supplementary physiological indicators and abnormal changes in voice patterns from the data of the similar teaching task types; The consistency of the supplemented physiological indicators and abnormal changes in voice patterns is verified. After the consistency check is passed, the supplemented physiological indicators and abnormal changes in voice patterns are integrated with the abnormal changes in physiological indicators and voice patterns related to the current VR teaching task type in the user's historical interaction data. The integrated data was analyzed to identify the correlation between physiological indicators and abnormal changes in speech patterns.

8. The VR teaching immersive interactive command recognition method according to claim 7, characterized in that, The consistency verification of the supplemented physiological indicators and abnormal changes in speech patterns includes: Obtain the distribution characteristics of each physiological indicator and voice pattern abnormal change dimension in the supplemented physiological indicator and voice pattern abnormal change data; Based on the current VR teaching task type, determine the verification criteria corresponding to the distribution characteristics; For each dimension of the supplementary physiological indicators and abnormal changes in voice patterns, the matching degree of distribution features is calculated according to the verification criteria. Based on the matching degree calculation results, the heterogeneity dimension in the supplementary physiological indicators and abnormal changes in speech patterns is identified and quantified; the heterogeneity dimension is the dimension with a matching degree lower than the matching degree threshold. Based on the quantification results of the heterogeneity dimension, the supplementary physiological indicators and abnormal changes in speech patterns are adjusted or filtered for consistency.

9. The VR teaching immersive interactive command recognition method according to claim 7, characterized in that, The analysis of the integrated data to identify the correlation between physiological indicators and abnormal changes in speech patterns includes: Construct a multi-layered processing structure; Based on the integrated data and the multi-layer processing structure, the correlation between physiological indicators and abnormal changes in speech patterns is identified; wherein, each layer of processing structure receives the output of the previous layer and performs feature transformation on the output; the last layer of processing structure in the multi-layer processing structure outputs the correlation between physiological indicators and abnormal changes in speech patterns.

10. A VR teaching immersive interactive command recognition system, characterized in that, The system includes: The information acquisition module is used to acquire the user's hand gestures, voice information, and specific information about the current VR teaching scene; The gesture processing module is used to process the hand movement information to obtain the probability distribution of the gesture command intent; The voice processing module is used to process the voice information to obtain the probability distribution of the voice command intent; The weight adjustment module is used to adjust the weights of the probability distribution of the gesture command intent and the probability distribution of the voice command intent based on the specific information. The intent determination module is used to align and fuse the probability distribution of the gesture command intent with the probability distribution of the voice command intent based on the adjusted weights, calculate the comprehensive intent probability distribution, and determine the command intent based on the comprehensive intent probability distribution.

Images