Teaching Virtual Simulation Method and System Based on Mixed Reality and Artificial Intelligence

Abstract

The present invention proposes a teaching virtual simulation method and system based on mixed reality and artificial intelligence. The method includes: in the MR environment, collecting multi-modal operation data of students through sensors and cameras, using the multi-modal operation data as raw data, and at the same time, generating an operation feature vector after each sampling. The operation feature vector contains all operation data at time t. Finally, preprocess the operation feature vector and output the preprocessed feature stream and time series; generate a personalized learning path according to the feature stream and time series to match the learning progress with the cognitive level of the students; generate instant feedback for each operation step according to the personalized learning path and record the response of the students to help the students adjust in real time during the operation; optimize the learning path of the students and generate a learning progress report according to the response to form a loop adjustment mechanism for the learning path. The present invention realizes highly dynamic learning path adjustment and rich multi-sensory feedback.

Description

Technical Field

[0001] The present invention belongs to the field of teaching virtual simulation, and particularly relates to a teaching virtual simulation method and system based on mixed reality and artificial intelligence. Background Art

[0002] In the modern teaching process, especially in teaching scenarios involving complex operations or high-risk environments (such as medical surgery training, chemical experiments, mechanical operations, etc.), virtual simulation technology is gradually becoming an important teaching tool. Through the virtual simulation system, students can familiarize themselves with the operation process and master skills in a low-risk environment, reduce their dependence on real equipment and environment, and lower the risks in actual operations. However, the current virtual simulation teaching technology still has many limitations, especially in personalized teaching, real-time feedback, and immersive experience. Traditional virtual simulation systems usually adopt preset teaching content and fixed operation processes, lacking the ability to dynamically adjust according to the individual learning progress and understanding level of students. This static teaching mode cannot adapt to the learning needs of different students, resulting in some students may encounter difficulties in task difficulty or lose learning interest due to lack of challenges. In addition, the current feedback methods of virtual simulation systems are relatively single, and most only correct and guide through text or simple voice prompts, unable to effectively simulate the multi-sensory interaction experience in real scenarios. For example, in medical teaching, the lack of tactile feedback makes it impossible for students to truly feel the force during the operation process, making it difficult to form an intuitive understanding of the operation, resulting in difficulties in being precise and confident in real operations. This single feedback mechanism is difficult to fully cover the learning needs of students. Especially when students operate improperly or encounter problems, the system cannot timely guide and correct the operation through multi-dimensional perception. In addition, most existing systems have not formed a systematic learning path, lacking automatic tracking and adjustment of the learning progress. Students often have difficulty self-assessing during the learning process and cannot consolidate knowledge targeted, restricting the further improvement of learning effects.

[0003] To solve the above problems, mixed reality (MR) and artificial intelligence (AI) technologies are gradually introduced into the virtual simulation teaching system, aiming to enhance the personalization and intelligence level of the system. The MR technology can provide a more realistic scene reproduction and multi-sensory interaction experience, enabling students to obtain a more immersive experience than single virtual reality (VR). The AI technology provides the possibility for personalized teaching. By analyzing the behaviors, operations, and performances of students during the simulation process, the AI can dynamically generate learning paths and provide feedback in a timely manner. Although these technologies have improved the intelligence of teaching to a certain extent, most existing systems still fail to achieve efficient real-time personalized teaching. Especially in the coordination of dynamically adjusting learning content and multi-dimensional perception feedback, there are still significant deficiencies in existing systems.

[0004] Therefore, how to achieve highly dynamic learning path adjustment and rich multi-sensory feedback through the synergy of AI and MR has become the key to further improving the teaching effect of virtual simulation teaching systems. Summary of the Invention

[0005] The object of the present invention is to propose a teaching virtual simulation method and system based on mixed reality and artificial intelligence, which can achieve highly dynamic learning path adjustment and rich multi-sensory feedback through the synergy of AI and MR.

[0006] To achieve the above object, in the first aspect of the present invention, a teaching virtual simulation method based on mixed reality and artificial intelligence is provided. The method includes the following steps:

[0007] S1. In the MR environment, collect the multi-modal operation data of students through sensors and cameras, and use the multi-modal operation data as the original data. At the same time, after each sampling, generate an operation feature vector, which contains all the operation data at time t. Finally, preprocess the operation feature vector and output the preprocessed feature stream and time series;

[0008] S2. Generate a personalized learning path according to the feature stream and time series to match the learning progress with the cognitive level of students;

[0009] S3. According to the personalized learning path, generate instant feedback for each operation step and record the response of students to help students adjust in real time during the operation;

[0010] S4. According to the response, optimize the learning path of students and generate a learning progress report to form a loop adjustment mechanism for the learning path;

[0011] The specific content of S2 includes:

[0012] Construct a learning state evaluation model M according to the feature stream and time series LS , which is used to quantify and evaluate the current learning state of students; among them, the learning state evaluation model M LS is expressed as follows:

[0013]

[0014] Among them, S j represents the learning state index, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the adjustment of the model to the weight and prevent excessive bias towards the weight of specific features; among them, the learning state index S jSet between [0, 1], representing the student's mastery of the current task. The value closest to 1 indicates the highest mastery, and the value closest to 0 indicates the need for further intensive training;

[0015] Design a task selection function f(S j ) that generates a personalized task sequence P = {p1, p2, …, p j} based on the learning status indicator S m}, where p k represents each task in the learning path, including the task objective, completion conditions, and feedback criteria, as follows:

[0016]

[0017] Among them, θ h and θ l are the upper and lower thresholds respectively, determined by experimental tuning. Among them, if the student's mastery level S j is the highest, the difficulty of the task is automatically increased; if the mastery level is lower than θ l , the task difficulty is reduced, and more operation tips are provided to assist the student in completing the task;

[0018] Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

[0019] Further, the multi-modal operation data includes eye movement position, hand movement, operation frequency, and residence time.

[0020] Further, the preprocessing specifically includes:

[0021] Let the original data be D raw = {d1, d2, …, d n}, where d i represents the original values collected by each sensor. Set the operation feature vector as X t , which is expressed as follows:

[0022]

[0023] Among them, ω i is the weight coefficient of different features, d i is the feature value, n represents the total number of features, and i represents the current feature;

[0024] Standardize all operation feature vectors X t . Let μ X and σ X be the sample feature mean and standard deviation. The standardization formula is:

[0025]

[0026] After standardization, the sliding window method is adopted to obtain time series features;

[0027] Assume that the window width is w, then the feature set of the time period {t, t+1, …, t+w-1} is

[0028] For the time series S t , in order to reduce redundancy and highlight key features, an aggregation dimensionality reduction method is adopted. The present invention constructs a covariance matrix C:

[0029]

[0030] where μ S is the feature mean vector within the window; perform principal component analysis on C to obtain the dimensionality-reduced feature vector X PCA ;

[0031] For each X calculated for each sliding window PCA , a feature stream is formed X stream is the feature change stream within continuous time.

[0032] Furthermore, the S3 specifically includes:

[0033] According to the personalized learning path, in combination with each task p k , and generate a multi-dimensional feedback combination according to the operation requirements of the task and construct a feedback feature matrix F k , the feedback combination is used to provide multi-sensory guidance and prompts for students in the MR scenario;

[0034] According to the feedback feature matrix F k , the student mastery degree S j and the task difficulty parameter δ k are introduced to construct a feedback intensity matrix Λ k , which adapts to the performance of the student in the current task and is expressed as follows:

[0035] Λ k = F k ·σ(S j -δ k ) + β·Reg(F k )

[0036] where σ(·) is an activation function used to standardize the value range of the intensity matrix; β·Reg(F k ) is a regularization term to prevent the system from over-relying on a certain feedback, and β is a regularization coefficient;

[0037] According to the feedback intensity matrix Λk Generate corresponding multi-dimensional feedback during the task execution. Meanwhile, after each feedback generation, record the student's response situation in real time as the feedback response vector R k ;

[0038] After completing all tasks in the task sequence P, combine all feedback response vectors R k into a feedback response sequence R seq ={R1, R2, …, R m}.

[0039] Furthermore, the feedback combination includes vision, touch, and hearing; the feedback feature matrix F k , is expressed as follows:

[0040] F k =φ(p k ) = [f vis f tac f aud T

[0041] where f vis , f tac , and f aud represent the weight parameters of visual, tactile, and auditory feedback respectively, used to set the importance of different types of feedback in a specific task p k ; φ(p k ) represents the feedback adaptation function, which selects the optimal feedback combination according to the task requirements and operation complexity and generates the feedback feature matrix F k .

[0042] Furthermore, control the intensity and triggering conditions of the feedback according to the feedback intensity matrix Λ k , specifically: when the student's mastery level S j is lower than the task difficulty δ k , the feedback intensity increases; while when S j is close to δ k , the feedback intensity weakens.

[0043] Furthermore, generate corresponding multi-dimensional feedback during the task execution according to the feedback intensity matrix Λ k , specifically including: visual feedback, tactile feedback, and auditory feedback; the real-time recording of the student's response situation includes the following key indicators: the number of errors e k , response time t k and the number of operation adjustments a k ; the feedback response vector is expressed as follows:

[0044] R k =[e k t​k a k T

[0045] Among them, e k represents the number of errors that occur in task p k , t k is the average response time of the student in the task, and a k is the number of adjustments, recording the response frequency of the student to the feedback during the operation.

[0046] Furthermore, the S4 specifically includes:

[0047] Taking the feedback response sequence R seq ={R1, R2, …, R k , R m} as the input data for path optimization, which contains the key response information of each task. Among them, each feedback response vector R k includes the number of errors e k , the response time t k and the number of operation adjustments a k ;

[0048] Extracting data from the feedback response sequence R seq R seq and calculating the difficulty adjustment coefficient Δ k of each task to achieve the dynamic adjustment of task p k in the optimized path P opt ;

[0049] According to the task difficulty parameter δ k , the optimized task difficulty:

[0050] δ′ k =δ k +Δ k

[0051] Generating a new task sequence P opt ={p′1, p′2, …, p′ m} for the student, where each task p′ k corresponds to the new difficulty coefficient δ′ k . At the same time, according to the new task difficulty, the system generates an updated feedback matrix F′ k for the task to ensure that the feedback intensity is adapted to the task difficulty;

[0052] After the execution of the optimized path, generating a learning progress report R report based on the student's performance, summarizing the completion of the optimized path P opt and outputting the learning progress report R report . ​

[0053] Furthermore, the difficulty adjustment coefficient Δ k is calculated as follows:

[0054]

[0055] where γ, λ, and θ are the weight coefficients of each feedback parameter, which are set by tuning through actual teaching experiments; represents a non - linear regularization term, which is used to non - linearly amplify the response time. When t k is the highest, this term is closest to 1, making the value of Δ k the largest, thereby reducing the task difficulty; when t k is the lowest, then this term approaches 0 most closely, and the difficulty remains unchanged or even increases;

[0056] The feedback matrix F′ k , is calculated as follows:

[0057] F′ k =α·F k +β·Reg(F k )

[0058] where α controls the update weight of the feedback matrix F k , and β controls the weight of the feedback matrix regularization term Reg(F k ), to avoid unnecessary over - amplification of the feedback intensity during the update.

[0059] In the second aspect of the present invention, a teaching virtual simulation system based on mixed reality and artificial intelligence is provided. The system includes:

[0060] A data acquisition unit, which is used to collect multi - modal operation data of students through sensors and cameras in the MR environment, take the multi - modal operation data as the original data, and at the same time, generate an operation feature vector after each sampling. The operation feature vector contains all operation data at time t, and finally pre - processes the operation feature vector to output the pre - processed feature stream and time series;

[0061] A personalized learning path generation unit, which is used to generate a personalized learning path according to the feature stream and time series, so that the learning progress matches the cognitive level of the students;

[0062] A real - time adjustment unit, which is used to generate instant feedback for each operation step according to the personalized learning path and record the response of the students, to help the students adjust in real - time during the operation;

[0063] A real - time optimization unit, which is used to optimize the learning path of the students according to the response and generate a learning progress report, so that the learning path forms a loop adjustment mechanism;

[0064] The steps executed by the personalized learning path generation unit specifically include:

[0065] Construct a learning state evaluation model M based on feature streams and time series LS , used to quantify and evaluate the student's current learning status; wherein the learning status evaluation model M LS , which is expressed as follows:

[0066]

[0067] Among them, S j represents the learning state indicator, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the model's adjustment of weights to prevent excessive bias towards specific feature weights; among them, the learning state indicator S j It is set between [0,1], indicating the students' mastery of the current task. Values closest to 1 indicate the highest mastery, and values closest to 0 indicate the need for further intensive training.

[0068] Design task selection function f(S j ) According to the learning state indicator S j Generate personalized task sequence P = {p1, p2, ..., p m}, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria, as follows:

[0069]

[0070] Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning; among them, if the student's mastery level S j The highest, the difficulty of the task is automatically increased; if the mastery is lower than θ l , then reduce the difficulty of the task and provide more operation prompts to assist students in completing the task;

[0071] Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

[0072] The beneficial technical effects of the present invention are at least as follows:

[0073] The invention deeply analyzes the real-time performance of students through AI algorithms to dynamically adjust the learning path, ensuring that the learning process of students can fit their actual level and comprehension ability. The AI system generates personalized learning paths based on data such as students' operation accuracy, dwell time, and error rate. When students encounter difficulties in a certain knowledge point or operation step, the system will automatically reduce the task difficulty and provide detailed demonstrations and guidance; when students show a high degree of mastery, the system will automatically generate more challenging tasks or further knowledge expansion. This ability to dynamically generate paths not only meets the learning needs of students at different levels but also prevents students from losing their learning interest in tasks with overly high or low difficulty. Compared with the fixed content design of traditional simulation systems, the invention realizes a highly personalized teaching process through real-time path optimization.

[0074] The invention introduces a multi-sensory interaction mechanism in the virtual simulation environment, using MR technology combined with multi-dimensional feedback such as vision, touch, and hearing to provide students with a highly immersive learning experience. The system provides visual cues (such as highlighting or color changes), tactile feedback (such as force sensing), and sound effect simulations (such as ambient sounds or alarm sounds) in a timely manner according to students' different operations and learning progress to enhance students' understanding of operations and practical perception. For example, in surgical simulations, when students' operations are incorrect or inaccurate, the system uses red highlighting and vibration feedback to remind students, and provides positive feedback through green visual feedback and sound rewards when the operations are correct. Through this multi-dimensional perception interaction system, the invention not only improves students' immersive experience but also effectively enhances students' intuitive understanding of operations, enabling them to obtain a feel and reaction close to real operations in the simulation environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention will be further described with reference to the accompanying drawings. However, the embodiments in the drawings do not constitute any limitation to the present invention. For those of ordinary skill in the art, other drawings can be obtained based on the following drawings without creative efforts.

[0076] Figure 1 It is a flowchart of the teaching virtual simulation method based on mixed reality and artificial intelligence of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0077] The embodiments of the present invention will be described in detail below. The examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0078] As Figure 1As shown in the figure, the teaching virtual simulation method based on mixed reality and artificial intelligence provided by the embodiments of the present invention includes the following steps:

[0079] S1. In the MR environment, collect multi-modal operation data of students through sensors and cameras, use the multi-modal operation data as the original data, and at the same time, generate an operation feature vector after each sampling. The operation feature vector contains all operation data at time t. Finally, preprocess the operation feature vector and output the preprocessed feature stream and time series.

[0080] S2. Generate a personalized learning path according to the feature stream and time series to match the learning progress with the cognitive level of students.

[0081] S3. Generate instant feedback for each operation step according to the personalized learning path and record the response of students to help students adjust in real time during the operation.

[0082] S4. Optimize the learning path of students according to the response and generate a learning progress report to form a loop adjustment mechanism for the learning path.

[0083] Specifically, S1 specifically includes:

[0084] In the MR environment, the present invention first collects multi-modal operation data of students through sensors and cameras, including eye movement position, hand movement, operation frequency, residence time, etc. Let the original data composed of these multi-modal features be D raw ={d1, d2,..., d n}}, where d i represents the original numerical value collected by each sensor. To capture the time series features, after each sampling, the system generates an operation feature vector X t , which contains all operation data at time t. For the multi-modal features of students, the present invention sets different weight factors ω i and assigns values according to the importance of the data (such as eye movement position, gesture recognition, etc.). X t is defined as follows:

[0085]

[0086] Among them, ω i is the weight coefficient of different features, and d i is the feature value. X t is the feature vector of the student's operation at time t and is used for subsequent analysis. This process assigns different weights to different modal data to meet the feature collection accuracy requirements in the patent.

[0087] Furthermore, to ensure the comparability of feature data, all X t need to be standardized. Let μX and σ X are the mean and standard deviation of the sample features, and the standardization formula is:

[0088]

[0089] After standardization, the sliding window method is used to obtain time series features. Assuming the window width is w, the feature set of the time period {t, t + 1, …, t + w - 1} is This sliding window operation captures the changing trends of various features in continuous actions, and is especially suitable for the generation requirements of dynamic learning paths.

[0090] Furthermore, for the time series S t , in order to reduce redundancy and highlight key features, an aggregation and dimensionality reduction method is adopted. The present invention constructs a covariance matrix C:

[0091]

[0092] where μ S is the mean feature vector within the window. Then, perform principal component analysis (PCA) on C to obtain the dimensionality-reduced feature vector X PCA . This operation keeps the feature dimension while maximizing important information and filtering out noise and redundant data. The final output X PCA is used as the standardized input of the feature data stream for the path generation model.

[0093] Furthermore, for each X calculated for each sliding window PCA , a feature stream is formed X stream is the feature change stream within continuous time and will be used as the input of the learning path generation model.

[0094] Specifically, the S2 specifically includes

[0095] the feature stream output from step 1 as input data. Each represents the dimensionality-reduced feature vector at time point j. The feature stream X stream records the multi-modal information changes of the student during the operation, reflecting the operation trends and features of the student, and is especially suitable for the personalized learning path generation requirements in the patent scenario.

[0096] After receiving X stream , the system inputs it into the learning state evaluation model to capture the current operation state and mastery of the student. This feature stream not only contains the operation features after dimensionality reduction processing, but also reflects the learning dynamics of the student at different time points, and can provide rich context information for generating personalized learning paths.

[0097] Furthermore, to achieve the personalized generation of dynamic learning paths, the system constructs a learning status evaluation model M customized for patents LS , which is used to quantify and evaluate the current learning status of students. This model receives the feature vectors at each time point and outputs the learning status index S j , which represents the mastery level and learning effect of students at this time point.

[0098] To ensure the accurate evaluation of the model for multi-modal features, a dynamic weight matrix W is added to the model. The elements in this matrix will be continuously adjusted according to time and operation behaviors to better adapt to the real-time learning situation of students. Specifically, the calculation formula of S j is as follows:

[0099]

[0100] where W(t) is the dynamic weight matrix, representing the weights for different features at time t. The change of t can be adaptively adjusted according to the operation habits or task completion status of students. The regularization term α·Reg(W(t)) is used to control the adjustment of the weights of the model to prevent excessive bias towards the weights of specific features, and α is the regularization coefficient. The value of S j is set between [0, 1], representing the mastery level of students for the current task: S j close to 1 indicates a high level of mastery, and close to 0 indicates the need for continued intensive training. S j will be used as the key basis for subsequent path selection.

[0101] Furthermore, according to the real-time mastery situation S j of students, the system will generate a personalized task sequence P = {p1, p2,..., p m}, where p k represents each task in the learning path, including the task objective, completion conditions, and feedback criteria. This task sequence generation model takes into account the personalized learning needs of patents and dynamically selects suitable task content according to the value of S j .

[0102] To ensure that the learning tasks match the current mastery situation of students, the task selection function f(S j ) is designed as follows:

[0103]

[0104] where, θ h and θ l are the upper and lower thresholds respectively, which are determined by experimental tuning. If the mastery level S j of students is relatively high (such as greater than θ h), the system will automatically increase the difficulty of the task (e.g., from basic operation task p k to a higher-difficulty reasoning task p k+1 ); if the mastery level is lower than θ l , the task difficulty will be reduced and more operation tips will be provided to assist the student in completing the task. This task sequence ensures that each step of the student's learning task matches the current mastery level.

[0105] Furthermore, the system finally outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task. This path P constitutes the student's dynamic learning process, and the difficulty of each task is based on the student's real-time mastery. The learning path P provides clear task requirements and standards for subsequent multi-dimensional feedback generation, ensuring that each step of the student's operation in the MR environment has targeted task guidance and feedback tracking.

[0106] Specifically, the S3 specifically includes

[0107] After receiving the personalized learning path P = {p1, p2,..., p m} generated in the previous step, the system enters each task p k , and generates a multi-dimensional feedback combination according to the operation requirements of the task. These feedback combinations (such as visual, tactile, and auditory, etc.) are used to provide multi-sensory guidance and tips for the student in the MR scenario. For this purpose, a feedback adaptation function φ(p k ) is defined to select the optimal feedback combination according to the task requirements and operation complexity and generate a feedback feature matrix F k . Among them, the composition of F k is:

[0108] F k = φ(p k ) = [f vis f tac f aud T

[0109] Among them, f vis , f tac and f aud respectively represent the weight parameters of visual, tactile, and auditory feedback, used to set the importance of different types of feedback in a specific task p k . For example, in a complex surgical operation task, F k may set a higher weight for tactile feedback f tac , while visual feedback f vis is a secondary item to ensure that the focus of the feedback meets the task requirements. According to the characteristics of the MR scenario, F k can be adjusted in real time when the task is switched to ensure the adaptability of each feedback.​

[0110] Furthermore, after determining the feedback type combination, the system needs to further control the intensity of the feedback. To achieve this, the system constructs a feedback intensity matrix Λ k for each task p k , to adapt to the student's performance in the current task. Λ k not only takes into account the feedback requirements of the current task, but also introduces the student mastery S j and the task difficulty parameter δ k to ensure the adaptability of the feedback. The feedback intensity generation formula is defined as follows:

[0111] Λ k = F k ·σ(S j -δ k ) + β·Reg(F k )

[0112] where σ(·) is an activation function (such as ReLU or sigmoid) used to normalize the value range of the intensity matrix; β·Reg(F k is a regularization term to prevent the system from over-relying on a certain type of feedback, and β is the regularization coefficient. Through Λ k , the system controls the intensity and triggering conditions of the feedback. For example, when the student's mastery S j is significantly lower than the task difficulty δ k , the feedback intensity increases; while when S j is close to δ k , the feedback intensity weakens. This dynamic adjustment mechanism ensures a high degree of adaptability of the feedback to the student's current learning state, avoiding overly frequent or sparse feedback.

[0113] Furthermore, based on the feedback intensity matrix Λ k , the system generates corresponding multi-dimensional feedback during the task execution, specifically including:

[0114] A. Visual feedback: In the MR scenario, use color highlighting, direction indication, etc. to help students understand the operation position and direction. The visual feedback intensity f vis determines the degree and frequency of highlighting.

[0115] B. Tactile feedback: Use the MR handle to generate vibration or pressure sensation to let students obtain tactile feedback. For example, when force control is required in task p k , the tactile feedback can remind students that the force is insufficient or excessive.

[0116] C. Auditory feedback: Play prompt sounds or voice guidance to help students identify key operation nodes. The auditory feedback intensity f aud controls the volume and repetition frequency of the prompts.

[0117] After each feedback generation, the system records the students' response situation in real time, including the following key metrics: the number of errors e k , the response time t k and the number of operation adjustments a k . Record these data as the feedback response vector R k , defined as:

[0118] R k = [e k t k a k T

[0119] where e k represents the number of errors that occur in task p k , t k is the average response time of the student in the task, and a k is the number of adjustments, recording the response frequency of the student to the feedback during the operation.

[0120] Furthermore, after completing all the tasks in the task sequence P, the system combines all the feedback response vectors R k into a feedback response sequence R seq = {R1, R2, …, R m}}. This sequence records the feedback response data of the student in the entire learning path and serves as the input data stream for the next path optimization. R seq provides detailed information such as task completion status, operation adjustment behavior, and response characteristics, so that the system can adjust the learning path specifically in the next stage to ensure that the student obtains a more appropriate feedback intensity and task difficulty in the next task.

[0121] Specifically, the said S4 specifically includes

[0122] taking the feedback response sequence R seq = {R1, R2, …, R m}} output from step 3 as the input data for path optimization, which contains the key response information of each task. Each feedback response vector R k is composed of the following elements: e k : error rate, referring to the operation error frequency of the student in task p k ; t k : average response time, indicating the reaction speed of the student in the task; a k : number of operation adjustments, referring to the frequency of the student correcting the operation in the task.

[0123] ​This feedback response sequence provides detailed data on the real-time learning performance for the path optimization of the system. Through path optimization driven by feedback data, the system can identify the weak links in students' operations, especially tasks with a high number of errors or slow operation responses.

[0124] Further, the system extracts data from the feedback response sequence R seq and calculates the difficulty adjustment coefficient Δ k for each task, achieving dynamic adjustment of task p k in the optimized path P opt . Define the calculation formula of the difficulty adjustment coefficient Δ k as follows:

[0125]

[0126] where γ, λ, and θ are the weight coefficients of each feedback parameter, which are set through optimization in actual teaching experiments: γ: controls the influence of the error rate e k , ensuring that when the student has a high error rate in task p k , the difficulty is appropriately reduced; λ: controls the influence of the response time t k , reflecting the influence of the student's operation speed on the task difficulty; θ: controls the influence of the number of operation adjustments a k , when the student has a high adjustment frequency in the task, it means that the task difficulty should be reduced to adapt to the student's learning progress.

[0127] Among them, the term in the formula is used to non-linearly amplify the response time. When t k is high (i.e., the student has a long reaction time), this term is close to 1, making the value of Δ k increase, thereby reducing the task difficulty; when t k is low (i.e., the student responds quickly), then this term approaches 0, and the difficulty remains unchanged or even increases. Through this adjustment, the system can generate an optimized path with adaptive difficulty for students with different operation performances.

[0128] Further, using the optimized task difficulty δ′ k = δ k + Δ k , the system generates a new task sequence P opt = {p′1, p′2, …, p′ m} for the student, and each task p′ k corresponds to the new difficulty coefficient δ′ k . At the same time, according to the new task difficulty, the system generates an updated feedback matrix F′ k for the task to ensure that the feedback intensity is adapted to the task difficulty. Define the optimized feedback matrix F′ k as follows:

[0129] F' k = α·F k + β·Reg(F k )

[0130] where α controls the update weight of the feedback matrix F k and β controls the weight of the regularization term Reg(F k ) to avoid unnecessary over-amplification of the feedback strength during the update. F' k Based on the balance relationship between the task difficulty and the feedback strength, corresponding visual, tactile, and auditory feedbacks are generated during actual operation to ensure that the feedback remains effective in the optimization task.

[0131] Furthermore, after the optimized path is executed, the system generates a learning progress report R report summarizing the completion status of the optimized path P opt . The core data in the report includes the completion status of each task, the average response time, the error rate, and the number of adjustments, etc., which are used to analyze the student's learning progress. R report The output of R helps teachers and students review the learning process and serves as a basis for further optimizing the path and feedback settings.

[0132] Finally, R report not only records the student's performance in the optimized path but also provides data support for future path generation, realizing the closed-loop improvement of path optimization and feedback adjustment.

[0133] The embodiment of the present invention also provides a teaching virtual simulation system based on mixed reality and artificial intelligence, and the system includes:

[0134] A data acquisition unit, configured to collect multi-modal operation data of students through sensors and cameras in the MR environment, use the multi-modal operation data as raw data, and at the same time, generate an operation feature vector after each sampling. The operation feature vector contains all operation data at time t, and finally preprocess the operation feature vector to output a preprocessed feature stream and time series;

[0135] A personalized learning path generation unit, configured to generate a personalized learning path according to the feature stream and time series, so that the learning progress matches the student's cognitive level;

[0136] A real-time adjustment unit, configured to generate instant feedback for each operation step according to the personalized learning path and record the student's response, helping the student to make real-time adjustments during the operation;

[0137] A real-time optimization unit, configured to optimize the student's learning path according to the response and generate a learning progress report, so that the learning path forms a loop adjustment mechanism;

[0138] The specific steps executed by the personalized learning path generation unit include:

[0139] Construct a learning status evaluation model M based on the feature stream and time series LS , which is used to quantify and evaluate the current learning status of students; among them, the learning status evaluation model M LS is expressed as follows:

[0140]

[0141] Among them, S j represents the learning status index, W(t) is the dynamic weight matrix, indicating the weights for different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the adjustment of the model to the weights and prevent excessive bias towards the weights of specific features; among them, the learning status index S j is set between [0,1], indicating the student's mastery of the current task. The closest to 1 represents the highest mastery, and the closest to 0 indicates the need for continued intensive training;

[0142] Design a task selection function f(S j ) to generate a personalized task sequence P = {p1, p2,..., p j} according to the learning status index S m}, where p k represents each task in the learning path, including the task objective, completion conditions, and feedback criteria, and is expressed as follows:

[0143]

[0144] Among them, θ h and θ l are the upper and lower thresholds respectively, which are determined by experimental tuning; among them, if the student's mastery degree S j is the highest, the difficulty of the task is automatically increased; if the mastery degree is lower than θ l , the task difficulty is reduced, and more operation tips are provided to assist the student in completing the task;

[0145] Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

[0146] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described system, device, and unit can refer to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a division of logical functions. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Additionally, the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0148] If the described functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs, and other various media that can store program codes.

[0149] Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A teaching virtual simulation method based on mixed reality and artificial intelligence, characterized in that The method includes the following steps: S1. In the MR environment, collect the multi-modal operation data of students through sensors and cameras. Use the multi-modal operation data as the original data. At the same time, after each sampling, generate an operation feature vector, which contains all the operation data at the moment. Finally, preprocess the operation feature vector and output the preprocessed feature stream and time series; S2. Generate a personalized learning path according to the feature stream and time series to match the learning progress with the cognitive level of the student; S3. Generate instant feedback for each operation step according to the personalized learning path and record the response of the student to help the student adjust in real time during the operation; S4. Optimize the learning path of the student according to the response and generate a learning progress report to form a loop adjustment mechanism for the learning path; The S2 specifically includes: Construct a learning state evaluation model based on the feature stream and time series , which is used to quantify and evaluate the current learning state of students; among them, the learning state evaluation model is expressed as follows: ； Among them, represents the learning status index, is the dynamic weight matrix, representing the weights for different features at time moment; represents the feature vector at each time point, represents the regularization coefficient; represents the regularization term, which is used to control the adjustment of the model's weights and prevent excessive bias towards the weights of specific features; among them, the learning status index is set between [0, 1], representing the student's mastery of the current task. The value closest to 1 indicates the highest mastery, and the value closest to 0 indicates the need for continued intensive training; Design task selection function Based on the learning status indicators Generate a personalized task sequence , where Each task in the learning path, including task objectives, completion conditions, and feedback criteria, is represented as follows: ； Among them, and are the upper and lower thresholds respectively, which are determined by experimental optimization; among them, if the student's mastery degree is the highest, the difficulty of the task is automatically increased; if the mastery degree is lower than , the task difficulty is reduced, and more operation tips are provided to assist the student in completing the task; The final system outputs a personalized learning path , including the content, difficulty, completion criteria, and feedback specifications of each task.

2. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 1, wherein The multi-modal operation data includes eye movement position, hand movement, operation frequency, and residence time.

3. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 1, characterized in that, The preprocessing specifically includes: Let the original data be , where represents the original values collected by each sensor, and set the operation feature vector as , which is expressed as follows: ； Among them, is the weight coefficient of different features, is the feature value, n represents the total number of features, and i represents the current feature; For all operation feature vectors perform normalization processing, and let and be the sample feature mean and standard deviation, and the normalization formula is: ； After standardization, the sliding window method is used to obtain time series features; Assume the window width is , then the feature set of time period is ; For time series , in order to reduce redundancy and highlight key features, an aggregation and dimensionality reduction method is used to construct a covariance matrix : ； Among them, is the mean feature vector within the window; perform principal component analysis on it to obtain the feature vector after dimensionality reduction; Calculated for each sliding window forms a feature stream , is the feature change stream within continuous time.

4. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 1, wherein The S3 specifically includes: According to the personalized learning path, combined with each task and generate a multi-dimensional feedback combination according to the operation requirements of the task and construct a feedback feature matrix based on the feedback combination wherein the feedback combination is used to provide multi-sensory guidance and prompts for students in the MR scenario; According to the feedback feature matrix , the student mastery level and the task difficulty parameter are introduced to construct the feedback intensity matrix , which adapts to the student's performance in the current task and is expressed as follows: ； Among them, is an activation function used to normalize the value range of the intensity matrix; is a regularization term to prevent the system from over-relying on a certain feedback, is a regularization coefficient; According to the feedback intensity matrix generate corresponding multi-dimensional feedback during the task execution, and at the same time, record the student's response situation in real time after each feedback is generated as a feedback response vector ; After completing all tasks in the task sequence all feedback response vectors are combined into a feedback response sequence .

5. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4, wherein The feedback combination includes visual, tactile, and auditory; the feedback feature matrix , is represented as follows: ； Among them, , and respectively represent the weight parameters of visual, tactile, and auditory feedback, and are used to set the importance of different types of feedback in a specific task ; represents a feedback adaptation function, which selects the optimal feedback combination according to the task requirements and operation complexity and generates a feedback feature matrix .

6. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4, characterized in that According to the feedback intensity matrix control the intensity and trigger conditions of the feedback, specifically: when the student's mastery is lower than the task difficulty , the feedback intensity increases; while when and are closer, the feedback intensity weakens.

7. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4, wherein The said according to the said feedback intensity matrix Generate corresponding multi-dimensional feedback during the task execution, specifically including: visual feedback, tactile feedback, and auditory feedback; The real-time recording of the student's response situation includes the following key indicators: the number of errors , response time and the number of operation adjustments ; The feedback response vector is represented as follows: ； Among them, represents the number of errors occurring in the task, is the average response time of the student in the task, is the number of adjustments, recording the response frequency of the student to the feedback during the operation.

8. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4, characterized in that The S4 specifically includes: Use the said feedback response sequence as the input data for path optimization, which contains the key response information of each task. Among them, each feedback response vector includes the number of errors , response time and the number of operation adjustments ; Extract data from the feedback response sequence and calculate the difficulty adjustment coefficient for each task to achieve dynamic adjustment of tasks in the optimization path ; According to the task difficulty parameter , the optimized task difficulty: ； Generate a new task sequence for students , where each task corresponds to a new difficulty coefficient . At the same time, according to the new task difficulty, the system generates an updated feedback matrix for the task to ensure that the feedback intensity is adapted to the task difficulty; After the optimized path is executed, a learning progress report is generated based on the student's performance , summarize the optimized path completion status, and output the learning progress report .

9. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 8, wherein The difficulty adjustment coefficient is calculated as follows: ； Among them, , and are the weight coefficients of each feedback parameter, which are set by optimizing through actual teaching experiments; represents a non - linear regularization term, which is used to non - linearly amplify the response time. When is the highest, this term is closest to 1, making the largest, thereby reducing the task difficulty; when is the lowest, then this term approaches 0, and the difficulty remains unchanged or even increases; The feedback matrix , is calculated as follows: ； Among them, The update weight of the control feedback matrix And the weight of the regularization term of the control feedback matrix Avoid excessive amplification of the feedback strength during the update. ​ 10. A teaching virtual simulation system based on mixed reality and artificial intelligence, characterized in that, The system includes: The data acquisition unit is used to collect multi-modal operation data of students through sensors and cameras in the MR environment, take the multi-modal operation data as the original data, and at the same time, generate an operation feature vector after each sampling. The operation feature vector contains all the operation data at the moment, and finally preprocess the operation feature vector to output the preprocessed feature stream and time series; A personalized learning path generation unit, configured to generate a personalized learning path according to the feature stream and time series to match the learning progress with the cognitive level of the student; A real-time adjustment unit, configured to generate instant feedback for each operation step according to the personalized learning path and record the response of the student to help the student adjust in real time during the operation; A real-time optimization unit, configured to optimize the learning path of the student according to the response and generate a learning progress report to form a loop adjustment mechanism for the learning path; The specific steps executed by the personalized learning path generation unit include: Construct a learning status evaluation model based on the feature stream and time series , which is used to quantify and evaluate the current learning status of students; among them, the learning status evaluation model is expressed as follows: ； Among them, represents the learning status index, is the dynamic weight matrix, indicating the weights of different features at time moment; represents the feature vector at each time point, represents the regularization coefficient; represents the regularization term, which is used to control the adjustment of the model's weights and prevent excessive bias towards the weights of specific features; among them, the learning status index is set between [0, 1], indicating the student's mastery of the current task. The value closest to 1 represents the highest mastery, and the value closest to 0 indicates the need for continued intensive training; Design task selection function According to the learning status indicators Generate a personalized task sequence , where Each task in the learning path, including task objectives, completion conditions, and feedback criteria, is represented as follows: ； Among them, and are the upper and lower thresholds respectively, which are determined by experimental optimization; among them, if the mastery level of the student is the highest, the difficulty of the task is automatically increased; if the mastery level is lower than , the task difficulty is reduced, and more operation tips are provided to assist the student to complete the task; The final system outputs a personalized learning path , including the content, difficulty, completion criteria, and feedback specifications of each task.

Images