Teaching virtual simulation method and system based on mixed reality and artificial intelligence

Abstract

The invention provides a teaching virtual simulation method and system based on mixed reality and artificial intelligence, and the method comprises the steps: collecting the multi-modal operation data of a student through a sensor and a camera in an MR environment, taking the multi-modal operation data as original data, generating an operation feature vector after each sampling, and obtaining the multi-modal operation data; the operation feature vector comprises all operation data at the t moment, finally preprocessing the operation feature vector, and outputting a preprocessed feature flow and a time sequence; generating a personalized learning path according to the feature flow and the time sequence, and enabling the learning progress to be matched with the cognitive level of the student; according to the personalized learning path, instant feedback is generated for each operation step, and the response condition of the student is recorded, so that the student is helped to perform real-time adjustment in the operation; and according to the response condition, the learning path of the student is optimized and a learning progress report is generated, so that the learning path forms a ring adjustment mechanism. According to the invention, highly dynamic learning path adjustment and rich multi-sensory feedback are realized.

Description

Technical Field

[0001] The present invention belongs to the field of teaching virtual simulation, and in particular to a teaching virtual simulation method and system based on mixed reality and artificial intelligence. Background Art

[0002] In modern teaching, especially in teaching scenarios involving complex operations or high-risk environments (such as medical surgery training, chemical experiments, mechanical operations, etc.), virtual simulation technology is gradually becoming an important teaching tool. Through the virtual simulation system, students can become familiar with the operation process and master skills in a low-risk environment, reduce dependence on real equipment and environment, and reduce the danger in actual operation. However, the current virtual simulation teaching technology still has many limitations, especially in personalized teaching, real-time feedback and immersive experience. Traditional virtual simulation systems usually use preset teaching content and fixed operation procedures, and lack the ability to dynamically adjust according to students' personal learning progress and understanding. This static teaching mode cannot adapt to the learning needs of different students, resulting in some students encountering difficulties in task difficulty or reducing their interest in learning due to lack of challenges. In addition, the current virtual simulation system feedback method is relatively single, and most of them only use text or simple voice prompts for error correction and guidance, which cannot effectively simulate the multi-sensory interactive experience in real scenes. For example, in medical teaching, the lack of tactile feedback makes it impossible for students to truly feel the force during the operation, and it is difficult to form an intuitive understanding of the operation, which makes it difficult to be accurate and confident in real operations. This single feedback mechanism is difficult to fully cover students' learning needs, especially when students operate improperly or encounter problems, the system cannot guide and correct operations in time through multi-dimensional perception. In addition, most existing systems have failed to form a systematic learning path and lack automatic tracking and adjustment of learning progress. Students often find it difficult to self-evaluate during the learning process and cannot consolidate knowledge in a targeted manner, which limits the further improvement of learning effects.

[0003] To solve the above problems, mixed reality (MR) and artificial intelligence (AI) technologies are gradually being introduced into virtual simulation teaching systems to enhance the personalization and intelligence of the system. MR technology can provide more realistic scene reproduction and multi-sensory interactive experience, allowing students to have a more immersive experience than single virtual reality (VR). AI technology provides the possibility of personalized teaching. By analyzing students' behaviors, operations and performances during the simulation process, AI can dynamically generate learning paths and provide timely feedback. Although these technologies have improved the intelligence of teaching to a certain extent, most existing systems still fail to achieve efficient real-time personalized teaching. In particular, the existing systems still have significant deficiencies in the dynamic adjustment of learning content and the coordination of multi-dimensional perceptual feedback.

[0004] Therefore, how to achieve highly dynamic learning path adjustment and rich multi-sensory feedback through the synergy of AI and MR has become the key to further improving the teaching effect of the virtual simulation teaching system. Summary of the invention

[0005] The purpose of the present invention is to propose a teaching virtual simulation method and system based on mixed reality and artificial intelligence, and to achieve highly dynamic learning path adjustment and rich multi-sensory feedback through the synergy of AI and MR.

[0006] In order to achieve the above object, a first aspect of the present invention provides a teaching virtual simulation method based on mixed reality and artificial intelligence, the method comprising the following steps:

[0007] S1. In the MR environment, the multimodal operation data of the students are collected through sensors and cameras, and the multimodal operation data are used as raw data. At the same time, after each sampling, an operation feature vector is generated, and the operation feature vector contains all the operation data at time t. Finally, the operation feature vector is preprocessed, and the preprocessed feature stream and time series are output;

[0008] S2, generate personalized learning paths based on feature streams and time series to match the learning progress with the students’ cognitive level;

[0009] S3. Generate instant feedback for each operation step and record the student's response according to the personalized learning path, so as to help students make real-time adjustments during the operation;

[0010] S4. According to the response situation, optimize the student's learning path and generate a learning progress report, so that the learning path forms a loop adjustment mechanism;

[0011] The S2 specifically includes:

[0012] Construct a learning state evaluation model M based on feature streams and time series LS , used to quantify and evaluate the student's current learning status; wherein the learning status evaluation model M LS , which is expressed as follows:

[0013]

[0014] Among them, S j represents the learning state indicator, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the model's adjustment of weights to prevent excessive bias towards specific feature weights; among them, the learning state indicator S jIt is set between [0,1], indicating the students' mastery of the current task. Values ​​closest to 1 indicate the highest mastery, and values ​​closest to 0 indicate the need for further intensive training.

[0015] Design task selection function f(S j ) According to the learning state indicator S j Generate personalized task sequence P = {p1, p2, ..., p m}, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria, as follows:

[0016]

[0017] Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning; among them, if the student's mastery level S j The highest, the difficulty of the task is automatically increased; if the mastery is lower than θ l , then reduce the difficulty of the task and provide more operation prompts to assist students in completing the task;

[0018] Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

[0019] Furthermore, the multimodal operation data includes eye movement position, hand movement, operation frequency and dwell time.

[0020] Furthermore, the preprocessing specifically includes:

[0021] Let the original data be D raw ={d1,d2,…,d n}, where d i represents the original values ​​collected by each sensor, and the operational feature vector is set to X t , which is expressed as follows:

[0022]

[0023] Among them, ω i is the weight coefficient of different features, d i is the feature value, n represents the total number of features, and i represents the current feature;

[0024] For all operational feature vectors X t For standardization, let μ X and σ X is the sample characteristic mean and standard deviation, and the standardization formula is:

[0025]

[0026] After standardization, the sliding window method is used to obtain time series features;

[0027] Assuming the window width is w, the feature set of the time period {t, t+1,…, t+w-1} is

[0028] For the time series S t In order to reduce redundancy and highlight key features, an aggregation dimensionality reduction method is adopted. The present invention constructs a covariance matrix C:

[0029]

[0030] Among them, μ S is the feature mean vector within the window; perform principal component analysis on C to obtain the feature vector X after dimensionality reduction PCA ;

[0031] X calculated for each sliding window PCA , forming a characteristic flow X stream It is the characteristic change flow in continuous time.

[0032] Furthermore, the S3 specifically includes:

[0033] According to the personalized learning path, combined with each task p k , and generate multi-dimensional feedback combinations according to the operational requirements of the task and construct the feedback feature matrix F according to the feedback combinations k , the feedback combination is used to provide multi-sensory guidance and prompts to students in the MR scene;

[0034] According to the feedback feature matrix F k , introduced the student mastery S j and the task difficulty parameter δ k Construct the feedback strength matrix Λ k , adapted to the student's performance in the current task, is expressed as follows:

[0035] Λ k =F k ·σ(S j -δ k )+β·Reg(F k )

[0036] Among them, σ(·) is the activation function, which is used to normalize the value range of the intensity matrix; β·Reg(F k ) is a regularization term to prevent the system from over-relying on a certain feedback, and β is a regularization coefficient;

[0037] According to the feedback strength matrix Λk Generate corresponding multi-dimensional feedback during the task execution process. At the same time, after each feedback is generated, the student's response is recorded in real time as the feedback response vector R k ;

[0038] After completing all tasks in the task sequence P, all feedback response vectors R k Combined into feedback response sequence R seq = {R1, R2, …, R m}.

[0039] Furthermore, the feedback combination includes vision, touch and hearing; the feedback feature matrix F k , which is expressed as follows:

[0040] F k =φ(p k )=[f vis f tac f aud ] T

[0041] Among them, f vis 、f tac and f aud They represent the weight parameters of visual, tactile and auditory feedback, respectively, and are used to set the weight of different types of feedback in a specific task p k The importance of φ(p k ) represents the feedback adaptation function, which represents the task requirements and operation complexity to select the optimal feedback combination and generate the feedback feature matrix F k .

[0042] Further, according to the feedback strength matrix Λ k Control the intensity and triggering conditions of feedback, specifically: when the student's mastery level S j Lower than the task difficulty δ k When S j With δ k As you get closer, the feedback intensity decreases.

[0043] Further, the feedback strength matrix Λ k Generate corresponding multi-dimensional feedback during the task execution process, including: visual feedback, tactile feedback and auditory feedback; the real-time recording of students' responses includes the following key indicators: number of errors e k , response time t k and operation adjustment times a k ; The feedback response vector is expressed as follows:

[0044] R k =[e k tk a k ] T

[0045] Among them, e k Represents task p k The number of errors that occurred in k is the average response time of students in the task, a k To adjust the times, record how often students respond to feedback during the operation.

[0046] Furthermore, the S4 specifically includes:

[0047] The feedback response sequence R seq = {R1, R2, …, R k ,R m} as the input data for path optimization, it contains the key response information of each task, where each feedback response vector R k Including the number of errors k , response time t k and operation adjustment times a k ;

[0048] From the feedback response sequence R seq R seq Extract data and calculate the difficulty adjustment coefficient Δ for each task k , to achieve task p k In the optimization path P opt Dynamic adjustment in

[0049] According to the task difficulty parameter δ k , optimized task difficulty:

[0050] δ′ k =δ k +Δ k

[0051] Generate a new task sequence P for students opt ={p′1,p′2,…,p′ m}, where each task p′ k Corresponding to the new difficulty coefficient δ′ k At the same time, according to the difficulty of the new task, the system generates an updated feedback matrix F′ for the task k , to ensure that the intensity of feedback is appropriate to the difficulty of the task;

[0052] After the optimized path is executed, a learning progress report R is generated based on the student's performance report , summarize the optimization path P opt The completion status and output learning progress report R report .

[0053] Furthermore, the difficulty adjustment coefficient Δ k The calculation is as follows:

[0054]

[0055] Among them, γ, λ and θ are the weight coefficients of each feedback parameter, which are set through actual teaching experiment tuning; Represents a nonlinear regularization term, which is used to nonlinearly amplify the response time. When t k When it is at its highest, this term is closest to 1, making Δ k The value is the largest, thus reducing the difficulty of the task; when t k When it is at its lowest, this item is closest to 0, and the difficulty remains the same or even increases;

[0056] The feedback matrix F′ k , calculated as follows:

[0057] F′ k =α·F k +β·Reg(F k )

[0058] Among them, α controls the feedback matrix F k The update weight of β controls the regularization term Reg(F k ) to avoid unnecessary over-amplification of feedback strength during the update.

[0059] In a second aspect of the present invention, a teaching virtual simulation system based on mixed reality and artificial intelligence is provided, the system comprising:

[0060] A data acquisition unit is used to collect multimodal operation data of students through sensors and cameras in an MR environment, and use the multimodal operation data as raw data. After each sampling, an operation feature vector is generated, and the operation feature vector includes all operation data at time t. Finally, the operation feature vector is preprocessed to output the preprocessed feature stream and time series.

[0061] A personalized learning path generation unit is used to generate a personalized learning path based on feature streams and time series, so that the learning progress matches the cognitive level of the students;

[0062] A real-time adjustment unit is used to generate instant feedback for each operation step and record the student's response according to the personalized learning path, helping students to make real-time adjustments during the operation;

[0063] A real-time optimization unit is used to optimize students’ learning paths and generate learning progress reports based on responses, so that the learning paths form a loop adjustment mechanism;

[0064] The steps executed by the personalized learning path generation unit specifically include:

[0065] Construct a learning state evaluation model M based on feature streams and time series LS , used to quantify and evaluate the student's current learning status; wherein the learning status evaluation model M LS , which is expressed as follows:

[0066]

[0067] Among them, S j represents the learning state indicator, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the model's adjustment of weights to prevent excessive bias towards specific feature weights; among them, the learning state indicator S j It is set between [0,1], indicating the students' mastery of the current task. Values ​​closest to 1 indicate the highest mastery, and values ​​closest to 0 indicate the need for further intensive training.

[0068] Design task selection function f(S j ) According to the learning state indicator S j Generate personalized task sequence P = {p1, p2, ..., p m}, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria, as follows:

[0069]

[0070] Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning; among them, if the student's mastery level S j The highest, the difficulty of the task is automatically increased; if the mastery is lower than θ l , then reduce the difficulty of the task and provide more operation prompts to assist students in completing the task;

[0071] Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

[0072] The beneficial technical effects of the present invention are at least as follows:

[0073] The invention uses AI algorithms to conduct in-depth analysis of students' real-time performance to dynamically adjust the learning path to ensure that the students' learning process can fit their actual level and understanding ability. The AI ​​system will generate personalized learning paths based on data such as students' operation accuracy, dwell time, and error rate. When students encounter difficulties in a certain knowledge point or operation step, the system will automatically reduce the difficulty of the task and provide detailed demonstrations and guidance; when students show a high degree of mastery, the system will automatically generate more challenging tasks or further knowledge expansion. This ability to dynamically generate paths not only meets the learning needs of students at different levels, but also avoids students losing interest in learning due to too high or too low task difficulty. Compared with the fixed content design of traditional simulation systems, the present invention realizes a highly personalized teaching process through real-time path optimization.

[0074] The present invention introduces a multi-sensory interaction mechanism in a virtual simulation environment, using MR technology combined with multi-dimensional feedback such as vision, touch and hearing to provide students with a highly immersive learning experience. The system provides visual prompts (such as highlights or color changes), tactile feedback (such as force sensing) and sound simulation (such as ambient sound or alarm sound) in a timely manner according to the different operations and learning progress of the students to enhance the students' understanding of operations and actual perception. For example, in a surgical simulation, when a student's operation is wrong or not accurate enough, the system reminds the student through red highlight prompts and vibration feedback, and gives positive feedback through green visual feedback and sound rewards when the operation is correct. Through this multi-dimensional perception interaction system, the present invention not only enhances the students' immersive experience, but also effectively enhances the students' intuitive understanding of the operation, so that they can obtain a feel and reaction close to the real operation in a simulation environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0076] Figure 1 This is a flow chart of the teaching virtual simulation method based on mixed reality and artificial intelligence of the present invention. DETAILED DESCRIPTION

[0077] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0078] like Figure 1As shown, the teaching virtual simulation method based on mixed reality and artificial intelligence provided by the embodiment of the present invention includes:

[0079] S1. In the MR environment, the multimodal operation data of the students are collected through sensors and cameras, and the multimodal operation data are used as raw data. At the same time, after each sampling, an operation feature vector is generated, and the operation feature vector contains all the operation data at time t. Finally, the operation feature vector is preprocessed, and the preprocessed feature stream and time series are output;

[0080] S2, generate personalized learning paths based on feature streams and time series to match the learning progress with the students’ cognitive level;

[0081] S3. Generate instant feedback for each operation step and record the student's response according to the personalized learning path, so as to help students make real-time adjustments during the operation;

[0082] S4. Based on the response, optimize the students’ learning path and generate a learning progress report, so that the learning path forms a loop adjustment mechanism.

[0083] Specifically, the S1 specifically includes:

[0084] In the MR environment, the present invention first collects the multimodal operation data of students through sensors and cameras, including eye movement position, hand movement, operation frequency, dwell time, etc. Suppose the original data composed of these multimodal features is D raw ={d1,d2,…,d n}, where d i Represents the original values ​​collected by each sensor. To capture the time series characteristics, after each sampling, the system generates an operation feature vector X t , contains all the operation data at time t. For the multimodal features of students, the present invention sets different weight factors ω i , assign values ​​according to the importance of the data (such as eye movement position, gesture recognition, etc.). t The definition is as follows:

[0085]

[0086] Among them, ω i is the weight coefficient of different features, d i is the characteristic value. t is the feature vector of the student operation at time t, which is used for subsequent analysis. This process assigns different weights to data of different modes to meet the feature acquisition accuracy required by the patent.

[0087] Furthermore, to ensure the comparability of feature data, all X t Need to be standardized. Let μX and σ X is the sample characteristic mean and standard deviation, and the standardization formula is:

[0088]

[0089] After standardization, the sliding window method is used to obtain time series features. Assuming the window width is w, the feature set of the time period {t, t+1,…, t+w-1} is This sliding window operation captures the changing trends of various features in continuous actions and is particularly suitable for the generation of dynamic learning paths.

[0090] Furthermore, for the time series S t In order to reduce redundancy and highlight key features, an aggregation dimensionality reduction method is adopted. The present invention constructs a covariance matrix C:

[0091]

[0092] Among them, μ S is the feature mean vector within the window. Then, principal component analysis (PCA) is performed on C to obtain the feature vector X after dimensionality reduction. PCA This operation keeps the feature dimension as large as possible while filtering out noise and redundant data. The final output is X PCA , as the standardized input of the feature data stream for the path generation model.

[0093] Furthermore, for each sliding window, the X PCA , forming a characteristic flow X stream is the feature change stream in continuous time, which will be used as the input of the learning path generation model.

[0094] Specifically, the S2 specifically includes

[0095] The feature stream output from step 1 As input data. Represents the reduced dimension feature vector at time point j. Feature flow X stream It records the changes in multimodal information of students during the operation process, reflects the students' operation trends and characteristics, and is particularly suitable for the needs of generating personalized learning paths in patent scenarios.

[0096] On receiving X stream After that, the system inputs it into the learning status evaluation model to capture the student's current operation status and mastery. This feature stream not only contains the operation features after dimensionality reduction, but also reflects the learning dynamics of students at different time points, which can provide rich contextual information for generating personalized learning paths.

[0097] Furthermore, in order to realize the personalized generation of dynamic learning paths, the system builds a patented customized learning status evaluation model M LS , which is used to quantify and evaluate the student’s current learning status. The model receives the feature vector at each time point And output the learning status indicator S j , which is used to indicate the students’ mastery level and learning effect at that point in time.

[0098] To ensure that the model accurately evaluates multimodal features, a dynamic weight matrix W is added to the model. The elements in this matrix are constantly adjusted according to time and operation behavior to better adapt to students' real-time learning situation. Specifically, S j The calculation formula is:

[0099]

[0100] Where W(t) is a dynamic weight matrix, which represents the weights of different features at time t. The change of t can be adaptively adjusted according to the student's operating habits or task completion. The regularization term α·Reg(W(t)) is used to control the model's adjustment of weights to prevent excessive bias in the weights of specific features. α is the regularization coefficient. S j The value of is set between [0,1], indicating the student's mastery of the current task: S j A value close to 1 indicates a high level of mastery, while a value close to 0 indicates that further intensive training is needed. j It will serve as the key basis for subsequent path selection.

[0101] Furthermore, according to the students’ real-time mastery of j , the system will generate a personalized task sequence P = {p1, p2, ..., p m}, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria. This task sequence generation model takes into account the personalized learning needs of patents. j The value of dynamically selects the appropriate task content.

[0102] In order to ensure that the learning task is consistent with the student’s current mastery, the task selection function f(S j ) is designed as follows:

[0103]

[0104] Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning. j Higher (such as greater than θ h), the system will automatically increase the difficulty of the task (for example, from the basic operation task p k Turning to more difficult reasoning tasks k+1 ); if the mastery is lower than θ l , the task difficulty is reduced and more operation prompts are provided to assist students in completing the task. This task sequence ensures that each step of the student's learning task matches the current level of mastery.

[0105] Furthermore, the system finally outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task. This path P constitutes the student's dynamic learning process, and the difficulty of each task is based on the student's real-time mastery. The learning path P provides clear task requirements and standards for subsequent multi-dimensional feedback generation, ensuring that students have targeted task guidance and feedback tracking for each step of operation in the MR environment.

[0106] Specifically, the S3 specifically includes

[0107] After receiving the personalized learning path P generated in the previous step, m}, the system enters each task p k , and generate multi-dimensional feedback combinations according to the operational requirements of the task. These feedback combinations (such as vision, touch, and hearing) are used to provide students with multi-sensory guidance and prompts in the MR scene. To this end, the feedback adaptation function φ(p k ), select the optimal feedback combination according to task requirements and operation complexity and generate the feedback feature matrix F k Among them, F k The composition is:

[0108] F k =φ(p k )=[f vis f tac f aud ] T

[0109] Among them, f vis 、f tac and f aud They represent the weight parameters of visual, tactile and auditory feedback, respectively, and are used to set the weight of different types of feedback in a specific task p k For example, in a complex surgical operation, F k May set tactile feedback f tac The weight of is higher, while the visual feedback f vis As a secondary item, to ensure that the focus of feedback meets the task requirements. k Real-time adjustments can be made when switching tasks to ensure the adaptability of each feedback.

[0110] Furthermore, after determining the feedback type combination, the system needs to further control the intensity of the feedback. To achieve this, the system k Construct the feedback strength matrix Λ k , to adapt to the student's performance in the current task. k Not only does it take into account the feedback needs of the current task, but it also introduces the student mastery level S j and the task difficulty parameter δ k , ensuring the adaptability of feedback. The feedback strength generation formula is defined as follows:

[0111] Λ k =F k ·σ(S j -δ k )+β·Reg(F k )

[0112] Where σ(·) is an activation function (such as ReLU or sigmoid) used to normalize the range of the intensity matrix; β·Reg(F k is a regularization term to prevent the system from over-relying on a certain feedback, and β is a regularization coefficient. k , the system controls the intensity and triggering conditions of feedback. For example, when the student's mastery level S j Significantly lower than the task difficulty δ k When S j With δ k When the learning state of the students is close, the feedback intensity is weakened. This dynamic adjustment mechanism ensures that the feedback is highly adaptable to the students' current learning state and avoids feedback that is too frequent or too sparse.

[0113] Furthermore, based on the feedback strength matrix Λ k , the system generates corresponding multi-dimensional feedback during the task execution process, including:

[0114] A. Visual feedback: Use color highlighting, direction indication, etc. in the MR scene to help students understand the operation position and direction. The visual feedback intensity f vis Determines the degree and frequency of highlighting.

[0115] B. Tactile feedback: Use the MR handle to generate vibration or pressure to give students tactile feedback, such as k When force control is needed during the test, tactile feedback can remind students whether the force is insufficient or too strong.

[0116] C. Auditory feedback: Play prompt sounds or voice guidance to help students identify key operation nodes. Auditory feedback intensity f aud Control the volume and repetition rate of the reminder.

[0117] After each feedback is generated, the system records the student's response in real time, including the following key indicators: number of errors e k , response time t k and operation adjustment times a k . Record these data as the feedback response vector R k , defined as:

[0118] R k =[e k t k a k ] T

[0119] Among them, e k Represents task p k The number of errors that occurred in k is the average response time of students in the task, a k To adjust the times, record how often students respond to feedback during the operation.

[0120] Furthermore, after completing all tasks in the task sequence P, the system converts all feedback response vectors R k Combined into feedback response sequence R seq = {R1, R2, …, R m This sequence records the student's feedback response data throughout the entire learning path and serves as the input data stream for the next step of path optimization. seq Detailed information such as task completion status, operation adjustment behavior, and response characteristics are provided so that the system can adjust the learning path in a targeted manner in the next stage to ensure that students receive more appropriate feedback intensity and task difficulty in the next task.

[0121] Specifically, the S4 specifically includes

[0122] The feedback response sequence R output from step 3 seq = {R1, R2, …, R m} as the input data for path optimization, it contains the key response information of each task. Each feedback response vector R k It is composed of the following elements: k : Error rate, which refers to the error rate of students in task p k Frequency of operation errors in; t k : Average response time, indicating the students’ response speed in the task; a k : The number of operation adjustments refers to the frequency with which students correct their operations during the task.

[0123] This feedback response sequence provides detailed data on real-time learning performance for the system's path optimization. Through feedback data-driven path optimization, the system can identify students' weak links in operation, especially tasks with many errors or slow operation responses.

[0124] Furthermore, the system responds to the feedback sequence R seq Extract data and calculate the difficulty adjustment coefficient Δ for each task k , to achieve task p k In the optimization path P opt Dynamic adjustment in. Define the difficulty adjustment coefficient Δ k The calculation formula is as follows:

[0125]

[0126] Among them, γ, λ and θ are the weight coefficients of each feedback parameter, which are set through actual teaching experiment tuning: γ: control error rate e k The impact of ensuring that when students are on task p k When the error rate is high, the difficulty is appropriately reduced; λ: control response time t k θ: the number of control operation adjustments a k When students make adjustments frequently during a task, it means that the task difficulty should be reduced to adapt to the students' learning progress.

[0127] Among them, the formula The term is used to perform nonlinear amplification on the response time. k When it is high (i.e., the student's reaction time is long), this term is close to 1, making Δ k The value increases, thereby reducing the difficulty of the task; when t k When it is low (i.e. the student reacts quickly), this item approaches 0, and the difficulty remains unchanged or even increases. Through this adjustment, the system can generate an optimized path of adaptive difficulty for students with different operating performances.

[0128] Furthermore, using the optimized task difficulty δ′ k =δ k +Δ k , the system generates a new task sequence P for students opt ={p′1,p′2,…,p′ m}, each task p′ k Corresponding to the new difficulty coefficient δ′ k At the same time, according to the difficulty of the new task, the system generates an updated feedback matrix F′ for the task k , to ensure that the feedback intensity is consistent with the task difficulty. Define the optimized feedback matrix F′ k as follows:

[0129] F′ k =α·F k +β·Reg(F k )

[0130] Among them, α controls the feedback matrix F k The update weight of β controls the regularization term Reg(F k ) to avoid unnecessary over-amplification of feedback strength during the update. k Based on the balance between task difficulty and feedback intensity, corresponding visual, tactile and auditory feedback is generated in actual operations to ensure that the feedback remains effective in the optimization task.

[0131] Furthermore, after the optimized path is executed, the system generates a learning progress report R based on the student's performance. report , summarize the optimization path P opt The core data in the report include the completion status of each task, average response time, error rate and number of adjustments, etc., which are used to analyze students' learning progress. report The output helps teachers and students review the learning process and serves as a basis for further optimizing the path and feedback settings.

[0132] Finally, R report It not only records students’ performance in the optimized path, but also provides data support for future path generation, achieving closed-loop improvement of path optimization and feedback adjustment.

[0133] The embodiment of the present invention further provides a teaching virtual simulation system based on mixed reality and artificial intelligence, the system comprising:

[0134] A data acquisition unit is used to collect multimodal operation data of students through sensors and cameras in an MR environment, and use the multimodal operation data as raw data. After each sampling, an operation feature vector is generated, and the operation feature vector includes all operation data at time t. Finally, the operation feature vector is preprocessed to output the preprocessed feature stream and time series.

[0135] A personalized learning path generation unit is used to generate a personalized learning path based on feature streams and time series, so that the learning progress matches the cognitive level of the students;

[0136] A real-time adjustment unit is used to generate instant feedback for each operation step and record the student's response according to the personalized learning path, helping students to make real-time adjustments during the operation;

[0137] A real-time optimization unit is used to optimize students’ learning paths and generate learning progress reports based on responses, so that the learning paths form a loop adjustment mechanism;

[0138] The steps executed by the personalized learning path generation unit specifically include:

[0139] Construct a learning state evaluation model M based on feature streams and time series LS , used to quantify and evaluate the student's current learning status; wherein the learning status evaluation model M LS , which is expressed as follows:

[0140]

[0141] Among them, S j represents the learning state indicator, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the model's adjustment of weights to prevent excessive bias towards specific feature weights; among them, the learning state indicator S j It is set between [0,1], indicating the students' mastery of the current task. Values ​​closest to 1 indicate the highest mastery, and values ​​closest to 0 indicate the need for further intensive training.

[0142] Design task selection function f(S j ) According to the learning state indicator S j Generate personalized task sequence P = {p1, p2, ..., p m}, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria, as follows:

[0143]

[0144] Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning; among them, if the student's mastery level S j The highest, the difficulty of the task is automatically increased; if the mastery is lower than θ l , then reduce the difficulty of the task and provide more operation prompts to assist students in completing the task;

[0145] Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a division of logical functions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0148] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

[0149] Although embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A teaching virtual simulation method based on mixed reality and artificial intelligence, characterized in that: The method comprises the following steps: S1. In the MR environment, the multimodal operation data of the students are collected through sensors and cameras, and the multimodal operation data are used as raw data. At the same time, after each sampling, an operation feature vector is generated, and the operation feature vector contains all the operation data at time t. Finally, the operation feature vector is preprocessed, and the preprocessed feature stream and time series are output; S2, generate personalized learning paths based on feature streams and time series to match the learning progress with the students’ cognitive level; S3. Generate instant feedback for each operation step and record the student's response according to the personalized learning path, so as to help students make real-time adjustments during the operation; S4. According to the response situation, optimize the student's learning path and generate a learning progress report, so that the learning path forms a loop adjustment mechanism; The S2 specifically includes: Construct a learning state evaluation model M based on feature streams and time series LS , used to quantify and evaluate the student's current learning status; wherein the learning status evaluation model M LS , which is expressed as follows: Among them, S j represents the learning state indicator, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the model's adjustment of weights to prevent excessive bias towards specific feature weights; among them, the learning state indicator S j It is set between [0,1], indicating the students' mastery of the current task. Values ​​closest to 1 indicate the highest mastery, and values ​​closest to 0 indicate the need for further intensive training. Design task selection function f(S j ) According to the learning state indicator S j Generate personalized task sequence P = {p1, p2, ..., p m }, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria, as follows: Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning; among them, if the student's mastery level S j The highest, the difficulty of the task is automatically increased; if the mastery is lower than θ l , the task difficulty is reduced and more operation prompts are provided to assist students in completing the task; Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

2. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 1 is characterized in that: The multimodal operation data includes eye movement position, hand movement, operation frequency and dwell time.

3. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 1 is characterized in that: The pre-processing specifically includes: Let the original data be D raw ={d1,d2,…,d n }, where d i represents the original values ​​collected by each sensor, and the operational feature vector is set to X t , which is expressed as follows: Among them, ω i is the weight coefficient of different features, d i is the feature value, n represents the total number of features, and i represents the current feature; For all operational feature vectors X t For standardization, let μ X and σ X is the sample characteristic mean and standard deviation, and the standardization formula is: After standardization, the sliding window method is used to obtain time series features; Assuming the window width is w, the feature set of the time period {t, t+1,…, t+w-1} is For the time series S t In order to reduce redundancy and highlight key features, an aggregation dimensionality reduction method is adopted. The present invention constructs a covariance matrix C: Among them, μ S is the feature mean vector within the window; perform principal component analysis on C to obtain the feature vector X after dimensionality reduction PCA ; X calculated for each sliding window PCA , forming a characteristic flow X stream It is the characteristic change flow in continuous time.

4. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 1 is characterized in that: The S3 specifically includes: According to the personalized learning path, combined with each task p k , and generate multi-dimensional feedback combinations according to the operational requirements of the task and construct the feedback feature matrix F according to the feedback combinations k , the feedback combination is used to provide multi-sensory guidance and prompts to students in the MR scene; According to the feedback feature matrix F k , introduced the student mastery S j and the task difficulty parameter δ k Construct the feedback strength matrix Λ k , adapted to the student's performance in the current task, is expressed as follows: L k =F k ·σ(S j -d k )+β·Reg(F k ) Among them, σ(·) is the activation function, which is used to normalize the value range of the intensity matrix; β·Reg(F k ) is a regularization term to prevent the system from over-relying on a certain feedback, and β is a regularization coefficient; According to the feedback strength matrix Λ k Generate corresponding multi-dimensional feedback during the task execution process. At the same time, after each feedback is generated, the student's response is recorded in real time as the feedback response vector R k ; After completing all tasks in the task sequence P, all feedback response vectors R k Combined into feedback response sequence R seq = {R1, R2, …, R m }.

5. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4 is characterized in that: The feedback combination includes vision, touch and hearing; the feedback feature matrix F k , which is expressed as follows: F k =φ(p k )=[f vis f tac f aud ] T Among them, f vis 、f tac and f aud They represent the weight parameters of visual, tactile and auditory feedback, which are used to set the weight of different types of feedback in a specific task p. k The importance of φ(p k ) represents the feedback adaptation function, which represents the task requirements and operation complexity to select the optimal feedback combination and generate the feedback feature matrix F k .

6. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4 is characterized in that: According to the feedback strength matrix Λ k Control the intensity and triggering conditions of feedback, specifically: when the student's mastery level S j Lower than the task difficulty δ k When S j With δ k As you get closer, the feedback intensity decreases.

7. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4 is characterized in that: The feedback strength matrix Λ k Generate corresponding multi-dimensional feedback during the task execution process, including: visual feedback, tactile feedback and auditory feedback; the real-time recording of students' responses includes the following key indicators: number of errors e k , response time t k and operation adjustment times a k ; The feedback response vector is expressed as follows: R k =[e k t k a k ] T Among them, e k Represents task p k The number of errors that occurred in k is the average response time of students in the task, a k To adjust the times, record how often students respond to feedback during the operation.

8. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 4 is characterized in that: The S4 specifically includes: The feedback response sequence R seq = {R1, R2, …, R k ,R m } as the input data for path optimization, it contains the key response information of each task, where each feedback response vector R k Including the number of errors k , response time t k and operation adjustment times a k ; From the feedback response sequence R seq R seq Extract data and calculate the difficulty adjustment coefficient Δ for each task k , to achieve task p k In the optimization path P opt Dynamic adjustment in According to the task difficulty parameter δ k , optimized task difficulty: d′ k =d k +D k Generate a new task sequence P for students opt ={p′1,p′2,…,p′ m }, where each task p′ k Corresponding to the new difficulty coefficient δ′ k At the same time, according to the difficulty of the new task, the system generates an updated feedback matrix F for the task k , to ensure that the intensity of feedback is appropriate to the difficulty of the task; After the optimized path is executed, a learning progress report R is generated based on the student's performance report , summarize the optimization path P opt The completion status and output learning progress report R report .

9. The teaching virtual simulation method based on mixed reality and artificial intelligence according to claim 8 is characterized in that: The difficulty adjustment factor Δ k The calculation is as follows: Among them, γ, λ and θ are the weight coefficients of each feedback parameter, which are set through actual teaching experiment tuning; Represents a nonlinear regularization term, which is used to nonlinearly amplify the response time. When t k When it is at its highest, this term is closest to 1, making Δ k The value is the largest, thus reducing the difficulty of the task; when t k When it is at its lowest, this item is closest to 0, and the difficulty remains the same or even increases; The feedback matrix F′ k , calculated as follows: F′ k =α·F k +β·Reg(F k ) Among them, α controls the feedback matrix F k The update weight of β controls the regularization term Reg(F k ) to avoid unnecessary over-amplification of feedback strength during the update.

10. A teaching virtual simulation system based on mixed reality and artificial intelligence, characterized in that: The system comprises: A data acquisition unit is used to collect multimodal operation data of students through sensors and cameras in an MR environment, and use the multimodal operation data as raw data. After each sampling, an operation feature vector is generated, and the operation feature vector includes all operation data at time t. Finally, the operation feature vector is preprocessed to output the preprocessed feature stream and time series. A personalized learning path generation unit is used to generate a personalized learning path based on feature streams and time series, so that the learning progress matches the cognitive level of the students; A real-time adjustment unit is used to generate instant feedback for each operation step and record the student's response according to the personalized learning path, helping students to make real-time adjustments during the operation; A real-time optimization unit is used to optimize students’ learning paths and generate learning progress reports based on responses, so that the learning paths form a loop adjustment mechanism; The steps executed by the personalized learning path generation unit specifically include: Construct a learning state evaluation model M based on feature streams and time series LS , used to quantify and evaluate the student's current learning status; wherein the learning status evaluation model M LS , which is expressed as follows: Among them, S j represents the learning state indicator, W(t) is the dynamic weight matrix, which represents the weight of different features at time t; represents the feature vector at each time point, α represents the regularization coefficient; Reg(W(t)) represents the regularization term, which is used to control the model's adjustment of weights to prevent excessive bias towards specific feature weights; among them, the learning state indicator S j It is set between [0,1], indicating the students' mastery of the current task. Values ​​closest to 1 indicate the highest mastery, and values ​​closest to 0 indicate the need for further intensive training. Design task selection function f(S j ) According to the learning state indicator S j Generate personalized task sequence P = {p1, p2, ..., p m }, where p k Represents each task in the learning path, including task objectives, completion conditions, and feedback criteria, as follows: Among them, θ h With θ l are the upper and lower thresholds respectively, which are determined through experimental tuning; among them, if the student's mastery level S j The highest, the difficulty of the task is automatically increased; if the mastery is lower than θ l , the task difficulty is reduced and more operation prompts are provided to assist students in completing the task; Finally, the system outputs a personalized learning path P, including the content, difficulty, completion criteria, and feedback specifications of each task.

Images