A virtual reality immersive simulation training method and system based on digital twinning

Abstract

This invention discloses a virtual reality immersive simulation training method and system based on digital twins, belonging to the field of training simulation technology, including the following steps: S1, collecting multi-source training data and preprocessing it; S2, performing dimensionality-upgrading mapping based on response differences, solving state transition relationships and constraint boundaries, and constructing a training twin model; S3, performing cross-segment state evolution learning, forward recursion along the operation trigger direction, and inverse reconstruction, updating the state transition relationships and constraint boundaries; S4, performing state propagation, performing Kalman filtering correction based on observation bias, and recursively updating the current training state; S5, constructing matching costs, locating deviation nodes, and adjusting the immersive feedback strategy; S6, rolling back the current training state, obtaining retraining feedback data, and updating the state transition relationships and constraint boundaries. This invention can realize the synchronous deduction of training operations and state responses, improving the accuracy of virtual reality simulation training.

Description

Technical Field

[0001] This invention relates to the field of training and simulation technology, and in particular to a virtual reality immersive simulation training method and system based on digital twins. Background Technology

[0002] With the development of virtual reality and digital twin technologies, simulation training is no longer limited to simple video teaching or process demonstrations, but is gradually moving towards immersive interaction, process recording, and training feedback. In training scenarios such as medical emergency care, nursing operations, and handling of complex equipment, trainees need to complete continuous operations in a virtual environment. The system then provides prompts or evaluations based on the operation process and status changes, thereby reducing the cost of real training and improving training safety and the efficiency of repeated training.

[0003] Existing virtual reality training systems typically provide 3D scene display, standardized process guidance, and evaluation of operational results. Some systems can also record the sequence of trainees' actions, completion time, and scores. However, most of these systems rely on preset scripts or fixed rules for feedback. Changes in the virtual scene often proceed according to a predetermined process, making it difficult to continuously extrapolate from the trainee's actual operation. When trainees exhibit inconsistent operating rhythms, repetitive steps, delayed state responses, or mid-process corrections, the system often only provides simple prompts, failing to accurately reflect the true changes in the training state.

[0004] Meanwhile, immersive training generates various types of data, including operation trigger records, state response records, and evaluation feedback records. These data typically come from different sources and have different sampling rhythms. Without unified temporal organization and state correlation processing, the training process records can easily become fragmented. Existing methods focus more on whether a single operation is completed, paying less attention to the connections between preceding and subsequent operations and the continuous changes in state responses. Therefore, in training debriefing and capability assessment, problems such as rough evaluations and inaccurate positioning can easily occur.

[0005] Furthermore, existing training feedback often remains at the level of result scoring or error prompts, lacking detailed analysis of the differences between the trainee's training process and the standard training process. When deviations occur at a local point in continuous operation, the system struggles to determine whether the deviation is caused by the operation sequence, state response, or the accumulation of previous steps, making it difficult to adjust the feedback content and retraining location accordingly. Even when trainees undergo retraining, the system typically only re-records the results, lacking the ability to continuously revise the training state model based on retraining feedback.

[0006] Therefore, how to provide a virtual reality immersive simulation training method and system based on digital twins is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] One objective of this invention is to propose a virtual reality immersive simulation training method and system based on digital twins. This invention fully utilizes digital twin modeling, Koopman state mapping, unscented Kalman filtering, Sinkhorn path registration, and recursive least squares update techniques. It describes in detail the method of organizing multi-source training data into a training event sequence and performing state deduction, deviation location, immersive feedback, and retraining updates for the trainee's training process. It has the advantages of high training process fidelity, accurate real-time tracking, clear deviation location, and highly targeted feedback guidance.

[0008] According to an embodiment of the present invention, a virtual reality immersive simulation training method based on digital twins includes the following steps: S1. Collect multi-source training data and preprocess it to form a training event sequence; S2. Divide the training event sequence into state segments, perform Koopman up-dimensional mapping based on the response difference between the preceding and following state segments, solve the state transition relationship and constraint boundary, and construct the training twin model. S3. Perform cross-segment state evolution learning on the training event sequence, call the training twin model to perform forward recursion along the operation trigger direction, and perform reverse restoration along the state response feedback direction, and update the state transition relationship and constraint boundary according to the positive and negative closure deviation of the corresponding state segment. S4. Select unmarked sampling points around the current training state, call the updated state transition relationship to propagate the state of the unmarked sampling points, perform Kalman filter correction based on the observation deviation corresponding to the student's real-time operation, and simultaneously adjust the advancement step size to recursively update the current training state. S5. Construct the matching cost between the student state and the standard state, use the Sinkhorn iterative algorithm to solve the entropy regular registration relationship, and locate the deviation node based on the local cost transition in the registration relationship to adjust the immersion feedback strategy. S6. Roll back the current training state according to the deviation node, obtain the retraining feedback data of the immersion feedback strategy, and update the state transition relationship and constraint boundary using the recursive least squares method.

[0009] Optionally, the preprocessing specifically includes: The multi-source training data is parsed, and a unified index is established according to the training task identifier, operation trigger identifier and sampling time. The multi-source training data is an asynchronous training record formed by different sampling links. Before entering the parsing, the field mapping, time mark and task ownership mark are completed. The sampling interval of the data after unified indexing is reorganized, and the moving median filter is used to remove abnormal jump data to obtain continuous training records; The continuous training records are segmented into events based on the operation trigger identifier, and the segmented records are mapped into operation trigger quantities, status response quantities, and evaluation feedback quantities. The operation trigger quantity, status response quantity, and evaluation feedback quantity are arranged in a progressive order according to the sampling time, and a sequential relationship is established between adjacent items to form a training event sequence.

[0010] Optionally, S2 specifically includes: S21. Traverse the training event sequence according to the sampling time, calculate the trigger interval and response increment of adjacent events, set the segment boundary at the position where the trigger interval exceeds the set interval threshold and the direction of the response increment changes, and divide the continuous events between adjacent segment boundaries into state segments. S22. Extract the event values ​​within the state segments according to the sampling order, arrange them into segment state vectors, and use the Koopman dimension-upper operator to perform delayed embedding and basis function expansion on the segment state vectors to obtain the dimension-upper segment vectors. S23. Pair up-dimensional segment vectors according to the order of state segments, take the response difference as the transition error between up-dimensional segment vectors, use the singular value decomposition truncation method to solve the state transition relationship, and take the minimum transition error as the solution condition. S24. Call the state transition relation to recursively deduce the state segments, and statistically analyze the state value range, response difference range, and recursion deviation range in the recursion results. Determine the constraint boundary based on the overlapping interval of each range, and associate the state transition relation with the constraint boundary to construct a training twin model.

[0011] Optionally, S22 specifically includes: S221. Using 3 to 8 consecutive sampled items in the state segment vector as time delay objects, construct delay coordinates, and calculate the first-order difference and second-order difference of the delay coordinates to calibrate the rate of change of operation and the second-order change of response within the state segment. S222. Construct a covariance matrix for the delayed coordinates, first-order difference, and second-order difference, and then superimpose the covariance matrices on their diagonals. to After the perturbation term of the trace mean of the matrix is ​​doubled, Cholesky decomposition is used for whitening transformation, and the effective delay dimension is screened based on the mutual information value between each dimension after whitening and the response difference. S223. The Koopman dimension-upper operator is used to expand the effective delay dimension using basis functions. After linear expansion and cross-term expansion, 8 to 64 radial basis function center points are set, and the expanded candidate dimension-upper components are compressed to 64 to 512 dimensions. S224. Calculate the reduction of the projection residual of each candidate upscaling component to the response difference, and include the upscaling components in order of the reduction of the projection residual from large to small. Stop including after the cumulative reduction of the residual reaches 85% to 95%. Arrange the included upscaling components according to the sampling order of the state segments to obtain the upscaling segment vector.

[0012] Optionally, S3 specifically includes: S31. Select adjacent state segments according to the successive relationship of state segments, determine the operation triggering direction from the previous state segment to the next state segment, and determine the state response feedback direction from the next state segment to the previous state segment. S32. A bidirectional dynamic mode decomposition algorithm is adopted to simultaneously decompose adjacent state segments, fit forward evolution operators and backward restoration operators under the same modal basis, and couple the forward evolution operators, backward restoration operators and state transition relations, specifically including: Arrange adjacent state segments according to the direction of operation triggering, and arrange adjacent state segments in reverse according to the direction of state response feedback, so as to keep the state items at the same sampling position corresponding; Perform joint singular value decomposition on the forward and reverse arrangement results, select the same set of principal singular vectors as the modal basis, and map adjacent state segments to the modal basis; The forward evolution operator and the backward reduction operator are fitted under the modal basis. The forward evolution operator is used to constrain the evolution of the previous state segment to the next state segment, and the backward reduction operator is used to constrain the recovery of the next state segment to the previous state segment. The forward evolution operator and the inverse reduction operator are embedded as forward and inverse calibration terms into the state transition relation, and the evolution direction and reduction direction in the state transition relation are coupled according to the forward and inverse calibration terms. S33. Call the coupled state transition relationship, continuously recursively deduce the previous state segment along the operation trigger direction, and call the reverse restoration operator to continuously invert the next state segment along the state response feedback direction. S34. Based on the continuous recursion results, continuous inversion results and the closure difference between the corresponding state segments, adjust the coupling coefficient between the forward evolution operator and the inverse restoration operator, and update the state transition relationship and constraint boundary synchronously.

[0013] Optionally, S4 specifically includes: S41. Determine the number of unscented sampling points according to the state dimension of the current training state. Using the current training state as the center sampling point, generate forward offset sampling points and reverse offset sampling points along each state dimension based on the constraint boundaries. Simultaneously, configure the sampling weights, where: The unmarked sampling point consists of a center sampling point, a forward offset sampling point, and a reverse offset sampling point, with the center sampling point representing the current training state. The forward offset sampling point and the reverse offset sampling point are respectively the sampling points formed by the forward and reverse offsets of the current training state along the corresponding state dimension within the constraint boundary; S42. Call the updated state transition relationship, propagate the unmarked sampling points one by one according to the current progress step size, perform boundary back pressure on the sampling points that exceed the constraint boundary after propagation, and perform weighted recycling of the propagated sampling points according to the sampling weight to obtain the predicted state. S43. Based on the state item corresponding to the student's real-time operation, perform a difference calculation with the corresponding state item in the predicted state to obtain the observation deviation. Construct a Kalman gain based on the discrete quantity of the unmarked sampling points and the observation deviation, and apply the Kalman gain to the observation deviation to correct the predicted state. S44. Based on the continuous change of the observation deviation and the distance of the predicted state after filtering correction, the remaining amount of the constraint boundary is scaled to advance the step size, the predicted state after filtering correction is updated to the current training state, and the process continues to advance according to the scaled step size.

[0014] Optionally, S5 specifically includes: S51. Extract the student state and standard state along the evolution path of the current training state, assign measurement weights to different state items according to the constraint boundary, and use the Mahalanobis measurement method to measure the state offset between the student state and the standard state, and combine the path order offset to form a matching cost matrix. S52. Perform entropy regular exponential mapping on the matching cost matrix to obtain the initial registration weights. Then, use the Sinkhorn iterative algorithm to perform row normalization and column normalization on the initial registration weights until the registration weight changes in two adjacent rounds satisfy the convergence condition, and obtain the entropy regular registration relationship. S53. Traverse the entropy regular registration relationship along the evolution path of the current training state. In the registration weights corresponding to each student state, select the standard state position with the largest weight as the main registration position, and mark the local cost transition amount according to the matching cost change rate corresponding to the adjacent main registration positions. S54. The main registration position where the local cost jump exceeds the jump threshold is determined as the deviation node. Based on the state offset direction and path position corresponding to the deviation node, the feedback triggering timing, feedback pointing state and rollback range in the immersion feedback strategy are adjusted.

[0015] Optionally, S52 specifically includes: S521. Apply negative exponential mapping to each item of the matching cost matrix using entropy regularization coefficients ranging from 0.01 to 0.20, and then apply the mapping results according to... to The lower limit truncation value is used to truncate the initial registration weights; S522. According to the order of student status in the evolution path, traverse the initial registration weights line by line, count the total weight of each line, and perform row normalization on each registration weight in the line with the corresponding total weight to obtain the row normalized weight. S523. According to the order of the standard states in the evolution path, traverse the row normalization weights column by column, count the total weight of each column, and perform column normalization on each registration weight in the column with the corresponding total weight to obtain the column normalization weights. S524. Using the column normalization weights as input for the next iteration, alternate between row normalization and column normalization, and compare the changes in column normalization weights between adjacent iterations: When the number of iterations reaches 20 to 80, and the change is less than to At that time, the normalization weight of the current column is locked to obtain the entropy regularization registration relationship.

[0016] Optionally, S6 specifically includes: S61. Based on the position of the deviation node in the evolution path of the current training state, roll back the current training state to the state corresponding to the deviation node, and start executing the immersion feedback strategy from the rolled-back state. S62. Collect retraining feedback data after implementing the immersion feedback strategy, pair the rolled-back state with the feedback state in the retraining feedback data, and obtain state update samples for updating state transition relationships, wherein: The retraining feedback data is the status response record generated during the re-execution of the training process after the deviation node is rolled back. The status response record is marked according to the status after rollback, the status after feedback, the direction of status change, and the feedback execution result. S63. Call the state transition relation to recursively recalculate the state update sample, use the residual between the recursive recalculation result and the state after feedback as the correction amount, and use the recursive least squares method to update the state transition relation. S64. Recursively recalculate the rolled-back state using the updated state transition relationship, and update the constraint boundaries based on the residual range after recursion.

[0017] According to an embodiment of the present invention, a virtual reality immersive simulation training system based on digital twins includes: The data processing module is used to collect multi-source training data and preprocess it to form a training event sequence; The twin construction module is used to divide the training event sequence into state segments, perform Koopman up-dimensional mapping based on the response difference between the previous and subsequent state segments, solve the state transition relationship and constraint boundary, and construct the training twin model. The state evolution module is used to learn the state evolution of training event sequences across segments. It calls the training twin model to perform forward recursion along the operation trigger direction and reverse reconstruction along the state response feedback direction. It updates the state transition relationship and constraint boundary based on the positive and negative closure deviations of the corresponding state segments. The real-time recursive module is used to select unscented sampling points around the current training state, call the updated state transition relationship to propagate the state of the unscented sampling points, perform Kalman filter correction based on the observation deviation corresponding to the student's real-time operation, and simultaneously adjust the advancement step size to recursively update the current training state. The registration and localization module is used to construct the matching cost between the student's state and the standard state. It uses the Sinkhorn iterative algorithm to solve the entropy regular registration relationship and locates the off-node based on the local cost jump in the registration relationship, and adjusts the immersion feedback strategy. The feedback update module is used to roll back the current training state according to the deviation node, obtain the retraining feedback data of the immersion feedback strategy, and update the state transition relationship and constraint boundary using the recursive least squares method.

[0018] The beneficial effects of this invention are: First, this invention organizes multi-source training data into a training event sequence and describes the continuous change relationship between operation triggers and state responses through a digital twin model. This avoids virtual training from relying solely on fixed scripts or single-point rule feedback, making the training process closer to the dynamic changes in real training.

[0019] Secondly, this invention continuously matches the trainee's training process with the standard training process through state recursion, filtering correction, and path registration. It can accurately locate deviation nodes when there are inconsistencies in the operation rhythm, lags in the state response, or deviations in local steps, thereby improving the accuracy of training evaluation and the detail of process analysis.

[0020] Finally, this invention rolls back the current training state based on the deviation node and continuously updates the state transition relationship and constraint boundary in combination with retraining feedback data. This allows the immersive feedback to go beyond simple prompts or result scoring, and instead provide targeted guidance around specific deviation locations, thereby improving the retraining effect and the adaptive capability of the simulation training system. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is an overall flowchart of a virtual reality immersive simulation training method based on digital twins proposed in this invention; Figure 2This is a flowchart illustrating the cross-segment state evolution learning process of a virtual reality immersive simulation training method based on digital twins proposed in this invention. Figure 3 This is a module structure diagram of a virtual reality immersive simulation training system based on digital twins proposed in this invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0023] refer to Figures 1-2 A virtual reality immersive simulation training method based on digital twins includes the following steps: S1. Collect multi-source training data and preprocess it to form a training event sequence; S2. Divide the training event sequence into state segments, perform Koopman up-dimensional mapping based on the response difference between the preceding and following state segments, solve the state transition relationship and constraint boundary, and construct the training twin model. S3. Perform cross-segment state evolution learning on the training event sequence, call the training twin model to perform forward recursion along the operation trigger direction, and perform reverse restoration along the state response feedback direction, and update the state transition relationship and constraint boundary according to the positive and negative closure deviation of the corresponding state segment. S4. Select unmarked sampling points around the current training state, call the updated state transition relationship to propagate the state of the unmarked sampling points, perform Kalman filter correction based on the observation deviation corresponding to the student's real-time operation, and simultaneously adjust the advancement step size to recursively update the current training state. S5. Construct the matching cost between the student state and the standard state, use the Sinkhorn iterative algorithm to solve the entropy regular registration relationship, and locate the deviation node based on the local cost transition in the registration relationship to adjust the immersion feedback strategy. S6. Roll back the current training state according to the deviation node, obtain the retraining feedback data of the immersion feedback strategy, and update the state transition relationship and constraint boundary using the recursive least squares method.

[0024] In this embodiment, the preprocessing specifically includes: The field parsing of multi-source training data is performed, and a unified index is established according to the training task identifier, operation trigger identifier and sampling time. The multi-source training data is an asynchronous training record formed by different sampling links. Field mapping, time stamping and task ownership mark are completed before entering the parsing. The sampling interval of the data after unified indexing is reorganized, and the moving median filter is used to remove abnormal jump data to obtain continuous training records; The continuous training records are segmented into events based on the operation trigger identifier, and the segmented records are mapped into operation trigger quantities, status response quantities, and evaluation feedback quantities. The operation trigger quantity, status response quantity, and evaluation feedback quantity are arranged in a progressive manner according to the sampling time, and a sequential relationship is established between adjacent items to form a training event sequence. In this embodiment, the multi-source training data is reorganized according to a sampling interval of 20ms to 100ms, and data with a sampling time offset of no more than 50ms are grouped into the same sampling time. The window length of the sliding median filter is set to 3 to 7 sampling points. When the change in the value of adjacent states exceeds 3 times the average change in the value of the same type of state, the corresponding data is marked as abnormal jump data. When segmenting events, if the time interval between adjacent operation triggers exceeds 0.5s to 1.5s, or if the direction of change of the state response quantity changes, event boundaries are set.

[0025] In this embodiment, S2 specifically includes: S21. Traverse the training event sequence according to the sampling time, calculate the trigger interval and response increment of adjacent events, set the segment boundary at the position where the trigger interval exceeds the set interval threshold and the direction of the response increment changes, and divide the continuous events between adjacent segment boundaries into state segments. S22. Extract the event values ​​within the state segments according to the sampling order, arrange them into segment state vectors, and use the Koopman dimension-upper operator to perform delayed embedding and basis function expansion on the segment state vectors to obtain the dimension-upper segment vectors. S23. Pair up-dimensional segment vectors according to the sequential order of state segments, use the response difference as the transition error between the preceding and following up-dimensional segment vectors, and solve the state transition relationship using the singular value decomposition truncation method, taking the minimum transition error as the solution condition. Specifically, this includes: Pair up-order segment vectors according to the order of state segments, and arrange the preceding and following up-order segment vectors in the pair into preceding and following up-order matrices, respectively. Based on the response difference between the preceding and following state segments, the transition residual from the preceding up-dimensional matrix to the following up-dimensional matrix is ​​calculated, and the degree of deviation between the transition residual and the response difference is taken as the transition error. Perform singular value decomposition on the preceding dimension-upgrading matrix, retain the principal singular values ​​in descending order of singular values, and construct a truncated pseudo-inverse matrix from the retained principal singular values ​​and their corresponding singular vectors; Solve the state transition relationship by multiplying the subsequent upgraded matrix with the truncated pseudo-inverse matrix, and adjust the singular value truncation range according to the transition error until the transition error reaches its minimum. S24. Call the state transition relation to recursively deduce the state segments, and statistically analyze the state value range, response difference range, and recursion deviation range in the recursion results. Determine the constraint boundary based on the overlapping interval of each range, and associate the state transition relation with the constraint boundary to construct a training twin model.

[0026] In this embodiment, S22 specifically includes: S221. Using 3 to 8 consecutive sampled items in the state segment vector as time delay objects, construct delay coordinates, and calculate the first-order difference and second-order difference of the delay coordinates to calibrate the rate of change of operation and the second-order change of response within the state segment. S222. Construct a covariance matrix for the delayed coordinates, first-order difference, and second-order difference, and then superimpose the covariance matrices on their diagonals. to After the perturbation term of the trace mean of the matrix is ​​doubled, Cholesky decomposition is used for whitening transformation, and the effective delay dimension is screened based on the mutual information value between each dimension after whitening and the response difference. S223. The Koopman dimension-upper operator is used to expand the effective delay dimension using basis functions. After linear expansion and cross-term expansion, 8 to 64 radial basis function center points are set, and the expanded candidate dimension-upper components are compressed to 64 to 512 dimensions. S224. Calculate the reduction of the projection residual of each candidate upscaling component to the response difference, and include the upscaling components in order of the reduction of the projection residual from large to small. Stop including after the cumulative reduction of the residual reaches 85% to 95%. Arrange the included upscaling components according to the sampling order of the state segments to obtain the upscaling segment vector.

[0027] In this embodiment, S3 specifically includes: S31. Select adjacent state segments according to the successive relationship of state segments, determine the operation triggering direction from the previous state segment to the next state segment, and determine the state response feedback direction from the next state segment to the previous state segment. S32. A bidirectional dynamic mode decomposition algorithm is adopted to simultaneously decompose adjacent state segments, fit forward evolution operators and backward restoration operators under the same modal basis, and couple the forward evolution operators, backward restoration operators and state transition relations, specifically including: Arrange adjacent state segments according to the direction of operation triggering, and arrange adjacent state segments in reverse according to the direction of state response feedback, so as to keep the state items at the same sampling position corresponding; Perform joint singular value decomposition on the forward and reverse arrangement results, select the same set of principal singular vectors as the modal basis, and map adjacent state segments to the modal basis; The forward evolution operator and the backward reduction operator are fitted under the modal basis. The forward evolution operator is used to constrain the evolution of the previous state segment to the next state segment, and the backward reduction operator is used to constrain the recovery of the next state segment to the previous state segment. The forward evolution operator and the inverse reduction operator are embedded as forward and inverse calibration terms into the state transition relation, and the evolution direction and reduction direction in the state transition relation are coupled according to the forward and inverse calibration terms. S33. Call the coupled state transition relationship, continuously recursively deduce the previous state segment along the operation trigger direction, and call the reverse restoration operator to continuously invert the next state segment along the state response feedback direction. S34. Based on the continuous recursion results, continuous inversion results and the closure difference between the corresponding state segments, adjust the coupling coefficient between the forward evolution operator and the inverse restoration operator, and update the state transition relationship and constraint boundary synchronously.

[0028] In this embodiment, S4 specifically includes: S41. Determine the number of unscented sampling points according to the state dimension of the current training state. Using the current training state as the center sampling point, generate forward offset sampling points and reverse offset sampling points along each state dimension based on the constraint boundaries. Simultaneously, configure the sampling weights, where: The unmarked sampling point consists of a center sampling point, a forward offset sampling point, and a reverse offset sampling point. The center sampling point represents the current training state. The forward offset sampling point and the reverse offset sampling point are respectively the sampling points formed by the forward and reverse offsets of the current training state along the corresponding state dimension within the constraint boundary; S42. Call the updated state transition relationship, propagate the unmarked sampling points one by one according to the current progress step size, perform boundary back pressure on the sampling points that exceed the constraint boundary after propagation, and perform weighted recycling of the propagated sampling points according to the sampling weight to obtain the predicted state. S43. Based on the state item corresponding to the student's real-time operation, perform a difference calculation with the corresponding state item in the predicted state to obtain the observation deviation. Construct a Kalman gain based on the discrete quantity of the unmarked sampling points and the observation deviation, and apply the Kalman gain to the observation deviation to correct the predicted state. S44. Based on the continuous change of the observation deviation and the distance of the predicted state after filtering correction, the remaining amount of the constraint boundary is scaled to advance the step size, the predicted state after filtering correction is updated to the current training state, and the process continues to advance according to the scaled step size.

[0029] In this embodiment, S5 specifically includes: S51. Extract the student state and standard state along the evolution path of the current training state, assign measurement weights to different state items according to the constraint boundary, and use the Mahalanobis measurement method to measure the state offset between the student state and the standard state, and combine the path order offset to form a matching cost matrix. S52. Perform entropy regular exponential mapping on the matching cost matrix to obtain the initial registration weights. Then, use the Sinkhorn iterative algorithm to perform row normalization and column normalization on the initial registration weights until the registration weight changes in two adjacent rounds satisfy the convergence condition, and obtain the entropy regular registration relationship. S53. Traverse the entropy regular registration relationship along the evolution path of the current training state. In the registration weights corresponding to each student state, select the standard state position with the largest weight as the main registration position, and mark the local cost transition amount according to the matching cost change rate corresponding to the adjacent main registration positions. S54. The main registration position where the local cost jump exceeds the jump threshold is determined as the deviation node. Based on the state offset direction and path position corresponding to the deviation node, the feedback triggering timing, feedback pointing state and rollback range in the immersion feedback strategy are adjusted.

[0030] In this embodiment, S52 specifically includes: S521. Apply negative exponential mapping to each item of the matching cost matrix using entropy regularization coefficients ranging from 0.01 to 0.20, and then apply the mapping results according to... to The lower limit truncation value is used to truncate the initial registration weights; S522. According to the order of student status in the evolution path, traverse the initial registration weights line by line, count the total weight of each line, and perform row normalization on each registration weight in the line with the corresponding total weight to obtain the row normalized weight. S523. According to the order of the standard states in the evolution path, traverse the row normalization weights column by column, count the total weight of each column, and perform column normalization on each registration weight in the column with the corresponding total weight to obtain the column normalization weights. S524. Using the column normalization weights as input for the next iteration, alternate between row normalization and column normalization, and compare the changes in column normalization weights between adjacent iterations: When the number of iterations reaches 20 to 80, and the change is less than to At that time, the normalization weight of the current column is locked to obtain the entropy regularization registration relationship.

[0031] In this embodiment, S6 specifically includes: S61. Based on the position of the deviation node in the evolution path of the current training state, roll back the current training state to the state corresponding to the deviation node, and start executing the immersion feedback strategy from the rolled-back state. S62. Collect retraining feedback data after implementing the immersion feedback strategy, pair the rolled-back state with the feedback state in the retraining feedback data, and obtain state update samples for updating state transition relationships, wherein: The retraining feedback data consists of the status response records generated during the re-execution of the training process after the deviation node is rolled back. The status response records are marked according to the status after the rollback, the status after feedback, the direction of status change, and the feedback execution result. S63. Call the state transition relation to recursively recalculate the state update sample, use the residual between the recursive recalculation result and the state after feedback as the correction amount, and use the recursive least squares method to update the state transition relation. S64. Recursively recalculate the rolled-back state using the updated state transition relationship, and update the constraint boundaries based on the residual range after recursion.

[0032] refer to Figure 3 A virtual reality immersive simulation training system based on digital twins includes: The data processing module is used to collect multi-source training data and preprocess it to form a training event sequence; The twin construction module is used to divide the training event sequence into state segments, perform Koopman up-dimensional mapping based on the response difference between the previous and subsequent state segments, solve the state transition relationship and constraint boundary, and construct the training twin model. The state evolution module is used to learn the state evolution of training event sequences across segments. It calls the training twin model to perform forward recursion along the operation trigger direction and reverse reconstruction along the state response feedback direction. It updates the state transition relationship and constraint boundary based on the positive and negative closure deviations of the corresponding state segments. The real-time recursive module is used to select unscented sampling points around the current training state, call the updated state transition relationship to propagate the state of the unscented sampling points, perform Kalman filter correction based on the observation deviation corresponding to the student's real-time operation, and simultaneously adjust the advancement step size to recursively update the current training state. The registration and localization module is used to construct the matching cost between the student's state and the standard state. It uses the Sinkhorn iterative algorithm to solve the entropy regular registration relationship and locates the off-node based on the local cost jump in the registration relationship, and adjusts the immersion feedback strategy. The feedback update module is used to roll back the current training state according to the deviation node, obtain the retraining feedback data of the immersion feedback strategy, and update the state transition relationship and constraint boundary using the recursive least squares method.

[0033] Example 1: To verify the feasibility of this invention in practice, it was applied to an immersive medical emergency training scenario. In this scenario, trainees need to continuously complete emergency treatment operations, and there is a clear sequential relationship between the operation rhythm, state response, and feedback results. Traditional virtual training methods mainly rely on fixed procedures and result scoring. When trainees deviate from the operation sequence, perform partial repetitive operations, or have delayed state responses, the system has difficulty accurately determining the location of the deviation, and retraining feedback tends to remain at the level of general prompts.

[0034] In this embodiment, multi-source training data is first collected during the trainee's training process, and the asynchronous records formed by different sampling links are organized into a training event sequence. The sampling interval is set to 50ms, and the sliding median filter window length is set to 5 sampling points. When the change in adjacent state values ​​exceeds three times the average change in values ​​of the same type of state, it is marked as abnormal jump data. Subsequently, events are segmented according to operation trigger identifiers, and the operation trigger quantity, state response quantity, and evaluation feedback quantity are arranged progressively according to the sampling time to form a training event sequence with sequential relationships.

[0035] The system divides the training event sequence into multiple state segments, each containing 8 to 32 consecutive events. Segment boundaries are set when the interval between adjacent operation triggers exceeds 1.0 s and the direction of state response change alters. The Koopman dimensionality increase operator is then used to establish state transition relationships and constraint boundaries, and the training twin model is corrected through forward recursion and backward reconstruction.

[0036] During real-time training, the system deploys unscented sampling points around the current training state, performs state propagation based on state transition relationships, and uses Kalman filtering to correct observation biases generated by the trainee's real-time operations. When the observation bias increases continuously, the step size decreases from 80ms to 40ms; when the observation bias stabilizes, the step size returns to 80ms to 100ms.

[0037] During the evaluation and feedback phase, the system constructs matching costs for the trainee state and the standard state along the evolution path of the current training state, and uses the Sinkhorn iterative algorithm to solve the entropy regularization registration relationship. The entropy regularization coefficient is set to 0.08, and the convergence threshold is set to 0.001. The system locates the deviation node based on the local cost transition and adjusts the immersion feedback strategy to the corresponding training position. After retraining, the system collects retraining feedback data and updates the state transition relationship and constraint boundary using the recursive least squares method with a forgetting factor of 0.96.

[0038] Verified through 120 training sessions, the average state response error of the traditional method was 14.6%, which was reduced to 6.9% after adopting this invention; the average positioning error for deviation from the target position was 2.4 event nodes for the traditional method, which was reduced to 0.8 event nodes after adopting this invention; and the first-pass rate for key operations after retraining increased from 72.5% to 88.7%. These results demonstrate that this invention can improve the accuracy of state tracking during training, the accuracy of deviation positioning, and the effectiveness of retraining feedback.

[0039] Table 1. Validation Data of Immersive Medical Emergency Training

[0040] As can be seen from the data in Table 1, compared with traditional virtual training methods, the present invention performs better in terms of state response tracking, deviation node location, retraining pass effect and feedback timeliness. This indicates that the present invention can more accurately reflect the state changes of trainees during the training process and provide more targeted immersive feedback based on the deviation position.

[0041] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A virtual reality immersive simulation training method based on digital twins, characterized in that, Includes the following steps: S1. Collect multi-source training data and preprocess it to form a training event sequence; S2. Divide the training event sequence into state segments, perform Koopman up-dimensional mapping based on the response difference between the preceding and following state segments, solve the state transition relationship and constraint boundary, and construct the training twin model. S3. Perform cross-segment state evolution learning on the training event sequence, call the training twin model to perform forward recursion along the operation trigger direction, and perform reverse restoration along the state response feedback direction, and update the state transition relationship and constraint boundary according to the positive and negative closure deviation of the corresponding state segment. S4. Select unmarked sampling points around the current training state, call the updated state transition relationship to propagate the state of the unmarked sampling points, perform Kalman filter correction based on the observation deviation corresponding to the student's real-time operation, and simultaneously adjust the advancement step size to recursively update the current training state. S5. Construct the matching cost between the student state and the standard state, use the Sinkhorn iterative algorithm to solve the entropy regular registration relationship, and locate the deviation node based on the local cost transition in the registration relationship to adjust the immersion feedback strategy. S6. Roll back the current training state according to the deviation node, obtain the retraining feedback data of the immersion feedback strategy, and update the state transition relationship and constraint boundary using the recursive least squares method.

2. The virtual reality immersive simulation training method based on digital twins according to claim 1, characterized in that, The preprocessing specifically includes: The multi-source training data is parsed, and a unified index is established according to the training task identifier, operation trigger identifier and sampling time. The multi-source training data is an asynchronous training record formed by different sampling links. Before entering the parsing, the field mapping, time mark and task ownership mark are completed. The sampling interval of the data after unified indexing is reorganized, and the moving median filter is used to remove abnormal jump data to obtain continuous training records; The continuous training records are segmented into events based on the operation trigger identifier, and the segmented records are mapped into operation trigger quantities, status response quantities, and evaluation feedback quantities. The operation trigger quantity, status response quantity, and evaluation feedback quantity are arranged in a progressive order according to the sampling time, and a sequential relationship is established between adjacent items to form a training event sequence.

3. The virtual reality immersive simulation training method based on digital twins according to claim 1, characterized in that, S2 specifically includes: S21. Traverse the training event sequence according to the sampling time, calculate the trigger interval and response increment of adjacent events, set the segment boundary at the position where the trigger interval exceeds the set interval threshold and the direction of the response increment changes, and divide the continuous events between adjacent segment boundaries into state segments. S22. Extract the event values ​​within the state segments according to the sampling order, arrange them into segment state vectors, and use the Koopman dimension-upper operator to perform delayed embedding and basis function expansion on the segment state vectors to obtain the dimension-upper segment vectors. S23. Pair up-dimensional segment vectors according to the order of state segments, take the response difference as the transition error between up-dimensional segment vectors, use the singular value decomposition truncation method to solve the state transition relationship, and take the minimum transition error as the solution condition. S24. Call the state transition relation to recursively deduce the state segments, and statistically analyze the state value range, response difference range, and recursion deviation range in the recursion results. Determine the constraint boundary based on the overlapping interval of each range, and associate the state transition relation with the constraint boundary to construct a training twin model.

4. The virtual reality immersive simulation training method based on digital twins according to claim 3, characterized in that, S22 specifically includes: S221. Using 3 to 8 consecutive sampled items in the state segment vector as time delay objects, construct delay coordinates, and calculate the first-order difference and second-order difference of the delay coordinates to calibrate the rate of change of operation and the second-order change of response within the state segment. S222. Construct a covariance matrix for the delayed coordinates, first-order difference, and second-order difference, and then superimpose the covariance matrices on their diagonals. to After the perturbation term of the trace mean of the matrix is ​​doubled, Cholesky decomposition is used for whitening transformation, and the effective delay dimension is screened based on the mutual information value between each dimension after whitening and the response difference. S223. The Koopman dimension-upper operator is used to expand the effective delay dimension using basis functions. After linear expansion and cross-term expansion, 8 to 64 radial basis function center points are set, and the expanded candidate dimension-upper components are compressed to 64 to 512 dimensions. S224. Calculate the reduction of the projection residual of each candidate upscaling component to the response difference, and include the upscaling components in order of the reduction of the projection residual from large to small. Stop including after the cumulative reduction of the residual reaches 85% to 95%. Arrange the included upscaling components according to the sampling order of the state segments to obtain the upscaling segment vector.

5. The virtual reality immersive simulation training method based on digital twins according to claim 1, characterized in that, S3 specifically includes: S31. Select adjacent state segments according to the successive relationship of state segments, determine the operation triggering direction from the previous state segment to the next state segment, and determine the state response feedback direction from the next state segment to the previous state segment. S32. A bidirectional dynamic mode decomposition algorithm is adopted to simultaneously decompose adjacent state segments, fit forward evolution operators and backward restoration operators under the same modal basis, and couple the forward evolution operators, backward restoration operators and state transition relations, specifically including: Arrange adjacent state segments according to the direction of operation triggering, and arrange adjacent state segments in reverse according to the direction of state response feedback, so as to keep the state items at the same sampling position corresponding; Perform joint singular value decomposition on the forward and reverse arrangement results, select the same set of principal singular vectors as the modal basis, and map adjacent state segments to the modal basis; The forward evolution operator and the backward reduction operator are fitted under the modal basis. The forward evolution operator is used to constrain the evolution of the previous state segment to the next state segment, and the backward reduction operator is used to constrain the recovery of the next state segment to the previous state segment. The forward evolution operator and the inverse reduction operator are embedded as forward and inverse calibration terms into the state transition relation, and the evolution direction and reduction direction in the state transition relation are coupled according to the forward and inverse calibration terms. S33. Call the coupled state transition relationship, continuously recursively deduce the previous state segment along the operation trigger direction, and call the reverse restoration operator to continuously invert the next state segment along the state response feedback direction. S34. Based on the continuous recursion results, continuous inversion results and the closure difference between the corresponding state segments, adjust the coupling coefficient between the forward evolution operator and the inverse restoration operator, and update the state transition relationship and constraint boundary synchronously.

6. The virtual reality immersive simulation training method based on digital twins according to claim 1, characterized in that, S4 specifically includes: S41. Determine the number of unscented sampling points according to the state dimension of the current training state. Using the current training state as the center sampling point, generate forward offset sampling points and reverse offset sampling points along each state dimension based on the constraint boundaries. Simultaneously, configure the sampling weights, where: The unmarked sampling point consists of a center sampling point, a forward offset sampling point, and a reverse offset sampling point, with the center sampling point representing the current training state. The forward offset sampling point and the reverse offset sampling point are respectively the sampling points formed by the forward and reverse offsets of the current training state along the corresponding state dimension within the constraint boundary; S42. Call the updated state transition relationship, propagate the unmarked sampling points one by one according to the current progress step size, perform boundary back pressure on the sampling points that exceed the constraint boundary after propagation, and perform weighted recycling of the propagated sampling points according to the sampling weight to obtain the predicted state. S43. Based on the state item corresponding to the student's real-time operation, perform a difference calculation with the corresponding state item in the predicted state to obtain the observation deviation. Construct a Kalman gain based on the discrete quantity of the unmarked sampling points and the observation deviation, and apply the Kalman gain to the observation deviation to correct the predicted state. S44. Based on the continuous change of the observation deviation and the distance of the predicted state after filtering correction, the remaining amount of the constraint boundary is scaled to advance the step size, the predicted state after filtering correction is updated to the current training state, and the process continues to advance according to the scaled step size.

7. The virtual reality immersive simulation training method based on digital twins according to claim 1, characterized in that, S5 specifically includes: S51. Extract the student state and standard state along the evolution path of the current training state, assign measurement weights to different state items according to the constraint boundary, and use the Mahalanobis measurement method to measure the state offset between the student state and the standard state, and combine the path order offset to form a matching cost matrix. S52. Perform entropy regular exponential mapping on the matching cost matrix to obtain the initial registration weights. Then, use the Sinkhorn iterative algorithm to perform row normalization and column normalization on the initial registration weights until the registration weight changes in two adjacent rounds satisfy the convergence condition, and obtain the entropy regular registration relationship. S53. Traverse the entropy regular registration relationship along the evolution path of the current training state. In the registration weights corresponding to each student state, select the standard state position with the largest weight as the main registration position, and mark the local cost transition amount according to the matching cost change rate corresponding to the adjacent main registration positions. S54. The main registration position where the local cost jump exceeds the jump threshold is determined as the deviation node. Based on the state offset direction and path position corresponding to the deviation node, the feedback triggering timing, feedback pointing state and rollback range in the immersion feedback strategy are adjusted.

8. The virtual reality immersive simulation training method based on digital twins according to claim 7, characterized in that, Specifically, S52 includes: S521. Apply negative exponential mapping to each item of the matching cost matrix using entropy regularization coefficients ranging from 0.01 to 0.20, and then apply the mapping results according to... to The lower limit truncation value is used to truncate the initial registration weights; S522. According to the order of student status in the evolution path, traverse the initial registration weights line by line, count the total weight of each line, and perform row-directed normalization on each registration weight in the line with the corresponding total weight to obtain the row-directed normalized weights. S523. According to the order of the standard states in the evolution path, traverse the row normalization weights column by column, count the total weight of each column, and perform column normalization on each registration weight in the column with the corresponding total weight to obtain the column normalization weights. S524. Using the column normalization weights as input for the next iteration, alternate between row normalization and column normalization, and compare the changes in column normalization weights between adjacent iterations: When the number of iterations reaches 20 to 80, and the change is less than to At that time, the normalization weight of the current column is locked to obtain the entropy regularization registration relationship.

9. The virtual reality immersive simulation training method based on digital twins according to claim 1, characterized in that, S6 specifically includes: S61. Based on the position of the deviation node in the evolution path of the current training state, roll back the current training state to the state corresponding to the deviation node, and start executing the immersion feedback strategy from the rolled-back state. S62. Collect retraining feedback data after implementing the immersion feedback strategy, pair the rolled-back state with the feedback state in the retraining feedback data, and obtain state update samples for updating state transition relationships, wherein: The retraining feedback data is the status response record generated during the re-execution of the training process after the deviation node is rolled back. The status response record is marked according to the status after rollback, the status after feedback, the direction of status change, and the feedback execution result. S63. Call the state transition relation to recursively recalculate the state update sample, use the residual between the recursive recalculation result and the state after feedback as the correction amount, and use the recursive least squares method to update the state transition relation. S64. Recursively recalculate the rolled-back state using the updated state transition relationship, and update the constraint boundaries based on the residual range after recursion.

10. A virtual reality immersive simulation training system based on digital twins, executing the virtual reality immersive simulation training method based on digital twins as described in any one of claims 1 to 9, characterized in that, include: The data processing module is used to collect multi-source training data and preprocess it to form a training event sequence; The twin construction module is used to divide the training event sequence into state segments, perform Koopman up-dimensional mapping based on the response difference between the previous and subsequent state segments, solve the state transition relationship and constraint boundary, and construct the training twin model. The state evolution module is used to learn the state evolution of training event sequences across segments. It calls the training twin model to perform forward recursion along the operation trigger direction and reverse reconstruction along the state response feedback direction. It updates the state transition relationship and constraint boundary based on the positive and negative closure deviations of the corresponding state segments. The real-time recursive module is used to select unscented sampling points around the current training state, call the updated state transition relationship to propagate the state of the unscented sampling points, perform Kalman filter correction based on the observation deviation corresponding to the student's real-time operation, and simultaneously adjust the advancement step size to recursively update the current training state. The registration and localization module is used to construct the matching cost between the student's state and the standard state. It uses the Sinkhorn iterative algorithm to solve the entropy regular registration relationship and locates the off-node based on the local cost jump in the registration relationship, and adjusts the immersion feedback strategy. The feedback update module is used to roll back the current training state according to the deviation node, obtain the retraining feedback data of the immersion feedback strategy, and update the state transition relationship and constraint boundary using the recursive least squares method.

Images