Digital twin driven double-layer reinforcement learning for fall prevention vr training strategy optimization method

Abstract

This invention relates to the fields of virtual reality, digital twins, and artificial intelligence, and provides a method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning. To address the safety concerns in existing fall prevention VR training, this method constructs a pre-trained digital twin environment and a user behavior simulation model, combining real-time multimodal data to generate user state vectors. A two-layer reinforcement learning architecture is then employed, where a strategy optimization generator produces a VR training strategy parameter set based on simulated performance characteristics, and safety verification and optimization are performed through simulation prediction. The optimized VR training strategy parameter set is deployed to a VR training terminal for training, using training feedback data to update the corresponding model, forming a dynamic adaptive optimization closed loop. By achieving personalized strategy generation and pre-verification of safety, this method significantly improves the safety and dynamic adaptability of fall prevention VR training, and can be applied to scenarios such as fall prevention for the elderly and motor skills training.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of virtual reality and artificial intelligence, and particularly to the field of adaptive training based on digital twins and reinforcement learning. Specifically, it relates to an optimization method for fall prevention VR training strategies driven by digital twins and two-layer reinforcement learning. Background Technology

[0002] Maintaining and improving daily living activities, especially dynamic balance, is a key element in supporting the elderly to achieve independent and safe living. Virtual reality (VR) technology provides an innovative means for conducting balance adaptation training in a controlled and repeatable environment. With the development of virtual reality and digital twin technologies, they have shown application potential in areas such as assisted living, health promotion, and functional capacity maintenance. In existing technologies, fall prevention VR training strategy generation methods mainly rely on two approaches: one is based on expert rules or predefined training libraries, matching static training schemes according to the user's basic information or simple assessment results; the other is based on single-layer reinforcement learning models, directly optimizing VR training strategy parameters based on the user's real-time interactive feedback in the VR environment.

[0003] While the existing methods can automate VR training to some extent, the generation and optimization of their VR training strategies heavily rely on online trial and error and user feedback in real VR environments. This makes it difficult for existing technologies to efficiently generate effective personalized training programs for individuals with significant differences in abilities while ensuring absolute safety during the training process.

[0004] Therefore, how to efficiently generate and optimize dynamic and personalized VR training strategies that match the user's abilities while ensuring absolute safety during the training process has become a core technical challenge that virtual reality urgently needs to solve in the field of balance ability adaptive training. This invention is proposed to address this technical challenge in the typical application scenario of "fall prevention". Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning. This method significantly improves the accuracy and personalization of dynamic adaptation in the VR training strategy generation process, providing a new technical solution for scenarios requiring high safety and personalization, such as fall prevention ability training.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning, applied to a system including a management server and a VR training terminal. The method includes: the management server acquiring real-time multimodal data of a target user and inputting it into a preset feature encoding network to generate a user state vector; inputting the user state vector into a pre-trained digital twin environment corresponding to the target user, wherein the pre-trained digital twin environment deploys a user behavior simulation model based on deep reinforcement learning, and the user behavior simulation model is pre-trained using the target user's historical VR training data; and calling a deep reinforcement learning-based strategy optimization generator, using the simulated performance characteristics of the user behavior simulation model within a preset time window as... The system takes a state input and generates a VR training strategy parameter set. This parameter set is then input into the pre-trained digital twin environment. An N-step simulation prediction of fall prevention safety is performed using the user behavior simulation model to obtain a safety prediction result, where N is a natural number greater than 1. Based on the safety prediction result, the VR training strategy parameter set undergoes safety verification and optimization. The verified and optimized VR training strategy parameter set is sent to the VR training terminal to execute the corresponding fall prevention VR virtual simulation training scenario. Real-time multimodal data of the target user is collected during the execution of the fall prevention VR virtual simulation training scenario, and the real-time multimodal data is used to update the user behavior simulation model and the strategy optimization generator to form an optimization closed loop for the VR training strategy.

[0007] As a preferred embodiment of the digital twin-driven two-layer reinforcement learning method for optimizing fall prevention VR training strategies described in this invention, the method involves performing an N-step simulation prediction of fall prevention safety using the user behavior simulation model. This includes: initializing a virtual training scene in the pre-trained digital twin environment with the VR training strategy parameter set; driving the user behavior simulation model to continuously execute actions for N time steps in the virtual training scene; calculating and recording the following indicators at each time step: the projected coordinates of the virtual center of mass relative to the foot support surface, the instantaneous torque of the main load-bearing joints, and the predicted fall risk probability, wherein the fall risk probability is calculated by a lightweight risk assessment network pre-trained with labeled data; and the safety prediction result is time-series data composed of the projected coordinates, instantaneous torque, and fall risk probability for N time steps.

[0008] The beneficial effects of this invention are as follows: By constructing a digital twin closed loop of "perception-simulation-decision-execution-feedback", the "generation-verification-deployment" process of the strategy is decoupled and brought forward in the VR training scenario, which fundamentally solves the problems of "lagging security verification" and "low efficiency of personalized generation" caused by the reliance on online trial and error in existing technologies. Specifically, firstly, by employing a two-layer reinforcement learning mapping architecture of "user state vector → simulated performance features → VR training strategy parameter set," the basis for strategy generation is elevated from raw data to a predictive assessment of user capabilities. This directly addresses the deficiency of insufficient accuracy in matching strategies with real-time user states, achieving more precise and proactive strategy generation. Secondly, by performing N-step simulation prediction and security verification on the generated VR training strategy parameter set in a pre-trained digital twin environment, security verification is transformed from post-event remediation in real training to pre-event verification and prevention in virtual space. This directly solves the problem of not being able to quantify and verify strategy security before deployment, and incorporates security protection measures into the training process. Thirdly, by synchronously updating the user behavior simulation model and strategy optimization generator using real training feedback data, a self-iteratory optimization loop is formed, enabling the system to dynamically evolve with user capabilities. This directly overcomes the bottleneck of limited adaptive capabilities of static strategy libraries or single models, achieving long-term, dynamic, and personalized adaptation. In summary, this invention, through the aforementioned closed loop, transforms the traditional passive adaptation mode of "trial and error-feedback" into an active safety design mode of "simulation-verification-optimization," ensuring the safety of VR training (especially in scenarios with high safety requirements such as fall prevention), improving personalized training, and enhancing the system's adaptive capabilities. Attached Figure Description

[0009] Figure 1 A flowchart illustrating the method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning, as provided in an embodiment of the present invention.

[0010] Figure 2 A schematic diagram of the overall framework and process of the method for optimizing fall prevention VR training strategies driven by digital twin-based two-layer reinforcement learning provided in an embodiment of the present invention;

[0011] Figure 3 This is a schematic diagram of the first sub-process of the method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning, as provided in an embodiment of the present invention. Detailed Implementation

[0012] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0013] The digital twin-driven two-layer reinforcement learning-based fall prevention VR training strategy optimization method provided in this invention is typically carried by devices such as management servers and VR training terminals. It can directly serve as the core engine and is widely applicable to VR functional training scenarios with extremely high requirements for training process safety and personalized adaptation. It can empower various VR training systems, significantly improving their intelligence level, including but not limited to the following scenarios: First, in the field of life support and health promotion, it can provide personalized and adaptive VR balance training solutions for users concerned about their activity safety and ability maintenance (such as older groups who need to improve their balance), helping them achieve a more autonomous and safer life. Second, in the field of sports performance and professional skills training, it can serve as an adaptive training system for athletes, pilots, and other professionals to improve their posture stability and motion control capabilities in complex environments, rapidly enhancing their dynamic balance performance in specific scenarios through closed-loop optimization. The following describes some embodiments of this invention in detail with reference to the accompanying drawings.

[0014] Example 1, please refer to Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the overall framework and process of the digital twin-driven two-layer reinforcement learning method for optimizing fall prevention VR training strategies, provided in an embodiment of the present invention. Figure 1 As shown, in this embodiment, the method is applied to a system including a management server and a VR training terminal, and the method includes, but is not limited to, the following steps S11-S17:

[0015] S11: The management server acquires the real-time multimodal data of the target user and inputs it into a preset feature encoding network to generate a user state vector.

[0016] Real-time multimodal data: refers to data that is synchronously collected and time-aligned by multiple sensors deployed on the anti-fall VR training environment and the target user's body, including but not limited to: 1) kinematic data (such as joint coordinates and center of mass trajectories collected by inertial and optical sensors); 2) dynamic data (such as plantar pressure and ground reaction force collected by pressure sensing devices); 3) physiological signals (such as electromyography, heart rate and heart rate variability); 4) interactive events (such as collisions and task completion events in VR scenes).

[0017] Pre-defined feature encoding network: This refers to a deep learning model pre-trained on a large amount of training data (including multimodal data input and corresponding manually labeled user state labels). It is a nonlinear function mapper for multi-source heterogeneous data fusion and dimensionality reduction, and typically includes: a) an input layer and a normalization layer, used to receive and normalize multimodal data streams with different physical dimensions; b) a multi-branch feature extraction sub-network (such as using a one-dimensional convolutional neural network (CNN) or a long short-term memory network (LSTM) for temporal motion data, and a two-dimensional CNN for spatial pressure data), used to automatically extract high-dimensional abstract features from each modality of data; c) a feature fusion layer (such as concatenation, attention-weighted fusion), which aggregates the information of the features extracted from each branch; d) an encoding output layer (usually a fully connected layer), which compresses and maps the fused high-dimensional feature vector into a low-dimensional, dense, and continuous user state vector.

[0018] The implementation is as follows: Figure 2 As shown, the management server first aggregates real-time multimodal data streams from VR training terminals (such as the IMU, controllers, and foot pressure pads of VR devices). After preprocessing (e.g., filtering and normalization), the real-time multimodal data is input into a pre-trained preset feature encoding network. This network encodes the input and ultimately outputs a user state vector. This user state vectorization represents the target user's comprehensive balance state, movement pattern, and physiological load state during VR training, solving the problem of real-time representation and utilization of high-dimensional heterogeneous data. It automates feature extraction and standardizes state representation, improving the accuracy, real-time performance, and overall system performance of subsequent personalized strategy optimization.

[0019] S12: Input the user state vector into the pre-trained digital twin environment corresponding to the target user, and drive the user behavior simulation model deployed therein to run, and obtain the simulated performance characteristics of the user behavior simulation model within a preset time window. The user behavior simulation model is a deep reinforcement learning-based model, and pre-training is completed using the target user's historical VR training data.

[0020] Pre-trained digital twin environment: This refers to a high-fidelity virtual simulation system deployed on a management server and built for the target user. Technically, it's a software simulator integrating a physics engine (simulating gravity, collisions, and mechanical responses) and a parameterized biomechanical human body model, capable of realistically replicating the physical interactions of real VR training scenarios. "Pre-trained" and "for the target user" refer to the fact that its environmental dynamic parameters (such as virtual body inertia and ground friction coefficient) and built-in intelligent agent model have been calibrated using relevant historical data from the target user, enabling its simulated behavior to personalizedally approximate the target user's physical reactions and behavioral patterns in a real environment.

[0021] User behavior simulation model: This refers to an agent deployed in the aforementioned pre-trained digital twin environment. Its technical implementation is based on a deep reinforcement learning architecture (such as the Proximal Policy Optimization (PPO) algorithm) and a neural network policy model with converged parameters. It takes the environmental state (such as the user's state vector and virtual scene information) as input and directly outputs the corresponding simulated actions (such as virtual joint torques and gait parameters). It is pre-trained offline using the target user's historical VR training data (including state, action, and reward sequences) to learn the target user's personalized behavioral response patterns and strategies.

[0022] Simulated performance characteristics: These represent a set of quantifiable performance and risk indicators generated by a user behavior simulation model running in a pre-trained digital twin environment, based on the input user state vector, within a future window of a set time length. These are data generated through simulation calculations and typically include, but are not limited to, the following indicators: simulated fall count, center of gravity stability margin, joint load extremes, task completion time, energy consumption estimates, etc. They are used to predictively assess the performance and risks that the target user may face if training continues in the current state.

[0023] The implementation is as follows: First, the user state vector is used as the initial input and loaded into the corresponding pre-trained digital twin environment configured for the target user. Second, the user state vector drives the pre-trained user behavior simulation model, also targeting the target user and deployed in the pre-trained digital twin environment, to start running. Based on the user state vector and its internally fixed neural network parameters, the user behavior simulation model performs a series of virtual actions in the pre-trained digital twin environment and interacts with the corresponding virtual environment. Furthermore, within a preset time window (e.g., the next 2 seconds), the system applies a set of standardized test virtual balance challenge tasks covering different difficulty gradients to the user behavior simulation model (e.g., maintaining a standing position on a swaying platform, resisting a momentary light push from a specific direction, and crossing virtual threshold obstacles of different heights), and records the trajectory data (e.g., center of mass trajectory, joint angle trajectory) generated by the user behavior simulation model when performing each test virtual balance challenge task. Finally, based on the recorded trajectory data, through defined mathematical calculations, such as through the sensor simulation module and performance evaluation module in the pre-trained digital twin environment, multi-dimensional quantitative indicators are extracted, and a series of simulated performance characteristics are recorded and calculated in real time, such as task success rate (based on preset success criteria statistics), average trajectory smoothness index (such as the root mean square value of the centroid velocity), and area of ​​the centroid undulation ellipse (fitting the area of ​​the ellipse distributed by the centroid's projection points on the horizontal plane). After normalizing these indicators, they are concatenated into a fixed-dimensional numerical vector, which serves as the simulated performance characteristics.

[0024] The reason for using a user behavior simulation model and pre-training it based on its historical VR training data is that, in order to make credible risk predictions in a virtual environment, an agent that can closely approximate the reactions of real users is needed. By using the target user's own historical interaction data to train the deep reinforcement learning model, it can learn the target user's unique balance adjustment strategies, reaction time, mechanical characteristics, etc., so that the simulation behavior in the digital twin environment has a high degree of personalization and fidelity, providing a reliable simulation basis for subsequent policy security assessment.

[0025] For example, taking the "crossing a virtual threshold" training of elderly user Zhang San as an example, step S11 has generated Zhang San's user state vector S_t at the training moment (representing his "center of gravity shifts forward too quickly, lower limb stability is moderate"). Based on this, the following processing is performed: First, the system inputs S_t into Zhang San's personalized pre-trained digital twin environment. This pre-trained digital twin environment has preset the same "threshold" scenario as the current training, and the dynamic parameters of its virtual human body model have been adjusted according to Zhang San's historical data. Subsequently, the system drives the user behavior simulation model corresponding to Zhang San (a deep reinforcement learning network that has learned Zhang San's typical crossing strategy) to start running. The user behavior simulation model receives S_t and outputs the simulated crossing action sequence. In the next 2-second preset time window, the system runs the above simulation. The physics engine calculates the interaction between the virtual body and the threshold, and may ultimately simulate the result of "successfully crossing" or "tripping on tiptoe". Finally, the system collects data from the entire simulation process and calculates the simulation performance characteristics, such as: the maximum centroid offset angle is 12 degrees and the estimated probability of toe collision is 30%. These characteristics quantify and predict the risk of Zhang San continuing training in the current state.

[0026] Step S12 drives the personalized user behavior simulation model to perform standardized tests in the digital twin environment, constructing a rapid simulation link of "state-behavior-performance". This transforms the real-time state of the target user into predictive multi-dimensional capability indicators (simulated performance characteristics), thereby providing a forward-looking and quantitative decision-making basis for strategy generation. This solves the problem of strategy blindness caused by the lack of forward-looking evaluation in existing methods, and constitutes the core premise for realizing personalized and security-oriented pre-optimization.

[0027] S13: Invoke the deep reinforcement learning-based policy optimization generator, using the simulated performance features as state input, to generate a VR training policy parameter set.

[0028] Among them, the deep reinforcement learning-based policy optimization generator represents a parameterized neural network model built on a deep reinforcement learning framework (such as the actor-critic architecture). In essence, it is a learnable policy function approximator. Its function is to take simulated performance features as input states, calculate through the forward propagation of its internal network, and output a set of optimized VR training policy parameters. Its decision-making mechanism lies in the fact that, through offline and online training, the parameters of the neural network converge, thereby learning the optimal or near-optimal mapping function from the user's state space (represented by simulated performance features) to the action space (i.e., the VR training strategy parameter set). The optimization objective of this mapping function is to maximize the user's long-term training benefits (such as improved balance ability) and minimize their risk of falling in the simulation evaluation of the digital twin environment. Therefore, when deploying the application, the strategy optimization generator receives simulated performance features that reflect the target user's current ability and risk, and automatically generates a VR training strategy parameter set that is personalized to it and guided by the above objectives, based on the mapping function it has learned internally. The matching means that the generated strategy parameter set can adapt to the target user's current ability level (for example, generating more challenging parameters for high-ability users and safer parameters for high-risk users), thereby achieving a dynamic balance between training difficulty and user state.

[0029] VR training strategy parameter set: This represents the executable technical parameters output by the strategy optimization generator, used to specifically configure and adjust the next VR training task. It is a set of instructions that directly control the behavior of the VR training terminal, including but not limited to: a) task difficulty parameters (such as the height and movement speed of virtual obstacles); b) perturbation application parameters (such as the magnitude, direction, and timing of the push applied to the user's virtual body); c) auxiliary guidance parameters (such as the intensity of visual cues and the strength mode of haptic feedback); d) training objective parameters (such as the balance dimension emphasized in this training). The VR training strategy parameter set is a key factor connecting the strategy optimization algorithm and the physical / virtual training execution terminal.

[0030] The implementation is as follows: The system calls the deployed deep reinforcement learning-based policy optimization generator, using simulated performance features as state input. The policy optimization generator performs corresponding forward propagation calculations based on its internal fixed network structure and weight parameters. Through its output layer (usually the Actor network part), it maps the input simulated performance features into a specific, numerical set of VR training policy parameters. The mapping process is essentially a specific application of a function optimization process trained by reinforcement learning with the goal of maximizing long-term cumulative rewards (such as improved training effect and reduced risk) in the current state. Its output aims to directly adjust the environment and task conditions for the next VR training to optimize the training effect and safety of the target user.

[0031] Step S13 uses a deep reinforcement learning strategy optimization generator to map simulated performance features to a set of VR training strategy parameters, achieving end-to-end optimization from "prediction to decision". This directly solves the problems of poor matching between strategies and users' dynamic capabilities and inefficient optimization. Data-driven automated generation replaces empirical rules, forming the core decision-making link that transforms forward-looking assessments into personalized action plans.

[0032] S14: Input the VR training strategy parameter set into the pre-trained digital twin environment, and perform N-step simulation prediction of fall prevention safety through the user behavior simulation model to obtain the safety prediction result, where N is a natural number greater than 1.

[0033] N-step simulation prediction: This refers to the simulation process in a pre-trained digital twin environment, starting with the user's state at the previous moment, configuring the virtual scene based on the parameter set of the VR training strategy to be evaluated, driving the user behavior simulation model to continuously execute the action sequence of N discrete time steps, and simultaneously calculating the physical and risk indicators of each time step. The value of N determines the look-ahead time length of the prediction. N being greater than 1 ensures that the evaluation can cover the dynamic process of strategy execution and potential delay risks.

[0034] Safety prediction results: These represent quantitative risk indicators calculated through an N-step simulation prediction process. They are used to assess the potential fall prevention safety of the VR training strategy parameter set in a digital twin environment. They are used to quantitatively characterize the potential risks of the VR training strategy parameter set before deployment, including but not limited to: predicted fall event occurrence rate (such as the number or probability of simulated falls occurring within N simulation steps), stability boundary measures (such as the minimum distance between the centroid and the boundary of the supporting polygon), and key biomechanical load extremes (such as the predicted maximum inversion moment of the ankle joint). This provides an objective safety assessment basis for subsequent VR training strategy parameter optimization.

[0035] The implementation is as follows: First, the VR training strategy parameter set is input as an environment configuration command into the pre-trained digital twin environment corresponding to the target user. The pre-trained digital twin environment dynamically adjusts the difficulty, perturbation mode, and other scene attributes of the virtual training scene according to the VR training strategy parameter set (e.g., loading obstacle models with corresponding height and friction coefficients, and setting the time series of perturbation forces). Second, the user behavior simulation model deployed in the pre-trained digital twin environment is driven to start from the current user state vector (implicit in the internal state of the user behavior simulation model or initialized by the user state vector) and run continuously for N time steps (N>1) in the adjusted new virtual environment according to its internal strategy. During the entire N-step simulation process, various indicators related to fall prevention safety are monitored and recorded in real time based on physics engine calculations and predefined safety assessment rules (e.g., collision detection and posture determination algorithms to determine whether a virtual fall has occurred). After the simulation, the monitoring data within N steps is summarized, and a comprehensive safety prediction result is generated through calculations (e.g., counting the number of falls, calculating the average or worst value of stability indicators).

[0036] Step S14 generates time-series security prediction results by performing N-step simulation prediction in the digital twin environment, which solves the problem of not being able to quantify and simulate dynamic risks before policy deployment. By constructing a high-fidelity simulation test process of "policy → environment → behavior → risk", dynamic, forward-looking and quantitative assessment of potential security risks is realized. This step is the core of security pre-verification and provides an objective basis for subsequent verification and optimization, thereby fundamentally addressing the challenges of high security risks and lack of forward-looking verification.

[0037] S15: Based on the security prediction results, perform security verification and optimization adjustments on the VR training strategy parameter set.

[0038] The implementation is as follows: A pre-defined safety threshold condition is numerically compared with the safety prediction result to perform a safety check. The check process is based on explicit, pre-defined numerical rules (such as "the predicted number of falls must be 0" and "the average stability margin must be greater than 5cm") for logical judgment. If the check passes, the current VR training strategy parameter set is considered safe and usable. If the check fails, an optimization and adjustment process is triggered, using a pre-defined adjustment strategy based on mathematical rules. For example, using gradient descent, the parameters of the VR training strategy parameter set are finely adjusted in the reverse direction of the gradient based on the safety index (such as proportionally reducing the difficulty parameter), or a set of corrected VR training strategy parameter sets is generated based on a rule base (such as "if a fall is predicted, the obstacle height is reduced by a fixed step size"). The output is the new VR training strategy parameter set after adjustment (or maintaining the original state).

[0039] For example, continuing from the previous example, suppose that after N simulation steps, the safety prediction result obtained for a VR training strategy parameter set for user "Zhang San" is: {"Predicted number of falls": 2, "Average stability margin": 3cm}. First, the system's preset safety threshold is: {"Predicted number of falls" ≤ 0, "Average stability margin" ≥ 5cm}. Based on this, the system compares the safety prediction result with the safety threshold. Since "Predicted number of falls = 2 > 0" and "Average stability margin = 3cm < 5cm", the verification fails, and the VR training strategy parameter set is determined to have a safety hazard. Subsequently, the system automatically adjusts the VR training strategy parameter set according to preset adjustment rules (e.g., "If the predicted number of falls > 0, then the scene difficulty parameter will be uniformly reduced by 20%). Assuming that the "virtual obstacle height" parameter in the original strategy is H, the parameter in the new strategy after adjustment becomes 0.8 * H. The system generates a new VR training strategy parameter set containing this corrected parameter.

[0040] Step S15 establishes an automated closed loop of "risk detection → parameter correction" by performing security verification based on quantification thresholds and driving iterative optimization of the policy optimization generator. This solves the defect of the disconnect between security assessment and optimization in traditional technologies, and realizes automatic security compliance verification and targeted optimization within the security boundary of the policy. It is the core link to ensure the security of the final output policy, thereby transforming security from passive assessment to active optimization.

[0041] S16: Send the VR training strategy parameter set, which has been verified and optimized, to the VR training terminal to execute the corresponding fall prevention VR virtual simulation training scenario.

[0042] The process is as follows: The management server sends the determined VR training strategy parameter set to the target user's VR training terminal. Upon receiving the parameter set, the VR training terminal's locally running VR scene rendering engine and physics simulation module dynamically and in real-time reconstruct the fall prevention virtual training scene presented to the target user based on this parameter set. Specifically, the values ​​in the VR training strategy parameter set are used as control variables, directly inputting and configuring the VR training terminal's virtual environment generation software. For example, the "obstacle height" parameter controls the scaling of the corresponding geometry in the 3D model; the "disturbance force" parameter sets the force / torque vector applied to the user's virtual body in the physics engine; and the "prompt method" parameter determines the generation logic and timing of visual / auditory prompts in the user interface. This process integrates VR rendering, physics simulation, and human-computer interaction technologies. For specific implementation details, refer to existing related technologies; they will not be elaborated upon here. After configuration, the VR training terminal drives its hardware to present the target user with a personalized virtual balance challenge environment corresponding to the VR training strategy parameter set. The target user then immerses themselves in this environment through natural interaction and begins training.

[0043] S17: Collect real-time multimodal data of the target user during the execution of the fall prevention VR virtual simulation training scenario, and use the real-time multimodal data to update the user behavior simulation model and the strategy optimization generator to form an optimization closed loop of the VR training strategy.

[0044] The implementation is as follows: When a target user executes the personalized fall prevention training scenario deployed in step S16 on the VR training terminal, the system synchronously collects real-time multimodal data during this process through various sensors integrated into the VR training terminal (such as IMU, optical tracking, and force feedback devices). The newly collected real-time multimodal data completely records the target user's actual reaction and performance to the optimized VR training strategy parameter set. Subsequently, this newly collected real-time multimodal data (usually forming training samples together with the corresponding executed VR training strategy parameter set and the final training effect label) is transmitted back to the management server. The management server uses this incremental data to update the parameters of two core models through offline batch processing or online learning algorithms (such as policy gradient-based reinforcement learning updates): a) Update the user behavior simulation model so that it can more accurately simulate the target's behavioral response in new states or when facing new strategies, improving its prediction fidelity; b) Update the policy optimization generator by optimizing its internal network parameters based on the new state-action-effect data pairs, enabling it to generate better and safer VR training strategy parameter sets in the future. After the model is updated, the system enters the next optimization cycle, thus forming a closed loop of continuous iteration for VR training strategy optimization.

[0045] This invention constructs a closed loop for adaptive strategy generation and verification based on digital twins. It generates standardized user state vectors by fusing real-time multimodal data and inputs them into a digital twin environment for N-step simulation, achieving quantitative security assessment and pre-verification of candidate VR training strategy parameter sets. Based on the simulation results, security verification is performed, driving iterative optimization of the strategy. Simultaneously, under security constraints, the corresponding model is continuously updated using real feedback data, thus forming a complete technical cycle of "perception-simulation-decision-execution-learning." This technically ensures the security of the VR training strategy parameter generation process, the accuracy of personalized adaptation, and the long-term adaptability of the system.

[0046] In one embodiment, the real-time multimodal data includes the target user's physiological time-series signal, a three-dimensional skeletal joint motion trajectory sequence, and a VR interaction event stream. Generating a user state vector includes: inputting the physiological time-series signal into a first encoding sub-network to obtain a physiological feature vector, the first encoding sub-network including a preprocessing layer for filtering baseline drift and an LSTM layer for extracting heart rate variability features; inputting the three-dimensional skeletal joint motion trajectory sequence into a second encoding sub-network to obtain a kinematic feature vector, the second encoding sub-network including an inverse kinematic layer for calculating joint angles and angular velocities and a temporal convolutional layer for capturing motion patterns; inputting the VR interaction event stream into a third encoding sub-network to obtain an interaction feature vector, the third encoding sub-network including an embedding layer for statistically analyzing event frequencies and a Transformer layer for modeling attention distribution; concatenating the physiological feature vector, the kinematic feature vector, and the interaction feature vector, and performing dimensionality reduction and fusion through a fully connected layer to output the corresponding user state vector.

[0047] Physiological time-series signals: These represent electrophysiological signals that are continuously collected and change over time through wearable physiological sensors (such as heart rate belts and skin conductance sensors), such as electrocardiogram (ECG / PPG) signals and skin conductance (GSR) signals. Their "time-series" characteristic indicates that the data is a one-dimensional time series generated at a fixed sampling rate, containing information reflecting the user's autonomic nervous activity (such as stress and fatigue).

[0048] Three-dimensional skeletal joint motion trajectory sequence: This refers to the sequence data defined in a three-dimensional spatial coordinate system, which is continuously output at a fixed frequency (such as 90Hz) through the optical tracking or inertial measurement unit (IMU) array of a VR system, characterizing the spatial position (X, Y, Z coordinates) of major human joints (such as hip, knee, ankle) over time.

[0049] VR Interaction Event Stream: This refers to a discrete, timestamped sequence of events generated in a fall prevention VR training scenario. These events include, but are not limited to, virtual obstacle collisions, task objective achievement, triggering of auxiliary prompts, and user interface operations (such as button clicks). It records the logical process of the target user's interaction with the corresponding virtual environment.

[0050] The first, second, and third encoding sub-networks represent three parallel and structurally independent deep neural network branches within the preset feature encoding network. Each encoding sub-network is designed to process corresponding modal data, and its network structure (such as layer type and connection method) matches the inherent characteristics (temporal, spatial, and discrete) of the modal data, aiming to achieve optimal feature extraction within the modality. They share the same training objective and converge in the fusion layer.

[0051] The implementation is as follows: First, three types of heterogeneous real-time multimodal data are routed to three parallel pre-trained encoding sub-networks: 1) Physiological time-series signals (such as ECG signals) are input into the first encoding sub-network. The signals are first pre-processed through a preprocessing layer (such as a bandpass filter) to filter out noise such as baseline drift. Then, the cleaned signals are passed through an LSTM layer, which utilizes its gating mechanism to model long-term temporal dependencies and automatically extract high-dimensional abstract representations such as heart rate variability (HRV) time-domain / frequency-domain features, outputting a physiological feature vector. 2) The three-dimensional skeletal joint motion trajectory sequence is input into the second encoding sub-network. First, an inverse kinematics layer, based on rigid body kinematics principles, transforms the original position sequence into a more biomechanically meaningful sequence of joint angles and angular velocities. Then, it is input into a temporal convolutional layer (TCN), where the sliding of the convolutional kernel along the time dimension efficiently captures local and global motion pattern features (such as gait periodicity and limb coordination), outputting a kinematic feature vector. 3) The VR interactive event stream (discrete symbol sequence) is input into the third encoding sub-network. First, each event type is mapped to a dense vector through an embedding layer, and its representation can be enhanced through statistics (such as event frequency). This vector is then input into a Transformer layer, which utilizes its self-attention mechanism to model the long-range dependencies between different events and the attention distribution of interaction intent (e.g., frequent collision events are given higher weights), outputting an interaction feature vector. Finally, the above three feature vectors are concatenated to form a joint feature vector, which is then input into a fully connected layer. The fully connected layer fuses the joint features through matrix multiplication and nonlinear activation, ultimately outputting a user state vector.

[0052] This invention addresses the problems of feature interference, information loss, and low extraction efficiency in simple fusion of multi-source heterogeneous data by defining multi-modal data in detail and designing a dedicated feature extraction sub-network based on the physical / logical characteristics of each data modality. This achieves targeted and efficient extraction of deep features of each modality, resulting in a more comprehensive and accurate reflection of the user's real-time state in the three dimensions of physiology, biomechanics, and cognitive interaction. This provides a more discriminative state input for subsequent strategy optimization loops, further improving the accuracy of personalized adaptation decisions in the entire system.

[0053] In one embodiment, the historical VR training data includes a sequence of historical VR training interaction data arranged in chronological order; the pre-training process of the user behavior simulation model includes: acquiring the historical VR training interaction data sequence of the target user, the historical VR training interaction data sequence including a historical user state vector, a historical VR training strategy parameter set, and corresponding historical user response data; using the historical user state vector and the historical VR training strategy parameter set as input states, and minimizing the prediction error of the historical user response data as the training objective, iteratively training the user behavior simulation model using a proximal policy optimization algorithm; during the training process, a biomechanical constraint loss term based on a physics engine is introduced, the biomechanical constraint loss term being used to penalize simulated actions in the user behavior simulation model that cause virtual joint angles to exceed physiological ranges or centroid projections to exceed the boundaries of supporting polygons.

[0054] Historical VR training interaction data sequence: This is a structured dataset consisting of time-aligned triples {historical user state vector s_t, historical VR training policy parameter set a_t, and historical user real response data r_t}. It completely reproduces the historical interaction trajectory between the target user and the VR training system and serves as a supervised learning sample for the pre-trained user behavior simulation model.

[0055] The implementation is as follows: First, acquire the historical VR training interaction data sequence of the target user. Each training iteration is transformed into a sequence decision problem: given a historical state (historical user state vector) and the action taken (historical VR training policy parameter set), what reaction will the user produce (historical user reaction data)? The training objective is to enable the user behavior simulation model to learn this mapping relationship. Second, iterative training is performed using the Proximal Policy Optimization (PPO) algorithm as a framework. In each iteration, the following processes are included, but are not limited to: 1) Inputting the historical user state vector and historical VR training policy parameter set as the current state (s) and action (a) into the user behavior simulation model to be trained (as a policy network Actor). 2) The user behavior simulation model outputs the predicted simulated action (corresponding to the movement command of the virtual body). This simulated action generates predicted user reaction data in the training environment (or through forward computation within the user behavior simulation model). This data is compared with the corresponding real historical user reaction data in the sequence, and the prediction error (such as mean squared error) is calculated. This prediction error serves as the negative value or component of the reward signal in the PPO algorithm, driving the user behavior simulation model to learn to minimize the prediction error, i.e., to mimic the user's real reaction. Among them, the pre-training of the user behavior simulation model adopts the proximal policy optimization (PPO) algorithm. This is because the PPO algorithm has the characteristics of high sampling efficiency and good training stability, making it suitable for offline training using limited user historical data. Furthermore, its trust domain constraint can prevent drastic changes in simulated behavior that violate biomechanics during the training process.

[0056] Furthermore, to ensure that the user behavior simulation model learns in accordance with the basic laws of human biomechanics, a biomechanical constraint loss term based on a physics engine is added to the training loss function. This biomechanical constraint loss term penalizes simulated actions in the user behavior simulation model that cause virtual joint angles to exceed physiological ranges or the centroid projection to exceed the boundary of the supporting polygon. Its processing logic is as follows: 1) In each training iteration, after the user behavior simulation model outputs a simulated action, the integrated physics engine module (following Newton's laws of motion) calculates the instantaneous state of the virtual human body under that simulated action, especially the angles of each major joint and the projection point of the human body's overall centroid on the ground. 2) The calculated joint angles are compared with the predefined range of human physiological activity (e.g., the knee extension angle should not exceed 180 degrees), and the degree of exceeding the range is quantified as a penalty value. The centroid projection point is also compared with the boundary of the supporting polygon calculated based on the foot position to determine if the projection exceeds the boundary (meaning a simulated fall), and the excess distance is quantified as another penalty value. 3) These penalty values ​​are weighted and summed to form the biomechanical constraint loss term, which, together with the aforementioned prediction error loss term, constitutes the total loss function. During backpropagation, the gradient generated by the biomechanical constraint loss term will guide the update of the user behavior simulation model parameters, suppress the generation of "unreasonable" action strategies that may lead to abnormal joint angles or simulated falls, thereby injecting biomechanical common sense as a strong constraint into the model learning process.

[0057] During training, a biomechanical constraint loss term based on a physics engine is introduced. This term is calculated in the following way to penalize the user behavior simulation model for generating simulated actions that do not conform to the laws of human biomechanics:

[0058] 1) Penalty for joint angles exceeding physiological range: For the j-th joint in the virtual human model, let its current simulated joint angle be θ. j Its predefined lower and upper limits for physiological activity range are respectively and (For example, the knee extension angle range is set to [-5°, 140°]), joint angle constraint loss L joint The calculation is as follows:

[0059] ;

[0060] Where λ j This is the penalty weight coefficient for that joint.

[0061] 2) Penalty term for centroid projection exceeding supporting polygon: Let the projection coordinates of the virtual centroid on the horizontal plane be (x... c ,y cThe set of vertices of the convex hull of the base of support (BoS) formed by the positions of the two feet is: By calculating point (x) c ,y c The shortest distance d to each side of the supporting polygon min To determine if it exceeds:

[0062] If d min ≥0 indicates that the centroid is projected inside or on the boundary of the supporting polygon, and the penalty is zero;

[0063] If dmin < 0, it means the centroid projection exceeds the supporting polygon, and the centroid stability constraint loss L com The calculation is as follows:

[0064] ;

[0065] Where λ com The weight is used to penalize the centroid stability.

[0066] 3) Total biomechanical constraint loss: The two constraint losses mentioned above are combined to form the total biomechanical constraint loss term L. bio :

[0067] ;

[0068] Among them, the loss item L bio Together with the prediction error loss term based on historical data fitting, they constitute the total loss function for training. Backpropagation guides the model to generate simulated movements that conform to biomechanical safety boundaries. Furthermore, a physiological activity range threshold is also considered. and It is pre-set based on authoritative human kinematics datasets (such as the "Standards for Human Joint Range of Motion") and can be fine-tuned according to the individual differences of the target users (such as age, gender, and physical condition).

[0069] In this embodiment of the invention, a biomechanical constraint loss term based on a physics engine is introduced into the pre-training of the user behavior simulation model. This transforms physical laws such as joint mobility and balance stability conditions into a differentiable loss function. The biomechanical constraint loss term and the data fitting target jointly optimize the user behavior simulation model, ensuring that the generated actions conform to individual historical patterns and follow basic human biomechanical principles. This improves the accuracy and reliability of the user behavior simulation model in security prediction and strategy evaluation in a digital twin environment, providing a more reliable simulation foundation for closed-loop optimization systems.

[0070] In one embodiment, the method further includes: constructing a virtual physical environment based on the Unity engine or Unreal engine, which includes various ground materials, obstacles, and visual interference; integrating a biomechanical human body model in the virtual physical environment, the biomechanical human body model including driveable major skeletal joints and muscle force effectors; loading and deploying the pre-trained user behavior simulation model as the central motion controller of the biomechanical human body model to construct the pre-trained digital twin environment.

[0071] The implementation is as follows: First, using the Unity or Unreal Engine as the underlying platform, and leveraging its provided graphics rendering pipeline and built-in physics engine (such as NVIDIA PhysX), a virtual physical environment with various variable parameters is constructed. This virtual physical environment is dynamically configurable and includes ground materials simulating different coefficients of friction (such as smooth ice surfaces and rough carpets), obstacles with different geometric shapes (such as thresholds and moving platforms), and dynamic visual disturbances (such as flashing lights and floating objects). This simulates the complex and ever-changing balance challenges of a real environment. Existing technologies for the Unity or Unreal Engine can be referenced, and will not be elaborated upon here. Second, a biomechanical human body model is integrated into this virtual physical environment. This model is a digital representation of the human body with a hierarchical skeletal structure. Its core includes driveable major skeletal joints (such as hips, knees, and ankles, conforming to biological joint degree-of-freedom constraints) and associated muscle force effectors (used to simulate the effect of muscle activation generating torque, which can be simplified as torque drivers). This biomechanical human body model interacts mechanically with the virtual environment through the physics engine (such as collision detection and gravity effects). Finally, the pre-trained user behavior simulation model is loaded and deployed into the biomechanical human body model as a central motion controller. The user behavior simulation model, as the controller, takes the environmental state (such as user state vector and scene information) as input, and outputs control commands (such as desired joint angle and torque) to each joint actuator or muscle effector through its neural network forward propagation. This drives the biomechanical human body model to make realistic movements that conform to the target user's behavior pattern in the virtual physical environment. This three-layer architecture of "controller-human body model-virtual environment" constitutes the pre-trained digital twin environment.

[0072] This invention integrates a game engine, a biomechanical human body model, and a pre-trained user behavior simulation model to construct a high-fidelity, interactive pre-trained digital twin environment. This pre-trained digital twin environment seamlessly couples the user behavior learning model with the physical simulation engine, enabling corresponding risk prediction and security verification to be performed in a highly realistic and reproducible virtual environment. This improves the credibility of simulation predictions and the engineering feasibility of closed-loop optimization, providing an efficient virtual testing platform for personalized strategy optimization.

[0073] In one embodiment, driving the deployed user behavior simulation model to run and obtaining the simulated performance characteristics of the user behavior simulation model within a preset time window includes: initializing the state of the deployed user behavior simulation model based on the user state vector; inputting a set of testable virtual balance challenge tasks covering different difficulty gradients into the user behavior simulation model within the preset time window; recording the trajectory data generated by the user behavior simulation model when executing each of the testable virtual balance challenge tasks, and calculating the following indicators based on the trajectory data: task success rate, average trajectory smoothness index, and area of ​​the centroid oscillating ellipse; normalizing the task success rate, the average trajectory smoothness index, and the area of ​​the centroid oscillating ellipse, and concatenating them into a vector of fixed dimensions as the simulated performance characteristics.

[0074] Among them, testable virtual balance challenge tasks refer to a set of standardized virtual scene perturbations or interaction targets that are pre-designed and applied in a pre-trained digital twin environment to actively evaluate the balance ability limits of the user behavior simulation model in the current state. Different testable virtual balance challenge tasks constitute a test set covering different challenge intensities (difficulty gradients), including but not limited to: tasks of standing on a continuously swaying platform within a specified time, tasks of crossing a set of obstacles with unequal spacing along a specified path, and tasks of resisting instantaneous virtual thrusts applied in different directions and intensities.

[0075] The implementation is as follows: First, based on the input user state vector, the initial state of the user behavior simulation model in the pre-trained digital twin environment is initialized. Then, within a preset time window, the system sequentially or in parallel applies a set of predefined test virtual balance challenge tasks covering different difficulty gradients to the user behavior simulation model. These tasks are implemented through environmental interfaces (such as adjusting platform motion equations, generating obstacle sequences, and applying force vectors). Second, the system drives the user behavior simulation model to attempt to complete these tasks one by one, and records the trajectory data generated by the virtual body controlled by the user behavior simulation model during the execution of each task at a high frequency, such as the center of mass (COM) trajectory, foot position trajectory, and joint angle trajectory. Furthermore, based on the recorded trajectory data, the system extracts quantitative performance indicators including but not limited to the following three dimensions through deterministic mathematical calculations, including but not limited to the following exemplary examples: 1) Task success rate: For each test virtual balance challenge task, a binary judgment is performed according to preset success criteria (such as not falling, reaching the target point within a specified time), and the success rate of all tasks is statistically analyzed. 2) Average trajectory smoothness index: This is typically calculated by the root mean square (RMS) value of the center of mass velocity or acceleration, or the integral of jerk (accelerometer). A lower value indicates smoother motion and more precise control. 3) Area of ​​the center-of-mass ellipse: This is calculated by statistically analyzing the projection points of the center of mass onto the horizontal plane within a preset time window and fitting an ellipse containing 95% of the data points. The area of ​​this ellipse is measured in cm². 2 The overall range and stability of the posture sway were quantified. Third, the three original indicators (which may have different numerical units) were normalized (e.g., scaled to the [0,1] interval) and then concatenated into a fixed-dimensional numerical vector, which is the simulated performance feature.

[0076] In this embodiment of the invention, by applying a testable virtual balance challenge task to a user behavior simulation model and generating a simulation performance feature vector based on multi-dimensional quantitative indicators such as task success rate, trajectory smoothness, and centroid sway area, an objective and standardized evaluation of the dynamic performance of the user behavior simulation model is achieved. This provides a refined and discriminative state input for subsequent VR training strategy optimization, thereby improving the accuracy and guidance of VR training strategy optimization.

[0077] In one embodiment, please refer to Figure 3 , Figure 3 This is a schematic diagram of the first sub-process of the method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning, as provided in an embodiment of the present invention. Figure 3As shown, in this embodiment, the N-step simulation prediction of fall prevention safety is performed through the user behavior simulation model to obtain the safety prediction result, including: S31: Initializing the virtual training scene in the pre-trained digital twin environment with the VR training strategy parameter set; S32: Driving the user behavior simulation model to continuously execute N time steps of action in the virtual training scene; S33: Calculating and recording the following indicators at each time step: the projection coordinates of the virtual center of mass relative to the foot support surface, the instantaneous torque of the main load-bearing joints, and the predicted fall risk probability, wherein the fall risk probability is calculated by a lightweight risk assessment network pre-trained with labeled data; S34: Serializing the projection coordinates, instantaneous torque, and fall risk probability recorded in the N time steps according to the time step order, and outputting them as the safety prediction result.

[0078] Pre-trained lightweight risk assessment network: This refers to a pre-trained, simple ("lightweight" means fewer parameters and faster computation) deep learning model. Its technical function is to assess and output a probability value of fall risk in real time based on input real-time simulated biomechanical data. This risk assessment network is trained with a large amount of labeled simulated or real data (data labeled "fall" or "no fall") to learn the nonlinear mapping relationship from dynamic and kinematic characteristics (such as center of mass shift and joint torque) to fall risk. It is usually composed of several layers of fully connected networks or small convolutional networks. During the simulation, it acts as an efficient online risk discriminator, making up for the shortcomings of judgment based solely on geometric rules (such as center of mass projection out of bounds).

[0079] The implementation is as follows: S31: The set of VR training strategy parameters to be evaluated (such as obstacle height and perturbation mode) is input into the pre-training digital twin environment as a configuration instruction. The pre-training digital twin environment dynamically adjusts the specific attributes of the virtual training scene accordingly (such as loading obstacle models of corresponding heights and setting perturbation parameters) to complete the initialization of the simulation environment used for this prediction. S32: Starting from the current user state vector (or continuing from the state at the end of the previous simulation step), the system drives the user behavior simulation model to start running in the initialized virtual training scene. The user behavior simulation model generates and executes actions for N time steps continuously according to its internal strategy and interacts with the virtual environment. The physics engine calculates the dynamic state of each time step in real time. S33: At each time step of the simulation execution, the system performs three parallel calculations: 1) The projection coordinates of the virtual centroid relative to the support surface of the two feet, which is a geometric stability index. Based on the virtual body state output by the physics engine, the coordinates of the projection point of the virtual centroid (CoM) on the horizontal plane relative to the support polygon (BoS) formed by the positions of the two feet are calculated. These coordinates directly reflect the static equilibrium margin. 2) The instantaneous torque of the main weight-bearing joints is a biomechanical load indicator. The system obtains the instantaneous torque of the main weight-bearing joints (such as ankle, knee, and hip) based on inverse dynamics calculation or physics engine feedback. This torque reflects the mechanical load intensity of the joint. 3) The predicted fall risk probability is a comprehensive risk assessment indicator. The multidimensional state information of the current time step (which may include the above coordinates, torque, and other kinematic features) is used as input to a pre-trained lightweight risk assessment network. The risk assessment network calculates through its forward propagation and outputs a scalar between 0 and 1, which is the predicted fall risk probability. The fall risk probability incorporates dynamic instability characteristics that are difficult to quantify with simple rules. S34: The projected coordinates, instantaneous torque, and fall risk probability calculated at each of the above N time steps are structured and organized in the order of the time steps to form a time series data block, i.e., the safety prediction result, which retains the complete temporal dynamic information of risk evolution.

[0080] This invention, through simultaneous calculation of geometric stability, biomechanical load, and comprehensive risk probability based on neural networks in N-step simulations, outputs safety prediction results including temporal evolution. This enables multi-dimensional, high-frequency in-depth analysis of potential risks in the VR training strategy parameter set for fall prevention, providing rich and accurate decision-making basis for subsequent safety verification and optimization of the VR training strategy parameter set. This allows safety optimization to precisely intervene at specific risk points and biomechanical factors, improving the method's safety assurance capability and the level of strategy optimization refinement.

[0081] In one embodiment, based on the safety prediction results, the VR training strategy parameter set is subjected to safety verification and optimization adjustment, including: performing time series analysis on the safety prediction results; if the probability of fall risk exceeds a first threshold for K consecutive time steps, or the instantaneous torque on any joint exceeds a second threshold, the VR training strategy parameter set is determined to have failed the safety verification, where K≤N and is a natural number; if the VR training strategy parameter set is determined to have failed the safety verification, the time step information with the highest fall risk in the safety prediction results is used as a negative reward signal and input into the strategy optimization generator, triggering it to perform gradient update on the VR training strategy parameter set based on the negative reward signal, generating a new VR training strategy parameter set after optimization adjustment; the new VR training strategy parameter set is input again into the pre-trained digital twin environment for N-step simulation prediction of fall prevention safety, and iteratively executed until the VR training strategy parameter set passes the safety verification or reaches the maximum number of iterations.

[0082] The implementation is as follows: First, the system performs time series analysis on the safety prediction results (N-step time-series data), and the verification is based on explicit numerical logic rules: if the system detects that the fall risk probability value exceeds a preset first threshold (e.g., 0.7) for K consecutive time steps (K≤N) in the safety prediction results, or if the instantaneous torque of any major load-bearing joint exceeds a preset second threshold (e.g., based on biomechanical safety limits) at any time step, the system determines that the parameter set of the VR training strategy to be evaluated has failed the safety verification. Second, the system triggers and executes iterative optimization adjustments based on reinforcement learning. If the verification fails: the system identifies the time step information with the highest fall risk probability (such as the complete state context of that time step) from the safety prediction results and quantifies it as a negative reward signal. Furthermore, the negative reward signal, along with the current state information (or simulated performance characteristics), is input into the policy optimization generator. Based on this feedback, the policy optimization generator (acting as an Actor network in reinforcement learning) performs one or more internal gradient updates (e.g., using update rules of PPO or SAC algorithms), adjusting its network parameters to generate a new set of optimized VR training policy parameters. This new set of VR training policy parameters aims to avoid corresponding action patterns that lead to high risks. Finally, the system inputs the newly generated VR training policy parameter set back into the pre-trained digital twin environment, repeating N steps of simulation prediction to obtain new safety prediction results. The above verification and adjustment process is then repeated. This iterative loop continues until either of the following stopping conditions is met: the latest generated VR training policy parameter set passes the safety verification (i.e., all time-step metrics are below the threshold), or the preset maximum number of iterations is reached (to prevent infinite loops).

[0083] In this embodiment of the invention, the high-risk moment information of the VR training strategy parameter set that fails the verification is used as a negative reward to drive the strategy optimization generator to perform gradient updates and iterative replanning. This achieves rigid constraints on the safety boundary and directional guidance of the optimization direction, which can quickly eliminate unsafe strategies and perform efficient strategy search in the direction of reducing the corresponding risks. While ensuring safety, it improves the convergence speed, controllability and safety and reliability of the final VR training strategy.

[0084] In one embodiment, updating the user behavior simulation model and the policy optimization generator using the real-time multimodal data includes: (i) online learning update of the user behavior simulation model: constructing new training samples based on newly collected real-time multimodal data and the parameter set of the corresponding executed VR training policy, and updating the parameters of the user behavior simulation model by minimizing a composite loss function, wherein the composite loss function includes a prediction error term based on the new training samples, and a penalty term for the deviation of the parameters of the user behavior simulation model from their historical convergence values, the strength of the penalty term being weighted by the diagonal elements of the historical Fisher information matrix corresponding to the parameters; (ii) updating the policy optimization generator based on priority experience replay: according to the execution of the VR... The changes in user state vectors before and after training the training policy parameter set are used to construct an empirical tuple. The absolute value of the temporal difference error of the empirical tuple is used as its initial sampling priority and stored in the empirical playback buffer of the policy optimization generator. Batch data is sampled from the empirical playback buffer according to the non-uniform distribution based on the sampling priority, and the sampling bias is corrected by the importance sampling weight. The gradient is calculated and the network parameters of the policy optimization generator are updated. The sampling priority of the sampled empirical tuple is updated. (III) Meta-learning adjustment of the user behavior simulation model and the policy optimization generator: The overall strength coefficient of the penalty term mentioned in (I) above is dynamically adjusted according to the KL divergence between the policy distributions of the output of the policy optimization generator before and after the update.

[0085] KL divergence: also known as Kullback-Leibler divergence, in this embodiment of the invention, represents an index used to quantify the update magnitude of the policy optimization generator. It calculates the Kullback-Leibler divergence between the policy distribution (or the statistical characteristics of the VR policy parameter set) output by the policy optimization generator before and after the update. The resulting scalar value is used to dynamically adjust the regularization constraint strength during the model update process to control the stability of policy iteration.

[0086] This invention provides a detailed technical implementation of "updating the user behavior simulation model and the policy optimization generator," which includes three interrelated online learning mechanisms:

[0087] (I) Online Learning Updates of the User Behavior Simulation Model. This process aims to fine-tune the user behavior simulation model using newly acquired incremental real-time multimodal data, while preventing catastrophic forgetting (i.e., forgetting previously learned useful knowledge). This is implemented as follows: First, the newly acquired real-time multimodal data is paired with the corresponding parameter sets of the executed VR training strategy to form new training sample pairs. Second, based on the new training sample pairs, the parameters of the user behavior simulation model are updated by minimizing the following composite loss function:

[0088] L_total = L_new(θ) + λ * Σ_i [ F_i * (θ_i - H_i)^2 ];

[0089] Where L_new(θ) is the prediction error loss calculated based on the new training sample pair, θ is the current parameter vector to be updated, H_i is the historical anchoring parameter value corresponding to the model parameter θ_i, F_i is the diagonal element of the Fisher information matrix that is proportional to the importance of parameter θ_i in the historical task, and λ is a hyperparameter controlling the strength of the forgetting penalty. H_i and F_i are determined based on the model parameter snapshots saved during historical training and the second-order gradient information of the loss function on the parameters. The value of H_i is determined to be the parameter value recorded in the parameter snapshot when θ_i converged on the corresponding historical task. When updating the parameter θ through backpropagation, this term F_i * (θ_i - H_i)^2 will penalize the behavior that deviates from the important historical parameter H_i, thereby slowing down the forgetting of learned knowledge.

[0090] Furthermore, after each historical training phase T, a snapshot of the model parameters θ_T is saved as the "anchor parameter" H for that phase. The historical anchor parameter value H_i is determined as follows: after completing the k-th historical training phase, a snapshot of the user behavior simulation model parameters θ_k at the convergence of that phase is saved, and θ_k is used as the H corresponding to that phase. Simultaneously, the approximate value F of the diagonal elements of the Fisher information matrix of the training loss function L_T with respect to the model parameters θ is calculated. Specifically, F_i is estimated by calculating the mean of the squared gradients of the loss function with respect to the parameters θ_i on multiple data batches collected in phase T. Therefore, when updating the user behavior simulation model, instead of simply minimizing the prediction error of the new training samples (L_new(θ)), we minimize a composite loss function L_total. This function L_total adds an elastic weight penalty term (λ * Σ_i [ F_i * (θ_i - H_i)^2 ]) to L_new(θ). Here, H_i is the anchor value of each model parameter θ_i when it performs well on historical tasks (from historical model snapshots), and F_i is the diagonal element of the Fisher information matrix reflecting the importance of the parameter to historical tasks (calculated based on the second-order gradient of the historical loss). This elastic weight penalty term penalizes the degree to which important historical parameters deviate from their historical values, and λ controls the intensity of the penalty. By optimizing this composite loss, the user behavior simulation model is constrained to retain its memory of historical tasks while adapting to new training samples.

[0091] The method for obtaining and calculating the diagonal elements of the historical Fisher information matrix is ​​as follows:

[0092] After completing the k-th historical training phase, save a snapshot θ_k of the user behavior simulation model parameters at the convergence point of that phase. Simultaneously, based on the training sample set of that training phase, approximate the diagonal elements of the Fisher information matrix using the following steps: 1) For the i-th model parameter θ_i, calculate its loss function gradient g_i on each training sample; 2) Average the squared gradients of all training samples in that training phase, using this average as an approximation of the Fisher information diagonal element F_i corresponding to parameter θ_i.

[0093] N is the number of samples;

[0094] 3) Store the calculated F_i together with the corresponding anchoring parameter θ_k for use in the calculation of the elastic weight penalty term in the subsequent online learning stage.

[0095] (ii) The policy optimization generator is updated based on priority experience replay. Here, priority experience replay represents a reinforcement learning experience sampling mechanism that determines the sampling priority of experience tuples based on the absolute value of their temporal difference error, and corrects sampling bias through importance sampling weights to improve learning efficiency.

[0096] The implementation is as follows: First, based on the newly acquired real-time multimodal data, a new user state vector is generated after training with the VR training policy parameter set a_t. Then, the corresponding new simulated performance feature s_{t+1} is obtained through a pre-trained digital twin environment. The difference between the new user state vector and the user state vector generated before training with the VR training policy parameter set a_t is calculated in a preset dimension to obtain the state improvement amount Δ, which is then mapped to a scalar reward value r_t. Second, the simulated performance feature s_t before training with the VR training policy parameter set, the VR training policy parameter set a_t, the scalar reward value r_t, and the new simulated performance feature s_{t+1} are combined to form an empirical tuple (s_t, a_t, r_t, s_{t+1}), which is stored in the empirical replay buffer of the policy optimization generator. The absolute value of the initial temporal difference error of the empirical tuple is calculated as the initial sampling priority of the empirical tuple. For example, when storing an empirical tuple (s_t, a_t, r_t, s_{t+1}) into the empirical playback buffer D, its initial priority p is set to the absolute value of the timing difference error (TD-error) δ of the empirical tuple, i.e., p = |δ|. , where γ is the discount factor, V is the state value function output by the commentator network, and s_t (State at time t) represents the simulated performance characteristics on which the driving policy optimization generator generates the VR training policy parameter set a_t at time t; a_t (Action at time t) represents the VR training policy parameter set output by the policy optimization generator at time t; r_t (Reward at time t) represents the reward value obtained after executing the VR training policy parameter set a_t, which is calculated from the improvement Δ of the user state before and after training the VR training policy parameter set, and is used to quantify the training effect of the VR training policy parameter set; s_{t+1} represents the new simulated performance characteristics generated based on the real-time multimodal data collected after executing the VR training policy parameter set a_t. Third, when the number of experience tuples in the experience replay buffer reaches a preset threshold, the following batch update process is executed: Fourth, for the i-th experience tuple in the experience replay buffer, based on its stored sampling priority p_i, the probability P(i) of being sampled is calculated (the higher the priority, the greater the probability), specifically: P(i) = p_i^α / Σ_k p_k^α, where i and k are indices, p_i is the "sampling priority" of the i-th experience tuple, α is the priority adjustment coefficient, a hyperparameter controlling the concentration of the sampling distribution, and Σ_k represents the summation over all experience tuples in the buffer. Fifth, based on the distribution formed by the sampling probabilities P(i), non-uniform random sampling is performed to obtain a batch of experience tuples. Sixth, calculate the importance sampling weight w_i for the i-th sampled empirical tuple in this batch, specifically: w_i = (N * P(i))^{-β}, where N is the total number of empirical tuples in the empirical replay buffer, and β is a hyperparameter controlling the bias correction intensity to correct the bias introduced by non-uniform sampling. Seventh, use the loss function adjusted by the importance sampling weight to calculate the gradient based on the empirical tuples in this batch, and update the network parameters θ of the policy optimization generator. Eighth, for the sampled empirical tuples, recalculate their temporal difference error TD-error δ' using the updated network parameters, and update their sampling priority p_i = |δ'| stored in the buffer to ensure that the sampling priority dynamically reflects the latest "value" of the experience.

[0097] (III) Meta-learning adjustment of the user behavior simulation model and the policy optimization generator. This process aims to achieve adaptive coordination between the user behavior simulation model update and the policy optimization generator update, avoiding the failure of the other due to excessively rapid changes in one. The implementation is as follows: First, after each batch update of the policy optimization generator, the KL divergence between the policy distributions output by the policy optimization generator before and after the update is calculated as the change in policy distribution. For example, before and after each update of the policy optimization generator parameter θ_π, the same batch of state inputs is used to obtain the output policy distributions π_old and π_new, and the KL divergence between the two policy distributions is calculated: D_KL = KL(π_old || π_new), as the change in policy distribution. Second, the change in policy distribution is used as a meta-reward signal and input into the online learning controller in the user behavior simulation model update. The online learning controller dynamically adjusts the value of the hyperparameter λ during the user behavior simulation model update process accordingly, so that the learning rate of the user behavior simulation model and the update magnitude of the policy optimization generator are adaptively coordinated. For example, when the policy distribution changes drastically (policy shifts rapidly), λ is increased to enhance the memory stability of the user behavior simulation model; when the changes are small, λ is decreased to enhance its adaptability to new real-time multimodal data. This creates a closed-loop synergy between the update rhythm of the user behavior simulation model and the policy optimization generator. For instance, by inputting D_KL into a simple proportional controller (e.g., λ_new = λ_base + η * D_KL, where η is the proportional coefficient), the hyperparameter λ in the update of the user behavior simulation model is adjusted in real time. When the VR training policy changes drastically (D_KL is large), λ is increased to enhance the stability of the user behavior simulation model; conversely, λ is decreased to enhance its adaptability.

[0098] This invention integrates elastic weight consolidation, priority experience replay, and a meta-learning collaborative mechanism based on KL divergence to update the user behavior simulation model and the policy optimization generator online, respectively. This solves problems such as catastrophic forgetting, low sample efficiency, and system oscillation caused by the loss of synchronization between the two models in continuous learning, and achieves stability, efficiency, and collaboration in model updates, ensuring the long-term stable operation and adaptive evolution capability of the optimization closed loop.

[0099] In one embodiment, the policy optimization generator is implemented using an actor-commentator network architecture based on an attention mechanism, and is trained and updated using an asynchronous advantage actor-commentator algorithm based on priority experience replay. In the gradient update process, a KL divergence constraint term for the policy parameter distribution is introduced to control the magnitude of the policy update.

[0100] The implementation is as follows: (I) The actor-critic network architecture based on the attention mechanism is the specific neural network implementation of the policy optimization generator. The policy optimization generator can be implemented using the actor-critic network architecture based on the multi-head attention mechanism. The specific network structure is as follows:

[0101] Actor Network: 1) Input Layer: Receives simulated performance feature vectors s. 2) Multi-head Self-Attention Layer: Projects the input features s through three independent linear transformation layers into a query matrix (Q), a key matrix (K), and a value matrix (V), each with dimension R. n*d k Where n is the sequence length (in this embodiment of the invention, the simulated performance feature is a single vector, so n=1), and dk is the projection dimension; the attention score is calculated by calculating Q and the transpose of K (K T The dot product of the query matrix and the key matrix yields the original relevance score between them. To prevent the relevance score from being affected by dimension d, the product of the query matrix and the key matrix is ​​calculated. k A large value leads to an excessively large dot product result, affecting the numerical stability of the subsequent Softmax function. Therefore, the dot product result is divided by... The scores are scaled, and then normalized to a probability distribution (i.e., attention weights) using the Softmax function.

[0102]

[0103] Among them, QK T This indicates the calculation of the matching degree between the query matrix and the key matrix. Using a scaling factor, the obtained attention weights are multiplied by the value matrix V to obtain a weighted summation of the context-aware feature representation. Furthermore, h attention heads are computed in parallel, and their outputs are concatenated and fused through a linear layer to obtain the attention-enhanced feature representation s′. 3) Feedforward Neural Network Layer: s′ is non-linearly transformed through two fully connected layers (containing a ReLU activation function in between). 4) Output Layer: Outputs the mean vector μ and log-standard deviation vector logσ of the VR training policy parameter set. 5) Policy Sampling: Utilizing reparameterization techniques... (in Generate the final VR training strategy parameter set a.

[0104] Critic Network: 1) Input Layer: Simultaneously receives the simulated performance feature vector s and the VR training policy parameter set a output by the actor network. 2) Feature Concatenation and Projection: Concatenates s and a into a joint feature vector and projects it to a unified dimension through a linear layer. 3) Cross-Attention Layer: Uses the projected state features as the query and the projected action features as the key and value; calculates the cross-attention between the state and action to obtain the state-action interaction feature representation. 4) Fully Connected Evaluation Layer: Passes the interaction features through several fully connected layers, finally outputting a scalar Q(s,a), representing the expected cumulative reward for taking action a in state s.

[0105] The attention mechanism plays the following roles in this embodiment of the invention: 1) In the actor network, the self-attention mechanism is used to capture the long-range dependencies between various dimensions within the simulated performance features (e.g., the correlation between task success rate and the area of ​​centroid sway), thereby gaining a more comprehensive understanding of the target user's current balance ability state. 2) In the critic network, the cross-attention mechanism is used to dynamically align state features and action features, evaluate the applicability of different policy parameters to the current user state, and improve the accuracy of value estimation.

[0106] Furthermore, here are some network parameter examples (adjustable): Number of attention heads h=4; Attention dimension d k =d v =64; The output dimension of the actor network is consistent with the dimension of the VR training strategy parameter set (for example, if the VR training strategy parameter set contains 5 parameters, then the output dimension is 5).

[0107] (II) The training and updating process of the asynchronous advantage actor-critic algorithm based on priority experience replay includes: First, multiple asynchronous parallel training agent threads are started, with each agent thread sharing the network parameters of the policy optimization generator. Specifically, the system starts multiple agent threads, each copying a set of shared global network parameters. In this embodiment, an agent thread represents an independent parallel computing instance running in a computer program. Each agent thread has independent computing resources and can asynchronously perform tasks such as sampling from the experience replay buffer and calculating gradients. Multiple agent threads running simultaneously constitute a distributed, parallel training environment, aiming to accelerate data acquisition and gradient calculation. Second, each agent thread independently and asynchronously performs the following operations: sampling a batch of experience tuples from the experience replay buffer, adjusting the loss function based on this batch of experience tuples using importance sampling weights w_i, and calculating the local gradients of the actor network and the critic network. Specifically, each agent thread runs independently and asynchronously: based on sampling priority, it samples a batch of experience tuples from the experience replay buffer; based on the collected experience tuples, it adjusts the loss function using the importance sampling weights w_i defined above, and independently calculates the local gradients of the actor network and the critic network. Here, the local gradient represents the gradient vector calculated independently by each independent agent thread for the shared global actor network and critic network parameters, based on its own batch of experience tuples sampled from the experience replay buffer within the distributed training framework based on the Asynchronous Advantage Actor-Critic (A3C) algorithm. Third, a central gradient aggregator is established to asynchronously receive the local gradients from each agent thread. The central gradient aggregator represents a centralized software module or process whose function is to receive the local gradients calculated by each asynchronously running agent thread, summarize (e.g., average) all local gradients to form a global gradient, and periodically (not in real-time) use this global gradient to update a shared, global set of network parameters. Specifically, a central gradient aggregator is established. The local gradients calculated by each agent thread are asynchronously sent to the central gradient aggregator. The central gradient aggregator periodically (e.g., after collecting local gradients from M agent threads) performs a weighted average or summation of all received local gradients to obtain the global gradient. Subsequently, the shared global network parameters are updated once using this global gradient. After the update is complete, the new global parameters are synchronously distributed to all agent threads for their use in the next round of computation. Fourth, the central gradient aggregator periodically applies the aggregated local gradients to update the global parameters of the shared actor network and critic network. After the global parameters are updated, they are synchronized to all agent threads. Fifth, during the gradient update process, a KL divergence constraint term for the policy parameter distribution is synchronously introduced to control the magnitude of the policy update.Specifically, the introduction of KL divergence constraints to control the update magnitude involves adding a KL divergence constraint term to the total loss function used by the central gradient aggregator to calculate the global gradient, in addition to the basic loss based on temporal difference error. The specific processing is as follows: Before each global parameter update, the current actor network policy distribution (defined by the old parameter θ_old) is recorded; the KL divergence D_KL(π_old || π_new) between the actor network policy distribution after using the new parameter θ_new and the old distribution is calculated; this KL divergence value is multiplied by a penalty coefficient and added as an additional loss term to the total loss. The KL divergence constraint term represents an additional regularization term added to the loss function of the policy optimization generator. This regularization term, calculated based on KL divergence, quantifies the difference between the policy output distribution (defined by the actor network parameters) before and after the update. Adding this difference term to the total loss and minimizing the total loss during gradient updates constrains the magnitude of each policy update, preventing drastic and unstable policy changes, thereby improving the stability of VR training.

[0108] Among them, the policy optimization generator adopts the asynchronous advantage actor-commentator (A3C) algorithm based on priority experience replay. This is because the asynchronous parallel framework of A3C can accelerate the collection and learning of a large amount of simulation experience data, while the priority replay mechanism can focus on learning those experiences with large prediction errors and higher value, thus improving the efficiency of policy generation while ensuring the stability of model updates.

[0109] This invention employs an actor-critic network with an integrated attention mechanism, an asynchronous parallel training architecture based on priority experience replay, and introduces KL divergence constraints in gradient updates to achieve efficient and stable training of the policy optimization generator. This solves the problems of slow convergence, low sample efficiency, and policy update oscillations in deep reinforcement learning, and improves the efficiency, robustness, and controllability of the policy search process. It enables the VR training policy optimization closed loop to converge to a high-quality, highly secure personalized policy more quickly and stably.

[0110] In one embodiment, the method further includes: during the execution of the fall prevention VR virtual simulation training scenario, the VR training terminal calculates the skeletal posture data of the target user in the current frame in real time; the skeletal posture data is matched and compared with a preset safe posture boundary template; if the skeletal posture data is detected to exceed the preset safe posture boundary, the rendering and force feedback of the fall prevention VR virtual simulation training scenario are interrupted, an emergency protection prompt is triggered, and abnormal event data containing the skeletal posture data is synchronized to the management server; the management server receives the abnormal event data and associates it with the corresponding VR training strategy parameter set to form a negative reward sample for reinforcement learning, which is used to update the user behavior simulation model and / or the strategy optimization generator to reduce the probability of generating VR training strategy parameter sets that lead to similar abnormal postures in the future.

[0111] Skeletal posture data: This represents the quantitative data of the user's body posture at the current moment, captured and calculated by sensors (such as a built-in IMU or optical tracking system) of the VR training terminal. It is expressed in three-dimensional coordinates of human skeletal joints (such as hip, knee, and ankle) and relative orientation angles, describing the instantaneous geometric configuration of the target user in physical space. Preset safe posture boundary template: This represents the safety threshold rules predefined and stored in the VR training terminal based on biomechanical safety principles. This template sets numerical allowable ranges for key parameters in the skeletal posture data (such as joint angles and trunk tilt angles) to determine in real time whether the user's posture is in a safe state. For example, the knee flexion angle should not exceed a certain maximum value to prevent hyperextension injury.

[0112] The implementation is as follows: First, during the execution of the training scene, the VR training terminal runs a posture calculation algorithm to calculate the skeletal posture data of the target user in the current frame based on the data collected by the sensors. The posture calculation algorithm outputs the following: based on the output of the VR training terminal's built-in inertial measurement unit (IMU) and optical tracking sensors, it calculates the three-dimensional spatial coordinates and rotation angles of the user's main joints in real time, generating skeletal posture data. Second, the VR training terminal compares the skeletal posture data with a locally stored preset safe posture boundary template. This comparison is based on a real-time logical comparison operation between numerical values ​​(such as angles and coordinates) and preset thresholds (boundaries) to determine whether the current posture exceeds the safe boundary (for example, detecting that the knee joint hyperextension angle exceeds the preset safe threshold). Furthermore, if an out-of-bounds error is detected (i.e., abnormal posture): the VR training terminal immediately interrupts the graphics rendering output of the current fall prevention VR training scene and the drive signal of the force feedback device, forcibly suspending or terminating virtual stimuli that may cause risks. Simultaneously, it triggers emergency protection prompts (such as screen flashing warnings and safety sound effects) to physically alert the user. Finally, the VR training terminal sends an abnormal event data packet containing abnormal skeletal posture data, timestamps, and other information to the management server. Upon receiving the data, the management server precisely correlates the abnormal event with the VR training policy parameter set corresponding to the training round that caused the abnormality. Finally, the management server constructs a negative reward sample for reinforcement learning from the correlated "VR training policy parameter set - abnormal event" pair. This negative reward sample, acting as a strong negative feedback signal (i.e., the policy led to a dangerous posture), is used to calculate the loss or adjust the reward when subsequently updating the user behavior simulation model and / or the policy optimization generator. Through optimization methods such as gradient descent, this drives the corresponding model parameter updates, thereby systematically reducing the probability of generating VR training policy parameter sets that may lead to similar abnormal postures in the future.

[0113] This invention, through real-time calculation of skeletal posture on a VR training terminal and comparison with preset safety boundaries, enables immediate safety interruption and anomaly reporting during fall prevention VR training. It also associates abnormal event data with the corresponding VR training strategy parameter set, constructs negative reward samples to drive corresponding model updates, solves the problem of real-time protection and long-term avoidance of sudden extreme dangerous postures, and achieves the goal of learning from real safety events to improve strategy security, thereby enhancing the system's real-time security and proactive risk avoidance capabilities.

[0114] It should be noted that the digital twin-driven two-layer reinforcement learning method for fall prevention VR training strategy optimization involved in the embodiments of the present invention is a balance ability training strategy optimization tool based on virtual reality and artificial intelligence technology. By processing the user's kinematics, interaction data and physiological feedback signals, it aims to optimize the user's training experience and effect in the virtual environment and ensure the safety of the training process. The embodiments of the present invention do not involve the diagnosis of any disease, and the generated and optimized strategy parameters do not constitute any medical treatment or rehabilitation plan. Before actual use, users should consult relevant professionals based on their own circumstances.

[0115] The digital twin-driven two-layer reinforcement learning method for fall prevention VR training strategy optimization described in the above embodiments can be recombined with the technical features included in different embodiments as needed to obtain a combined implementation scheme, but all are within the protection scope claimed by this invention.

[0116] Those skilled in the art will understand that the methods provided in the embodiments of the present invention can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. The methods can also be implemented as a computer program product stored in one or more computer-readable storage media, including but not limited to: disks, optical disks, read-only memory (ROM), random access memory (RAM), flash memory, etc. When the computer program product is executed by one or more data processing devices (such as computers), the devices perform the steps as described in any of the foregoing method embodiments.

[0117] Software tools, components, or models not belonging to this company that appear in the embodiments of this invention are merely illustrative examples and do not represent actual use. The data collection methods used in the embodiments of this invention comply with relevant laws and regulations, such as the GDPR (General Data Protection Regulation) or information security standards of other countries and regions.

[0118] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning, characterized in that, The method, applied to a system including a management server and a VR training terminal, includes: The management server acquires real-time multimodal data of the target user and inputs it into a preset feature encoding network to generate a user state vector; The user state vector is input into the pre-trained digital twin environment corresponding to the target user, and the user behavior simulation model deployed therein is driven to run, so as to obtain the simulated performance characteristics of the user behavior simulation model within a preset time window. The user behavior simulation model is a deep reinforcement learning-based model, and the historical VR training data of the target user is used to complete the pre-training. A deep reinforcement learning-based policy optimization generator is invoked, using the simulated performance features as state input, to generate a VR training policy parameter set; The VR training strategy parameter set is input into the pre-trained digital twin environment, and the user behavior simulation model is used to perform an N-step simulation prediction of fall prevention safety to obtain the safety prediction result, where N is a natural number greater than 1. Based on the security prediction results, the VR training strategy parameter set is subjected to security verification and optimization adjustment; The VR training strategy parameter set, which has been verified and optimized, is sent to the VR training terminal to execute the corresponding fall prevention VR virtual simulation training scenario. Real-time multimodal data of the target user is collected during the execution of the fall prevention VR virtual simulation training scenario, and the user behavior simulation model and the strategy optimization generator are updated using the real-time multimodal data to form an optimization closed loop of the VR training strategy.

2. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, The real-time multimodal data includes the target user's physiological time-series signals, three-dimensional skeletal joint motion trajectory sequences, and VR interactive event streams; Generate user state vectors, including: The physiological time-series signal is input into the first encoding sub-network to obtain a physiological feature vector. The first encoding sub-network includes a preprocessing layer for filtering out baseline drift and an LSTM layer for extracting heart rate variability features. The three-dimensional skeletal joint motion trajectory sequence is input into the second encoding sub-network to obtain the kinematic feature vector. The second encoding sub-network includes an inverse kinematic layer for calculating joint angles and angular velocities and a temporal convolutional layer for capturing motion patterns. The VR interactive event stream is input into the third encoding sub-network to obtain the interactive feature vector. The third encoding sub-network includes an embedding layer for statistical event frequency and a Transformer layer for modeling attention distribution. The physiological feature vector, the kinematic feature vector, and the interaction feature vector are concatenated and then dimensionality-reduced and fused through a fully connected layer to output the corresponding user state vector.

3. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, The historical VR training data includes a sequence of historical VR training interaction data arranged in chronological order; the pre-training process of the user behavior simulation model includes: Obtain the historical VR training interaction data sequence of the target user, which includes a historical user state vector, a historical VR training strategy parameter set, and corresponding historical user response data. Using the historical user state vector and the historical VR training strategy parameter set as input states, and minimizing the prediction error of the historical user response data as the training objective, the near-end policy optimization algorithm is used to iteratively train the user behavior simulation model. During training, a biomechanical constraint loss term based on a physics engine is introduced. This biomechanical constraint loss term is used to penalize simulated actions in the user behavior simulation model that cause virtual joint angles to exceed physiological ranges or centroid projections to exceed the boundaries of the supporting polygons.

4. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, The method further includes: A virtual physical environment with various ground materials, obstacles, and visual interference can be built using the Unity engine or Unreal engine. In the virtual physical environment, an integrated biomechanical human body model is included, which contains driveable major skeletal joints and muscle force effectors. The pre-trained user behavior simulation model is loaded and deployed as the central motion controller of the biomechanical human body model to construct the pre-trained digital twin environment.

5. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, Drive the deployed user behavior simulation model to run, and obtain the simulated performance characteristics of the user behavior simulation model within a preset time window, including: Based on the user state vector, initialize the state of the user behavior simulation model deployed therein; Within a preset time window, a set of testable virtual balance challenge tasks covering different difficulty levels are input into the user behavior simulation model; Record the trajectory data generated when the user behavior simulation model executes each of the test virtual balance challenge tasks, and calculate the following indicators based on the trajectory data: task success rate, average trajectory smoothness index, and area of ​​the centroid undulating ellipse; The task success rate, the average trajectory smoothness index, and the area of ​​the centroid undulating ellipse are normalized and concatenated into a fixed-dimensional vector as the simulation performance features.

6. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, The user behavior simulation model is used to perform an N-step simulation prediction of fall prevention safety, and the safety prediction results are obtained, including: The virtual training scene in the pre-trained digital twin environment is initialized with the VR training strategy parameter set; The user behavior simulation model is driven to continuously execute actions for N time steps in the virtual training scenario; At each time step, the following metrics are calculated and recorded: the projected coordinates of the virtual center of mass relative to the foot support surface, the instantaneous torque of the main load-bearing joints, and the predicted fall risk probability, which is calculated by a lightweight risk assessment network pre-trained with labeled data. The projected coordinates, instantaneous torque, and fall risk probability recorded at the N time steps are serialized in time step order and output as a safety prediction result.

7. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 6, characterized in that, Based on the security prediction results, the VR training strategy parameter set is subjected to security verification and optimization adjustments, including: Time series analysis is performed on the safety prediction results. If the probability of falling exceeds the first threshold for K consecutive time steps, or the instantaneous torque on any joint exceeds the second threshold, the VR training strategy parameter set is determined to have failed the safety check, where K≤N and is a natural number. If the VR training strategy parameter set is determined to have failed the security check, the time step information with the highest fall risk in the security prediction result is used as a negative reward signal and input into the strategy optimization generator to trigger it to perform gradient update on the VR training strategy parameter set based on the negative reward signal, thereby generating a new VR training strategy parameter set after optimization and adjustment. The new VR training strategy parameter set is then input into the pre-trained digital twin environment to perform an N-step simulation prediction of fall prevention safety, and the process is iterated until the VR training strategy parameter set passes the safety check or reaches the maximum number of iterations.

8. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, Updating the user behavior simulation model and the policy optimization generator using the real-time multimodal data includes: The user behavior simulation model is updated online through learning: New training samples are constructed based on newly acquired real-time multimodal data and the parameter set of the corresponding executed VR training strategy. The parameters of the user behavior simulation model are updated by minimizing a composite loss function. The composite loss function includes a prediction error term based on the new training samples and a penalty term for the deviation of the parameters of the user behavior simulation model from their historical convergence values. The strength of the penalty term is weighted by the diagonal elements of the historical Fisher information matrix corresponding to the parameters. The policy optimization generator is updated based on priority experience replay: Based on the changes in user state vectors before and after training according to the parameter set of the VR training strategy, an empirical tuple is constructed. The absolute value of the temporal difference error of the empirical tuple is used as its initial sampling priority and stored in the empirical playback buffer of the strategy optimization generator. Batch data is sampled from the empirical replay buffer based on a non-uniform distribution of sampling priority, and sampling bias is corrected using importance sampling weights. The gradient is then calculated and the network parameters of the policy optimization generator are updated accordingly. Furthermore, update the sampling priority of the sampled empirical tuples; Meta-learning adjustment of the user behavior simulation model in conjunction with the policy optimization generator: The overall strength coefficient of the penalty term is dynamically adjusted based on the KL divergence between the policy distributions output by the policy optimization generator before and after the update.

9. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 8, characterized in that, The policy optimization generator is implemented using an actor-commentator network architecture based on an attention mechanism, and is trained and updated using an asynchronous advantage actor-commentator algorithm based on priority experience replay. In the gradient update process, a KL divergence constraint term for the policy parameter distribution is introduced to control the magnitude of the policy update.

10. The method for optimizing fall prevention VR training strategies using digital twin-driven two-layer reinforcement learning as described in claim 1, characterized in that, The method further includes: During the execution of the fall prevention VR virtual simulation training scenario, the VR training terminal calculates the skeletal posture data of the target user in the current frame in real time; The skeletal posture data is matched and compared with a preset safe posture boundary template; If the skeletal posture data is detected to exceed the preset safe posture boundary, the rendering and force feedback of the fall prevention VR virtual simulation training scene are interrupted, an emergency protection prompt is triggered, and abnormal event data containing the skeletal posture data is synchronized to the management server. The management server receives the abnormal event data and associates it with the corresponding VR training strategy parameter set to form a negative reward sample for reinforcement learning. This sample is used to update the user behavior simulation model and / or the strategy optimization generator to reduce the probability of generating VR training strategy parameter sets that lead to similar abnormal postures in the future.

Images