A virtual reality rehabilitation scene automatic construction method and system based on semantic understanding

Abstract

This invention discloses a method and system for automatically constructing virtual reality rehabilitation scenes based on semantic understanding. The method includes: receiving multimodal data containing rehabilitation training needs and generating a rehabilitation intention vector; based on the rehabilitation intention vector, constructing an initial semantic graph of the rehabilitation scene using a graph attention network, and converting the semantic graph into a three-dimensional scene layout heatmap using a conditional generative adversarial network; based on the three-dimensional scene layout heatmap, learning a personalized preference reward function from the patient's historical rehabilitation data through inverse reinforcement learning to generate a personalized difficulty adaptation map; applying rehabilitation dynamic constraints to the personalized difficulty adaptation map to generate a physically plausible scene physics model; and inputting the scene physics model into a neural rendering pipeline to render and output an immersive and interactive virtual rehabilitation training scene in real time. Using this invention, the cost of scene construction can be reduced, and the real-time matching accuracy between rehabilitation training and patient capabilities can be improved, along with the immersiveness of the training.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality technology, and in particular to a method and system for automatically constructing virtual reality rehabilitation scenes based on semantic understanding. Background Technology

[0002] Virtual reality (VR) rehabilitation training provides patients with personalized and quantifiable treatment experiences through immersive environments, demonstrating significant potential in the fields of motor dysfunction and cognitive rehabilitation in recent years. However, the construction of existing VR rehabilitation scenarios heavily relies on the collaboration of rehabilitation experts and developers, resulting in long scenario generation cycles, high costs, and difficulty in rapidly and dynamically adjusting to the patient's real-time status. Traditional methods often use fixed scenario templates, lacking deep semantic understanding of multimodal patient input (such as voice commands and motion videos), leading to low matching between rehabilitation tasks and the patient's actual abilities. Furthermore, current systems rarely consider human kinematic constraints and individual preferences during scenario generation, easily resulting in physically unreasonable or unsafe object layouts, affecting training effectiveness and user immersion. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for automatically constructing virtual reality rehabilitation scenes based on semantic understanding, so as to overcome the shortcomings of the prior art, reduce the cost of scene construction, and improve the real-time matching accuracy of rehabilitation training and patient abilities, as well as the immersive training experience.

[0004] One embodiment of this application provides a method for automatically constructing virtual reality rehabilitation scenes based on semantic understanding, the method comprising: Receive multimodal data containing rehabilitation training needs, including voice commands, text descriptions, or demonstration videos, and map different modal data to a unified semantic space through a cross-modal semantic encoder to generate a rehabilitation intention vector; Based on the rehabilitation intention vector, an initial semantic graph of the rehabilitation scene is constructed using a graph attention network. The nodes of the initial semantic graph represent virtual objects or environmental elements, and the edges represent the interaction relationships between nodes. The semantic graph is then converted into a three-dimensional scene layout heatmap using a conditional generative adversarial network. Based on the aforementioned 3D scene layout heatmap, a personalized preference reward function is learned from the patient's historical rehabilitation data through inverse reinforcement learning, and Monte Carlo tree search is used to dynamically adjust the object complexity and interaction challenge in the scene to generate a personalized difficulty adaptation map. Rehabilitation dynamics constraints are applied to the personalized difficulty adaptation map. The spatial position and relative distance of objects in the scene are verified according to human kinematic parameters and safety boundaries. The pose and motion trajectory of the objects are adjusted through iterative optimization to generate a scene physical model that conforms to physical rationality. The physical model of the scene is input into the neural rendering pipeline, and combined with a preset material feature library and dynamic lighting model, a virtual rehabilitation training scene with immersive and interactive effects is rendered and output in real time.

[0005] Another embodiment of this application provides an automatic construction system for virtual reality rehabilitation scenes based on semantic understanding, the system comprising: The receiving module is used to receive multimodal data containing rehabilitation training needs, including voice commands, text descriptions or demonstration videos, and to map different modal data to a unified semantic space through a cross-modal semantic encoder to generate a rehabilitation intention vector. The construction module is used to construct an initial semantic graph of the rehabilitation scene based on the rehabilitation intention vector using a graph attention network. The nodes of the initial semantic graph represent virtual objects or environmental elements, the edges represent the interaction relationships between nodes, and the semantic graph is converted into a three-dimensional scene layout heatmap using a conditional generative adversarial network. The search module is used to learn a personalized preference reward function from the patient's historical rehabilitation data through inverse reinforcement learning based on the three-dimensional scene layout heatmap, and to dynamically adjust the complexity of objects and the interactive challenge in the scene using Monte Carlo tree search to generate a personalized difficulty adaptation map. The verification module is used to apply rehabilitation dynamic constraints to the personalized difficulty adaptation map, verify the spatial position and relative distance of objects in the scene according to human kinematic parameters and safety boundaries, and adjust the pose and motion trajectory of objects through iterative optimization to generate a scene physical model that conforms to physical rationality. The output module is used to input the scene physical model into the neural rendering pipeline, and combine it with a preset material feature library and dynamic lighting model to render and output an immersive and interactive virtual rehabilitation training scene in real time.

[0006] Another embodiment of this application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding claims when running.

[0007] Another embodiment of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any of the preceding claims.

[0008] Compared with existing technologies, the present invention provides an automatic construction method for virtual reality rehabilitation scenarios based on semantic understanding, which can reduce the cost of scenario construction and improve the real-time matching accuracy of rehabilitation training and patient capabilities, as well as the immersive training experience. Attached Figure Description

[0009] Figure 1A hardware structure block diagram of a computer terminal for an automatic construction method of virtual reality rehabilitation scene based on semantic understanding, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating an automatic construction method for virtual reality rehabilitation scenes based on semantic understanding, provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an automatic virtual reality rehabilitation scene construction system based on semantic understanding, provided in an embodiment of the present invention. Detailed Implementation

[0010] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0011] This invention first provides a method for automatically constructing virtual reality rehabilitation scenes based on semantic understanding. This method can be applied to electronic devices, such as computer terminals, specifically ordinary computers.

[0012] The following detailed explanation uses a computer terminal as an example. Figure 1 This is a hardware structure block diagram of a computer terminal for an automatic construction method of virtual reality rehabilitation scenes based on semantic understanding, provided in an embodiment of the present invention. (See diagram below.) Figure 1 As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0013] See Figure 2 The present invention provides a method for automatically constructing virtual reality rehabilitation scenes based on semantic understanding, which may include the following steps: S201, Receive multimodal data containing rehabilitation training needs, the multimodal data including voice commands, text descriptions or demonstration videos, and map different modal data to a unified semantic space through a cross-modal semantic encoder to generate a rehabilitation intention vector; Specifically, the system can receive user voice commands through a voice acquisition module, convert the voice signal into a text sequence using a voice recognition algorithm, and simultaneously receive text descriptions through a text input interface to generate original text data of the same modality. The core of this step is to establish a dual-path text data acquisition channel to capture rehabilitation training needs in both speech and manual text formats. This transforms unstructured speech signals and free text descriptions into standardized, homomodal raw text data, eliminating data heterogeneity caused by differences in input formats. This provides a regular text data source for subsequent cross-modal semantic extraction. The specific implementation method is as follows: The voice acquisition module is a dedicated hardware adapter unit for picking up user rehabilitation voice commands. It possesses basic capabilities for environmental noise reduction and signal filtering, adapting to the typical environment of a rehabilitation training room and effectively filtering out interference signals such as environmental noise and equipment operating noise. The module's voice sampling frequency is set to 16000Hz, which fully covers the frequency range of human speech, preventing the loss of detailed voice information in rehabilitation commands. The sampling precision is set to 16bit to ensure the fineness of the acquired voice signal. It also has a built-in noise reduction threshold parameter set to 35dB. When the environmental noise exceeds this value, dynamic noise reduction processing is automatically activated to preserve the effective voice command signal to the greatest extent. Users can express their rehabilitation training needs through real-time voice interaction. In the example, the user inputs the voice command "Perform seated upper limb abduction rehabilitation training, with slow and even movements, and the training range controlled within the comfortable range of the shoulder joint." The voice acquisition module continuously picks up this voice signal and performs noise reduction and filtering preprocessing, outputting a clean, original analog voice signal.

[0014] Speech recognition algorithms are the core algorithms for converting speech signals into text sequences. This particular algorithm employs an end-to-end deep speech recognition approach, overcoming the errors of traditional segmented recognition and directly converting continuous analog speech signals into structured text sequences. The algorithm incorporates a semantic recognition confidence parameter with a confidence threshold set at 0.8. It performs frame-by-frame segmentation, feature extraction, and semantic decoding on the preprocessed speech signal, parsing the text information frame by frame and ultimately concatenating them into a complete text sequence. The recognition result is considered valid only when the overall confidence score is higher than 0.8; otherwise, it is considered a failure, and the user is required to re-enter the command. In this example, the algorithm is used to parse the aforementioned upper limb rehabilitation training speech commands, accurately converting them into a complete text sequence while fully preserving core requirements such as training posture, movement type, movement speed, and training range.

[0015] The text input interface is a standardized interactive channel for users to actively input their text-based rehabilitation needs. It supports two input modes: manual input and preset text selection, adapting to the operating habits of different users, especially rehabilitation patients with limited limb movement or who cannot interact via voice. The interface has a text compliance verification function, which can automatically filter invalid characters, garbled text, and meaningless sentences, retaining only valid text content containing rehabilitation training areas, training movements, training requirements, and training intensity. The character length of the input text supports a range of 0 to 200 characters, which can meet the text descriptions of various routine rehabilitation training needs. In the example, the user manually inputs the text description "seated shoulder joint abduction rehabilitation training, avoiding overstretching of the joint, suitable for rehabilitation exercises for mild upper limb dysfunction" through the text input interface. The interface automatically completes invalid character filtering and content verification, outputting neat text data.

[0016] After completing the speech-to-text sequence and manual text data collection, the system performs unified processing on the two types of text data in the same modality, unifying the text encoding format, character specifications, and semantic storage format, eliminating the format differences between speech-to-text and manually entered text, integrating and summarizing the two types of data, and finally generating standardized original text data in the same modality. This fully preserves all the user's rehabilitation training text-related needs and provides unified text data support for subsequent cross-modal semantic fusion.

[0017] The video acquisition module receives demonstration videos, and the pose estimation algorithm is used to extract key frames and human skeleton sequences from the videos. The skeleton sequences are then encoded into motion feature vectors to generate motion modal feature data. The core of this step is to analyze the visualized rehabilitation demonstration videos, filter effective movement frames from continuous video frames, accurately capture the joint postures and movement trajectories of the human body during rehabilitation exercises, and transform the visual video data into calculable and coded structured movement feature data. This achieves the digital conversion of video modal data and forms a multimodal data complement with text data. The specific implementation method is as follows: The video acquisition module is used to capture rehabilitation demonstration videos uploaded by users or recorded on-site. The module supports parsing and reading common video formats, with a fixed frame rate of 30fps (30 frames per second). This accurately captures subtle changes in slow-moving rehabilitation movements, avoiding posture analysis errors caused by lost frames. The video resolution is uniformly 1080P, ensuring clarity of images of limbs and joints, providing a high-definition foundation for subsequent posture extraction. The module has video preprocessing capabilities, automatically cropping invalid footage, removing blank segments and irrelevant images at the beginning and end of the video, retaining only complete rehabilitation demonstration video clips. The example video received is a seated upper limb abduction rehabilitation demonstration video, 5 seconds long, completely recording the entire execution process of the standard rehabilitation movement.

[0018] Pose estimation algorithms are the core technology for extracting human motion information. This algorithm employs a high-precision two-dimensional human pose estimation system specifically adapted to the slow, standardized movement characteristics of rehabilitation patients, accurately locating the spatial positions of core rehabilitation joints. The algorithm incorporates keyframe extraction rules, using the amount of motion change between frames as the criterion. The inter-frame pixel change threshold is set to 0.05. When the pixel change of human joint points between adjacent video frames exceeds this threshold, it is considered a valid motion change frame and marked as a keyframe. A fixed keyframe extraction interval of 5 frames is maintained to avoid data redundancy caused by an excessive number of keyframes and to prevent the loss of motion details due to excessively large frame intervals. This rule filters keyframes from continuous video frames that can completely represent the start, process, peak, and reset state of rehabilitation movements, comprehensively covering the entire process of rehabilitation movement posture features.

[0019] After selecting the keyframes, the pose estimation algorithm analyzes each keyframe frame to extract a standard human skeleton sequence. This skeleton sequence contains 17 key points of core human joints, covering key joints for rehabilitation training such as the shoulder, elbow, wrist, hip, knee, and ankle joints. It also includes key points of the torso, comprehensively representing the rehabilitation movement postures of the upper limbs, lower limbs, and torso. The algorithm records the two-dimensional coordinate information of each joint point in each keyframe in real time, forming a continuous temporal sequence of the human skeleton. Simultaneously, it normalizes all joint coordinate data, mapping the coordinate values ​​uniformly to the range of 0 to 1, eliminating numerical differences caused by video frame size and shooting distance, and ensuring data consistency.

[0020] After extracting and normalizing the skeletal sequence, the system uses a temporal coding network to encode the structured human skeletal sequence, transforming temporal and spatial joint motion data into fixed-dimensional motion feature vectors. The generated motion feature vectors are 256-dimensional, with each set of values ​​corresponding to core motion features such as joint range of motion, speed, joint angle, and timing, thus fully quantifying the movement patterns and posture standards of the rehabilitation demonstration movements. Finally, the encoded motion feature vectors are standardized and stored to generate standardized motion modal feature data, achieving the digital and characteristic expression of the video demonstration movements.

[0021] The original text data and action modality feature data are input into the cross-modal semantic encoder, which consists of a pre-trained language model and a graph convolutional network, to extract text semantic embeddings and action semantic embeddings respectively, generating heterogeneous semantic embedding vectors. The core of this step is to rely on a dual-structure cross-modal semantic encoder to perform deep semantic mining on text modal data and action modal data respectively, stripping away surface data information, extracting deep semantic features, and generating two types of heterogeneous semantic embedding vectors with unified dimensions and independent semantics. This achieves the initial deep semantic extraction of multimodal data, laying the feature foundation for subsequent cross-modal semantic fusion. The specific implementation method is as follows: The cross-modal semantic encoder is a dedicated encoding unit adapted for both text and action modality semantic extraction. It consists of two core modules connected in parallel: a pre-trained language model and a graph convolutional network. These two modules independently complete the semantic extraction for their respective modalities without interference, and their output feature dimensions are uniformly matched, ensuring the feasibility of subsequent fusion. The encoder incorporates a data splitting mechanism that automatically identifies the modality type of the input data, automatically allocating raw text data to the pre-trained language model and action modality feature data to the graph convolutional network, achieving accurate modality matching encoding.

[0022] The pre-trained language model is specifically designed for text semantic extraction. Pre-trained on a massive corpus of rehabilitation texts, it incorporates dedicated semantic parsing logic adapted to rehabilitation training scenarios, enabling accurate identification of professional semantic information within rehabilitation texts. The model sets the maximum input text length to 128 characters, covering the vast majority of text descriptions required for rehabilitation training. Through a three-layer core structure of word embedding, semantic encoding, and contextual analysis, the model extracts core semantic information from the original text data, including key elements such as rehabilitation training areas, training movement types, training intensity, movement constraints, and applicable disease types. The model ultimately outputs a 512-dimensional text semantic embedding. This embedding vector numerically quantifies all rehabilitation semantic information in the text, fully preserving the training intent and constraints expressed in the text.

[0023] Graph convolutional networks (GCNNs) are core networks for semantic extraction of skeletal movement features. Adapted to the graph structure data features of human skeletal sequences, they can mine spatial and temporal semantics from movement data. The network has six layers and uses 3x3 convolutional kernels for feature convolution calculations, accurately capturing the spatial relationships between joints, movement linkage patterns, and temporal motion features. Taking 256-dimensional movement modal feature data as input, the network quantifies and extracts deep semantics such as standard posture, range of motion, movement specifications, and difficulty level of demonstrated rehabilitation movements through multi-layer convolutional operations, feature aggregation, and semantic mapping. The final output is a 512-dimensional movement semantic embedding, realizing the transformation of visualized movement features into abstract semantic features.

[0024] Text semantic embedding and action semantic embedding share the same dimensions but differ in semantic source and representation dimensions, thus belonging to heterogeneous semantic features. These two types of vectors together constitute a heterogeneous semantic embedding vector group, or heterogeneous semantic embedding vector. Text semantic embedding focuses on representing the user's subjective rehabilitation needs, training constraints, and goals, while action semantic embedding focuses on representing objective standard training movements, posture norms, and motion characteristics. The two types of vectors complement each other, fully covering the user's core rehabilitation training intentions, and providing complete heterogeneous feature data for subsequent cross-modal semantic alignment and fusion.

[0025] By using a cross-modal attention fusion mechanism, heterogeneous semantic embedding vectors are projected onto a unified semantic space and the semantic representations of different modalities are aligned, ultimately generating a rehabilitation intention vector that represents the intention of rehabilitation training.

[0026] The core of this step is to eliminate the semantic heterogeneity differences between text and action modalities through an attention fusion mechanism. This maps two independent heterogeneous semantic vectors to the same standardized semantic space, completing the alignment, complementarity, and fusion of semantic information. Ultimately, it generates a unique, accurate, and unified rehabilitation intention vector, fully quantifying the user's personalized rehabilitation training needs. The specific implementation method is as follows: The cross-modal attention fusion mechanism is the core algorithm for achieving multimodal semantic fusion. Its core function is to calculate the semantic association weights between text semantic embeddings and action semantic embeddings, accurately match the corresponding rehabilitation semantic information in the two modalities, filter redundant modal information, and strengthen core intent features. The mechanism sets the attention calculation dimension to 512, consistent with the dimension of the heterogeneous semantic embedding vectors, and configures a semantic attention decay coefficient of 0.2. This parameter is used to weaken the weight proportion of irrelevant semantic features, strengthen the core rehabilitation semantic features with high matching degree, and avoid invalid semantic information interfering with the fusion result. The mechanism compares the feature values ​​of text semantic embeddings and action semantic embeddings dimension by dimension, calculates the association matching degree of each set of semantic features, and generates a corresponding attention weight matrix. The weight values ​​range from 0 to 1; higher values ​​indicate a higher matching degree between the two modalities in that dimension, belonging to the core rehabilitation intent information.

[0027] The unified semantic space is a standardized semantic representation space specifically built for the fusion of multimodal rehabilitation data. It has a fixed 512-dimensional dimension, and all semantic features of text and action modalities must be projected onto this space for standardized mapping. This semantic space incorporates a rehabilitation-specific semantic scale, with each dimension corresponding to fixed rehabilitation semantic attributes, including core rehabilitation features such as training area, action type, training difficulty, range of motion, safety constraints, and training duration. All modal semantic projections follow this scale for feature regularization, completely eliminating the semantic representation differences between text and action modalities and achieving spatial unification of multimodal data.

[0028] After completing the semantic space projection, the system performs cross-modal semantic alignment operation. The built-in semantic alignment correction threshold is 0.01, which corrects the semantic offset dimension that exists after the projection of two types of heterogeneous vectors. When the numerical deviation of the same dimension semantic features between modalities exceeds 0.01, the deviation correction is automatically completed based on attention weights to ensure that the training requirements of the text description and the training specifications of the demonstration action are fully matched and semantically unified, avoiding the problem of conflict between the text requirements and the demonstration intention, and achieving accurate alignment of semantic representations of different modalities.

[0029] After semantic projection and alignment, the mechanism uses an attention weight matrix to weight and fuse the two types of heterogeneous semantic embedding vectors. High-weight core semantic features are superimposed, while low-weight redundant and ineffective features are removed. This deeply integrates the subjective semantic needs of the text with the objective standard semantics of the actions, ultimately generating a 512-dimensional rehabilitation intention vector. Each dimension of this vector precisely corresponds to a core rehabilitation training intention feature, comprehensively, quantitatively, and uniformly representing all of the user's rehabilitation training needs. It is free of semantic omissions and modal biases, providing accurate intention data support for subsequent construction of a semantic graph of the rehabilitation scenario.

[0030] S202, Based on the rehabilitation intention vector, an initial semantic graph of the rehabilitation scene is constructed using a graph attention network. The nodes of the initial semantic graph represent virtual objects or environmental elements, and the edges represent the interaction relationships between nodes. The semantic graph is then converted into a three-dimensional scene layout heatmap using a conditional generative adversarial network. Specifically, it can analyze the key entities and action requirements in the rehabilitation intention vector, retrieve matching node candidate sets from the preset virtual object library and environmental element library, and generate a node candidate set. The core of this step is to semantically decompose and analyze the requirements of the high-dimensional abstract rehabilitation intention vector, accurately extract the entity objects and action requirements corresponding to rehabilitation training, complete element matching and screening based on a standardized resource library, and aggregate them to form a candidate set of nodes that can be used for semantic graph construction, providing basic units for subsequent scene relationship modeling. The specific implementation method is as follows: The rehabilitation intention vector is a high-dimensional unified semantic representation vector generated through cross-modal semantic fusion. It has a fixed dimension of 512, with each set of dimension values ​​corresponding to a semantic feature of rehabilitation training. This encompasses core semantic information such as the training area, type of training movement, difficulty level, training environment type, and interaction method. All dimension values ​​are uniformly constrained between 0 and 1; values ​​closer to 1 indicate higher semantic feature expression strength. This step uses a high-dimensional semantic feature parsing algorithm to decompose the vector. The algorithm incorporates fixed semantic feature mapping rules and sets a feature response threshold of 0.6. When the value of a certain dimension feature in the rehabilitation intention vector exceeds this threshold, it is considered a valid core semantic feature. The system automatically extracts the semantic label corresponding to that dimension, and finally summarizes all valid labels to obtain complete key entities and movement requirement information. Key entities refer to physical units such as virtual objects and scene environment components that require interaction during rehabilitation training, while movement requirements refer to the training movement specifications such as the limb movement method, range of motion, and frequency that the patient needs to complete.

[0031] The pre-built virtual object library and environmental element library are standardized rehabilitation scene resource databases. Both libraries are compatible with all types of limb rehabilitation training scenarios and have a complete semantic tagging system and attribute parameter system. The virtual object library specifically stores interactive virtual objects required for various rehabilitation training, covering object units for different rehabilitation types such as upper limb fine motor training, lower limb gait training, trunk balance training, and joint flexion and extension training. These include virtual grip strength training balls, limb flexion and extension training supports, balance training pedals, fingertip fine motor training components, and weight-bearing training modules. Each type of virtual object is bound to a unique semantic tag, corresponding rehabilitation area, applicable movement type, and interactive attribute parameters, which can accurately match different rehabilitation needs. The environmental element library specifically stores the overall environment construction units of virtual rehabilitation scenarios, including indoor standard rehabilitation rooms, barrier-free flat training areas, stepped gait training areas, static balance training spaces, and dynamic obstacle training scenarios. Each type of environmental element is labeled with scene semantic tags, spatial scale parameters, and corresponding training scenario types to match the environmental layout requirements of different rehabilitation training.

[0032] After extracting key entities and action requirements, the system initiates a semantic similarity matching algorithm to retrieve and match resources. This algorithm is the core algorithm for achieving precise matching between requirements and resources. The semantic similarity calculation result ranges from 0 to 1, with a matching judgment threshold of 0.75. If the similarity value is greater than this threshold, the resource is considered a valid match and can be included in the candidate range. The algorithm aligns the parsed key entity tags and action requirement tags with the bound semantic tags of all resources in the virtual object library and environmental element library one by one, calculates the bidirectional semantic association similarity, and filters out all virtual objects and environmental elements that meet the threshold conditions. Taking the requirements for fine motor rehabilitation training of the hand as an example, the parsed key entity requirement is a small hand interaction prop, and the action requirement is a low-amplitude fine motor action such as grasping, pressing, and lifting. Through similarity calculation, the algorithm matches virtual objects such as virtual silicone grip balls, fingertip pressing training blocks, and small handheld weight plates, and simultaneously matches environmental elements such as a flat desktop training environment and a static indoor wind-free training space.

[0033] After completing the screening of all matching resources, the system automatically removes duplicate object and environmental element units, integrates all valid matching resources, and forms a candidate set of nodes that is non-repeating, highly adaptable, and in line with the rehabilitation intention. Each object and environmental element in the set is used as an independent graph node, providing complete basic node materials for subsequent relationship modeling of graph attention networks.

[0034] The candidate set of nodes is input into the graph attention network. The semantic relevance weights between nodes are calculated through the attention mechanism, and the interaction relationship edges between nodes are constructed based on the weights to generate a weighted initial semantic graph. The core of this step is to leverage the adaptive semantic association modeling capability of graph attention networks to quantify the degree of adaptation of rehabilitation scene interactions between candidate nodes, define node relationships through weight values, build an initial scene semantic graph with semantic weight attributes, and realize the digital modeling of the relationships between rehabilitation scene elements. The specific implementation method is as follows: Graph Attention Networks (GANs) are deep learning network models specifically designed for feature learning and node relationship mining in graph-structured data. Compared to traditional graph convolutional networks, this network possesses adaptive attention weight allocation capabilities, accurately capturing differentiated semantic relationships between nodes in different scenarios and adapting to the complex structural features of multi-element, multi-relationship rehabilitation scenarios. This solution employs a multi-head graph attention mechanism, setting an 8-head attention splitting parameter. This parameter allows for multi-channel splitting and learning of the semantic feature dimensions of nodes, avoiding the omission of related features caused by single-channel feature learning and significantly improving the accuracy of node correlation calculation. The network input is the node candidate set generated in step one. All virtual object and environmental element nodes within the set are initialized to generate their own unique node semantic feature vectors. These feature vectors are uniformly set to 256 dimensions, integrating multi-dimensional features such as node functional attributes, adaptation to rehabilitation actions, spatial usage characteristics, and training adaptation priorities, providing complete feature data support for weight calculation.

[0035] The core function of the attention mechanism is to quantify the rationality and relevance of the interaction between any two nodes in a rehabilitation training scenario. The specific calculation process is as follows: the network first performs a global traversal and pairing of the semantic feature vectors of all nodes, generating a set of pairwise combinations of all nodes. Then, the attention scoring function performs cross-operation on the feature vectors of each pair, outputting an initial node association score. Subsequently, a softmax normalization function is used to globally normalize all initial scores, mapping all association scores to a uniform value range of 0 to 1, ultimately obtaining a standardized semantic relevance weight. This weight value directly defines the scene association strength between two nodes. The closer the value is to 1, the more likely the two nodes must be used together in the target rehabilitation scenario, indicating extremely strong interaction and high scene adaptability; the closer the value is to 0, the less likely the two nodes have any scene interaction requirement, indicating extremely low adaptability.

[0036] Taking the node combination in the fine motor rehabilitation scene of the hand as an example, the semantic relevance weight calculation result of the training desktop node and the virtual grip ball node is 0.93, proving that the two are highly bound, and the grip ball must rely on the desktop environment to complete static fine motor training; the relevance weight of the indoor static training environment node and the training desktop node is 0.96, which is the core basic relationship of the scene; while the relevance weight of the balance pedal node and the desktop training prop node is only 0.22, and the two have no effective training relationship. After completing the weight calculation of all node pairings, the system uses each independent scene node as the topological vertex and the normalized semantic relevance weight as the attribute parameter of the edge to build a dedicated interaction relationship edge for each group of nodes with effective relationship. All nodes are connected to the weighted interaction relationship edge to form a complete topological graph structure. This structure is the weighted initial semantic graph, which fully restores the potential interaction relationship of all elements in the rehabilitation scene and retains the scene attributes and relationship features of all candidate nodes.

[0037] The initial semantic graph is topologically optimized to remove redundant nodes and low-weight edges, and necessary spatial constraints are added to generate a simplified semantic graph of the rehabilitation scene. The core of this step is to simplify the structure and complete the logic of the initial semantic graph, eliminate invalid and redundant units, weaken invalid relationships, supplement the hard constraints of scene spatial layout, optimize the structural rationality and logical rigor of the semantic graph, and generate a standardized semantic graph adapted to the generation of 3D scenes. The specific implementation method is as follows: The initial semantic graph contains all candidate nodes selected in step one and all associated edges generated in step two. It contains some redundant units with low matching degree to the core rehabilitation training needs and no actual training effect. Simultaneously, it contains a large number of invalid relational edges with low correlation, which increases computational redundancy in subsequent 3D scene generation. Therefore, a topology optimization algorithm is needed to simplify the structure. The topology optimization algorithm has a built-in two-dimensional selection threshold: a node semantic contribution threshold and a relational edge weight threshold. The node semantic contribution threshold is set to 0.4, and the low-weight relational edge selection threshold is set to 0.3. The node semantic contribution is used to measure the supporting role of a single node in core rehabilitation training, with a value ranging from 0 to 1. The higher the value, the higher the training value of the node. Nodes with a value below 0.4 are considered redundant nodes with no scene construction value. Relational edge weights below 0.3 indicate extremely weak interaction between two nodes, lacking actual training interaction logic, and are thus judged as low-weight invalid edges.

[0038] Topology optimization first performs redundant unit removal. The algorithm traverses all nodes in the initial semantic graph and, based on the core training requirements of the original rehabilitation intention vector, calculates the semantic contribution of each node. All redundant nodes with a contribution below 0.4 are removed. Simultaneously, all interaction edges are traversed, and all low-weight edges with a weight value below 0.3 are deleted. For example, for the core requirement of upper limb fine grasping training, nodes such as the lower limb balance training pedal and the step training module remaining in the initial semantic graph have a semantic contribution of only 0.15 to 0.3, which does not meet the threshold standard and will be directly removed. Correspondingly, the edges between these redundant nodes and the core training nodes are also simultaneously determined as invalid edges and cleaned up, significantly simplifying the overall structure of the semantic graph. All remaining nodes and edges align with the core rehabilitation training requirements.

[0039] After redundancy simplification, the algorithm continues to perform spatial constraint relationship supplementation operations to improve the spatial layout logic of the semantic graph and compensate for the deficiency of the initial semantic graph, which only has semantic associations and lacks spatial rules. The supplemented spatial constraint relationships include three core rules: node spatial distance constraints, hierarchical subordination constraints, and interaction position constraints. All constraint rules are adapted to the spatial movement logic and safety specifications of human rehabilitation training. Spatial distance constraints define the maximum and minimum spatial interval values ​​between different interactive nodes to avoid overlapping or excessive distance between virtual objects, which would prevent interactive training from being completed. Hierarchical subordination constraints clarify that environmental nodes are the underlying carrier units and object interaction nodes are the upper-level training units, establishing the basic hierarchical structure of the scene. Interaction position constraints limit the placement area of ​​specific training objects to ensure that the object arrangement conforms to the movement habits of human rehabilitation movements.

[0040] By binding all spatial constraints one by one to the simplified node and associated edge structure, the semantic graph not only has semantic association attributes, but also standardized spatial layout constraint logic. Finally, the entire topology optimization process is completed, generating a simplified rehabilitation scene semantic graph with a concise structure, accurate semantics, and compliant spatial logic. This provides accurate and efficient structured input data for the subsequent generation of 3D scene heatmaps.

[0041] A simplified semantic graph of the rehabilitation scene is input into a conditional generative adversarial network. The semantic graph is used as a conditional generator to generate a probability density field of object distribution in three-dimensional space, and finally outputs a three-dimensional scene layout heatmap.

[0042] The core of this step is to leverage the conditional generation capabilities of conditional generative adversarial networks to transform two-dimensional structured semantic graph information into a three-dimensional spatial object layout probability distribution. This is then used to generate heatmaps through probability visualization, realizing the transformation of the rehabilitation scene from semantic logic to three-dimensional spatial layout. The specific implementation method is as follows: Conditional Generative Adversarial Networks (GANs) are deep generative models based on conditional constraints. Their core consists of two main functional modules: a generator and a discriminator. Unlike ordinary GANs, this model can generate data distributions that conform to specific logic based on external conditional constraints. In this solution, a simplified rehabilitation scene semantic graph is used as the sole global constraint, controlling the generation logic of the 3D scene layout throughout the process to ensure that the generated results align with the semantic needs of rehabilitation training and spatial constraints. All structured information carried by the simplified rehabilitation scene semantic graph, including node attributes, association weights, and spatial constraints, is fully input into the network's conditional input layer, serving as the core generation basis for the generator.

[0043] The generator is the core module for generating 3D spatial probability distributions. It integrates a 3D deconvolution feature decoding algorithm, which can parse semantic data from a 2D graph structure into 3D spatial data. After receiving all the constraint information from the semantic graph, the generator performs a global meshing of the entire preset rehabilitation 3D space, dividing the overall space into uniformly scaled 3D spatial units. Each spatial unit has a scale of 0.1m × 0.1m × 0.1m, precisely matching the layout accuracy requirements of virtual rehabilitation objects. The algorithm then calculates the fit of each 3D spatial unit, combining the node fit types, association weights, and spatial constraint rules in the semantic graph to output the placement fit probability of each type of rehabilitation object for each spatial unit. The probability values ​​of all spatial units are integrated and summarized to form an object distribution probability density field covering the entire 3D rehabilitation space.

[0044] All values ​​in the probability density field are uniformly constrained to the range of 0 to 1. The magnitude of the probability value directly represents the rationality of placing target rehabilitation objects at the corresponding spatial location. The closer the value is to 1, the higher the layout adaptability, training practicality, and human interaction rationality of that spatial location, making it the optimal object placement area. The closer the value is to 0, the less suitable the location is for placing training objects, as it does not conform to the logic of rehabilitation training layout. Taking the three-dimensional space of a desktop fine rehabilitation training scenario as an example, the spatial unit probability value of the core training area of ​​the desktop can reach 0.90 to 1.00, which is the optimal layout area. The probability value of the edge area of ​​the desktop is 0.50 to 0.89, which is the suboptimal layout area. The probability values ​​of the empty and suspended areas outside the desktop are all below 0.40, making them unsuitable for placing fine training interactive objects.

[0045] The network's discriminator module synchronously performs global verification and optimization on the generated probability density field. It matches and verifies the probability values ​​of each spatial unit against the semantic graph constraints, corrects abnormal probability values ​​that violate spatial constraints and semantic relationships, and removes noisy data from the global probability distribution to ensure the overall rationality and accuracy of the probability density field. After verification and optimization, the system performs visual mapping on the 3D spatial probability density field, mapping the probability value gradient from 0 to 1 to differentiated thermal color levels. High-probability areas are presented with bright thermal effects, while low-probability areas are presented with dark background effects. Simultaneously, it fully binds the 3D coordinate information and probability attributes of all spatial units, ultimately generating and outputting a complete 3D scene layout heatmap. This heatmap intuitively quantifies the adaptability of the object layout in the entire 3D rehabilitation space, providing core spatial data support for subsequent personalized scene difficulty adaptation and physical model construction.

[0046] S203, based on the three-dimensional scene layout heatmap, learn a personalized preference reward function from the patient's historical rehabilitation data through inverse reinforcement learning, and use Monte Carlo tree search to dynamically adjust the complexity of objects and the interactive challenge in the scene to generate a personalized difficulty adaptation map. Specifically, historical rehabilitation data can be extracted from the patient's electronic rehabilitation records, including the trajectory of completed movements, error rate, and subjective difficulty score, to generate a dataset of the patient's historical behavior. The core of this step is to rely on standardized electronic rehabilitation records to mine real behavioral and experiential data from patients' past rehabilitation training, select effective core data dimensions and complete standardized integration, and construct a dedicated patient historical behavior dataset that can be used for algorithm learning. This provides real data support for the subsequent derivation of personalized reward functions. The specific implementation method is as follows: Electronic rehabilitation records are standardized data carriers that store patients' rehabilitation training information throughout the entire cycle. They record the entire process of each virtual reality rehabilitation training session, including training time, training items, movement completion status, system detection data, and patient subjective feedback. All data is anonymized and stored in compliance with regulations. The data sampling frequency is fixed at 30 frames per second, which can completely recreate the dynamic process of the patient's rehabilitation training, ensuring data continuity and integrity. The system can accurately retrieve record data within the corresponding cycle based on the current rehabilitation training scenario and training objectives, eliminating invalid test data, interrupted training data, and other abnormal data, retaining valid training sample data, and providing a reliable data source for data extraction.

[0047] The trajectory of a completed movement is core objective data characterizing a patient's ability to perform rehabilitation movements. Specifically, it refers to the spatial motion path data of key joints and limbs when a patient completes designated training movements in a virtual rehabilitation scenario. The trajectory data includes three core parameters: spatial coordinates, movement velocity, and movement acceleration. The spatial coordinate parameter is based on a three-dimensional spatial coordinate system, with units in meters (m) and an accuracy of up to 0.01m, used to record the spatial position of the limb at every moment. The movement velocity parameter is in meters per second (m / s) with an accuracy of 0.01m / s, used to characterize the speed of limb movement. The movement acceleration parameter is in meters per second squared (m / s²). 2 ), accuracy 0.01m / s 2 This is used to characterize the range of change in limb movement. In the example, for upper limb lifting rehabilitation training, the system extracts the complete movement trajectory of the shoulder, elbow, and wrist joints of the upper limb from nearly 20 effective training sessions, records the coordinates, velocity, and acceleration values ​​corresponding to each frame, and fully reconstructs the patient's movement habits and limb activity capabilities.

[0048] Error rate is a core objective indicator for quantifying the accuracy of a patient's movements. It refers to the proportion of movements that do not meet the standard procedure in a single rehabilitation training session out of the total number of prescribed movements. The calculation formula is the number of errors divided by the total number of training sessions, with a value ranging from 0 to 1. The closer the value is to 1, the more errors there are and the lower the rehabilitation fit; the closer the value is to 0, the stronger the standard procedure. This indicator is automatically calculated by the system based on a preset standard movement trajectory threshold. The standard movement trajectory threshold is a compliant movement range preset by the system. If the patient's limb movement trajectory exceeds this range, it is judged as a movement error. In the example, a single grasping rehabilitation training session for a patient included 20 grasping movements, of which 3 movements deviated from the standard range, resulting in an error rate of 0.15, accurately reflecting the patient's movement accuracy level in this training session.

[0049] Subjective difficulty rating is subjective feedback data reflecting the patient's perceived fit to the training. It is submitted voluntarily by the patient after completing a single rehabilitation training session. The rating represents the patient's subjective perception of the difficulty of the current training scenario. It uses a 100-point scale, ranging from 0 to 100, where 0 represents extremely low difficulty and no training pressure; 50 represents moderate difficulty and suitability to the patient's rehabilitation status; and 100 represents extremely high difficulty and inability to complete the training smoothly. This rating effectively compensates for differences in patient perception that objective data cannot reflect, achieving a combination of objective training data and subjective experience data. In the example, after completing a stepped obstacle avoidance rehabilitation training session, the patient submitted a subjective difficulty rating of 65, indicating that the patient considered the current scenario to be too difficult and experienced some training pressure.

[0050] After extracting the three core data types, the system standardizes, cleans, and integrates the scattered individual training data, unifying the sampling dimensions, numerical precision, and data format of all data. Invalid samples with missing data or abnormal values ​​are removed. The system then binds and associates the motion trajectory data, error rate, and subjective difficulty score for each effective training session, arranging them in chronological order to generate a structured patient historical behavior dataset. This dataset comprehensively preserves the behavioral and somatosensory characteristics of patients during long-term rehabilitation training, serving as the core data foundation for subsequent inverse reinforcement learning algorithms to learn patients' personalized rehabilitation preferences.

[0051] An initial state space is constructed based on a 3D scene layout heatmap. An inverse reinforcement learning algorithm is used to inversely deduce the implicit personalized preference reward function parameters from the patient's historical behavior dataset, thereby generating a preference reward function model. The core of this step is to use the previously generated 3D scene layout heatmap as a spatial benchmark to build an algorithmic state space adapted to the virtual reality rehabilitation scene. It then leverages inverse reinforcement learning algorithms to uncover implicit preference rules behind patients' historical training behaviors, derives exclusive reward function parameters, and constructs a model that can quantify patients' personalized rehabilitation adaptation preferences. This provides a criterion for dynamically adjusting the difficulty of the subsequent scene. The specific implementation method is as follows: A 3D scene layout heatmap is the core foundation for constructing the initial state space. This heatmap quantifies the object distribution density, spatial openness, and interaction complexity of different areas within the virtual rehabilitation scene using pixel grayscale values. The grayscale value ranges from 0 to 255; a higher grayscale value indicates a denser arrangement of scene objects and higher interaction difficulty in the corresponding area, while a lower grayscale value indicates a more open space and lower training difficulty. Based on the grayscale distribution characteristics, spatial region division characteristics, and object arrangement characteristics of this heatmap, the system completes the construction of the initial state space. The state space is the set used in the algorithm to represent all possible layout states of the rehabilitation scene, including four core dimensions: scene object quantity state, obstacle distribution state, interaction difficulty state, and spatial layout complexity state. All state parameters correspond to the visualization features of the heatmap, realizing the digital quantification of the virtual scene state.

[0052] The parameters of the initial state space have clearly defined value ranges and physical meanings. The number of scene objects ranges from 5 to 30, representing the total number of interactive rehabilitation objects and environmental objects within a single scene. The obstacle distribution ranges from 0 to 1, with higher values ​​indicating denser obstacle arrangement and smaller usable space. The difficulty of interactive actions ranges from 1 to 10, with higher values ​​indicating higher requirements for the range and precision of limb movements required for rehabilitation actions. The spatial layout complexity ranges from 1 to 8, with higher values ​​indicating more disordered scene object arrangement and more complex spatial hierarchy. The system combines the grayscale distribution of the 3D scene layout heatmap to assign basic parameters to the initial state space, forming a complete set of scene states covering low, medium, and high gradients of scene layout states.

[0053] Inverse reinforcement learning is a core algorithm for uncovering patients' personalized rehabilitation preferences. Unlike traditional reinforcement learning's positive reward setting, this algorithm does not require manually preset reward rules. Instead, it observes the patient's historical best training behavior trajectory and reverse-engineers reward rules that match the patient's behavioral choices. Its core principle is to fit the mapping relationship between the patient's training behavior and the scene difficulty and layout, quantifying the scene features the patient prefers and avoids. The algorithm's core input consists of all training samples from the patient's historical behavior dataset, including the patient's action completion and subjective feedback under different scene states. The core output consists of various parameters of the personalized preference reward function.

[0054] The preference reward function is a mathematical model that quantifies patients' rehabilitation adaptation preferences. It includes three core learnable parameters: scene complexity reward coefficient, movement difficulty adaptation reward coefficient, and training comfort reward coefficient. The scene complexity reward coefficient ranges from -1 to 1, with positive values ​​indicating that patients prefer high-complexity scenes and negative values ​​indicating that patients prefer low-complexity scenes. The movement difficulty adaptation reward coefficient also ranges from -1 to 1, with positive values ​​indicating that patients are adapted to high-difficulty movement training and negative values ​​indicating that patients are more adapted to low-difficulty basic movement training. The training comfort reward coefficient also ranges from -1 to 1, with positive values ​​indicating that patients feel comfortable in scene layouts and negative values ​​indicating that patients feel uncomfortable in scene layouts.

[0055] The algorithm iteratively fits the patient's historical behavioral data one by one. For each set of training data, combined with the state space parameters of the corresponding scene, it calculates the behavioral matching loss value under different parameter combinations. It continuously corrects the values ​​of the three types of reward coefficients through gradient iteration until the loss value converges to within a preset threshold of 0.001, thus completing the parameter fitting. In the example, after iterative fitting by the algorithm, the patient's scene complexity reward coefficient is 0.32, the action difficulty adaptation reward coefficient is 0.28, and the training comfort reward coefficient is 0.45, indicating that the patient adapts to moderately high scene complexity and action difficulty, and has a higher acceptance of relaxed and comfortable scene layouts. After parameter fitting, the fixed reward coefficients are combined with the reward function calculation formula to construct a personalized preference reward function model specific to the patient, enabling quantitative evaluation of the adaptation reward value for any rehabilitation scene layout.

[0056] By combining the preference reward function model with the Monte Carlo tree search algorithm, the tree is expanded with the current scene layout as the root node. The cumulative reward value under different combinations of object complexity and interaction challenge is evaluated, and the search tree evaluation result is generated. The core of this step is to integrate a personalized reward model with an intelligent search algorithm. Based on the initial scene layout, it expands the scene state in multiple dimensions. By simulating scene configurations with different difficulties and layouts, it quantifies the adaptation reward value of each configuration, completes the evaluation of the merits of all-dimensional scene solutions, and provides data support for selecting the optimal personalized scene configuration. The specific implementation method is as follows: Monte Carlo Tree Search (MCS) is the core search algorithm for intelligent evaluation and optimization of scenario solutions. It possesses the core characteristics of global search, random sampling, and iterative evaluation, enabling it to cover all reasonable combinations of object complexity and interaction challenges, thus avoiding local optima. The algorithm's execution flow includes four core stages: node initialization, tree expansion, random sampling, and reward backtracking. Numerical evaluation is performed entirely based on a preference reward function model, ensuring that the evaluation results fully align with the patient's personalized rehabilitation preferences.

[0057] The current scene layout is the root node of the algorithm search tree. The root node corresponds to the initial scene configuration generated by the previous 3D scene layout heatmap, which includes the initial number of objects, obstacle density, action interaction difficulty, and spatial layout parameters. It is the basic benchmark for all subsequent scene expansion schemes. The basic reward value of the root node can be directly calculated through the preference reward function model and used as a reference benchmark for evaluating subsequent nodes.

[0058] The tree expansion stage is the core process of multi-dimensional scene state derivation based on the root node. The algorithm uses object complexity and interaction challenge as the two core expansion dimensions, decomposing into multi-level sub-nodes. Object complexity is determined by three sub-parameters: the number of objects in the scene, the richness of object types, and the number of object spatial arrangement layers. The number of objects ranges from 5 to 30, the richness of object types ranges from 1 to 10, representing the number of different functional rehabilitation objects in the scene, and the number of object spatial arrangement layers ranges from 1 to 5, representing the three-dimensional arrangement layers of objects in the scene. Interaction challenge is determined by three sub-parameters: action accuracy requirement, limb range of motion, and dynamic interaction frequency. Action accuracy requirement ranges from 1 to 10, with higher values ​​requiring greater precision in patient limb control. Limb range of motion ranges from 0 to 180 degrees, representing the range of limb movement angles for rehabilitation actions. Dynamic interaction frequency ranges from 1 to 20, representing the number of interactive actions required per unit of training time.

[0059] The algorithm randomly combines and iterates the parameters in two dimensions to generate a large number of child nodes with different parameter combinations. Each child node corresponds to a brand-new scenario difficulty configuration scheme. All child nodes retain the same core theme of rehabilitation training as the root node, and only adjust the difficulty and layout parameters to achieve difficulty gradient expansion while ensuring that the scenario rehabilitation function remains unchanged.

[0060] In the random sampling and reward evaluation phase, the algorithm calculates the single-step immediate reward value for each extended sub-node's corresponding scenario configuration by calling the personalized preference reward function model. Then, it simulates the patient's complete training process in that scenario using Monte Carlo random sampling, extrapolating the multi-step reward values ​​throughout the training process. Finally, it sums all the immediate reward values ​​to obtain the cumulative reward value for the corresponding scenario configuration. The cumulative reward value ranges from -10 to 10. A higher value indicates a higher degree of matching between the scenario configuration and the patient's personalized rehabilitation preferences, limb abilities, and somatosensory adaptation. A lower value indicates that the scenario configuration is not suitable for the patient's rehabilitation state, and may have issues such as being too difficult or too easy, or having an unreasonable layout. In the example, the cumulative reward value for a low-complexity, low-challenge scenario is 2.15, the cumulative reward value for a medium-complexity, medium-challenge scenario is 7.68, and the cumulative reward value for a high-complexity, high-challenge scenario is 3.42, clearly demonstrating the differences in adaptability between different scenario configurations.

[0061] The reward backtracking process updates the cumulative reward values ​​of all child nodes to each level of the search tree, annotates the parameter combination and corresponding reward value of each node, and fully records the evaluation results of all extended scenario solutions. Finally, it forms a complete search tree evaluation result that includes the difficulty configuration of all gradient scenarios, the corresponding cumulative reward value, and the adaptation features, providing comprehensive evaluation data support for the subsequent selection of the optimal scenario.

[0062] Based on the search tree evaluation results, select the scene configuration with the highest reward value, adjust the number of object types, obstacle density, and motion sequence difficulty in the scene, and finally output a personalized difficulty adaptation map.

[0063] The core of this step is to select the optimal scenario configuration scheme that matches the patient's personalized rehabilitation status based on the full-dimensional evaluation results of Monte Carlo tree search, to make targeted fine adjustments to the core parameters of the scenario, to optimize the overall difficulty and interaction characteristics of the scenario, and finally to generate a standardized personalized difficulty adaptation map that matches the patient's individual abilities. The specific implementation method is as follows: The system performs a full-domain comparison and filtering of the cumulative reward values ​​of all nodes in the search tree evaluation results. Following the principle of selecting the highest reward value, it eliminates unsuitable scenario schemes with negative reward values ​​and inefficient training scenario schemes with low reward values, and locks the scenario configuration corresponding to the node with the highest cumulative reward value as the optimal target configuration. This optimal configuration is the best solution that takes into account the patient's limb rehabilitation ability, historical training habits, and subjective physical comfort. It can ensure the training effect of rehabilitation training while avoiding problems such as patient fatigue and resistance to training due to excessive difficulty, or ineffective training due to excessive difficulty.

[0064] After determining the optimal scene configuration, the system fine-tunes the core parameters of the scene sequentially. First, it adjusts the number of object types, as this directly determines the training richness and complexity of the scene. Based on the optimal configuration parameters, the system adds or removes functional objects that align with the rehabilitation training goals, eliminating redundant and ineffective objects. In the example, if the optimal configuration corresponds to 18 object types, including 12 categories of rehabilitation interactive objects, the system will retain basic rehabilitation training objects, add fine-tuning interactive objects, and remove repetitive functional objects, precisely adjusting the total number of objects in the scene to the corresponding values ​​to ensure that the object types are suitable for the patient's rehabilitation training stage.

[0065] Next, the obstacle density is adjusted. Obstacle density refers to the proportion of obstacle objects to the total number of objects in the scene, ranging from 0 to 0.6. A higher value indicates more obstacles and a higher training difficulty. Based on the optimal density parameters, the system adjusts the number, placement, and occlusion range of obstacles. In the example, the optimal obstacle density is 0.25. The system will evenly distribute a small number of light obstacles along the scene's activity path, maintaining the basic training challenge without excessively restricting the patient's limb movement, thus adapting to the patient's upper limit of motor ability.

[0066] Finally, the difficulty of the movement sequence is adjusted. The difficulty is determined by the continuity of movements in a single training session, the frequency of movement switching, and the accuracy threshold. Based on optimal configuration parameters, the system optimizes the combination sequence of rehabilitation movements, adjusts the ratio of simple to complex movements, and corrects the accuracy threshold for movement completion. In the example, for patients in the recovery phase, the system reduces the proportion of rapidly switching movements, increases the proportion of basic steady-state movements, and relaxes the threshold for small movement deviations to better suit the patient's limb recovery status. For patients in the later stages of rehabilitation, the system appropriately increases the frequency of movement switching and the accuracy requirements to enhance the training effect.

[0067] After completing the fine-tuning and iterative adjustments of all parameters, the system integrates and optimizes the scene object distribution, obstacle layout, action difficulty rules, and spatial interaction logic. It then standardizes and encapsulates the overall scene layout, unifying scene difficulty parameter identifiers, object hierarchy relationships, and interaction trigger rules to form a visualized, quantifiable, and adaptable personalized difficulty adaptation map that can be optimized for subsequent physical constraints. This adaptation map fully embodies the patient's personalized rehabilitation adaptation characteristics, accurately matching the patient's current rehabilitation ability and training needs, and providing a precise preliminary layout benchmark for subsequent optimization of the scene's physical rationality.

[0068] S204, apply rehabilitation dynamics constraints to the personalized difficulty adaptation map, verify the spatial position and relative distance of objects in the scene according to human kinematic parameters and safety boundaries, and adjust the pose and motion trajectory of objects through iterative optimization to generate a scene physical model that conforms to physical rationality. Specifically, the category, spatial coordinates and hierarchical relationship of objects can be extracted from the personalized difficulty adaptation map, and the initial scene layout point cloud can be generated by combining the patient's human kinematic parameters. The core of this step is to extract and integrate personalized rehabilitation scenario data. This involves combining scenario layout data adapted to the patient's training ability with the patient's body movement characteristic data, transforming abstract scenario layout information into refined 3D point cloud data. This provides the original spatial data foundation for subsequent safety constraint verification and physical modeling. The specific implementation method is as follows: The personalized difficulty adaptation map is a standardized scene layout map output from the previous steps. It has been personalized in terms of object complexity and interaction difficulty based on the patient's rehabilitation ability. The map encapsulates the core configuration information of all components of the virtual rehabilitation scene and is the sole data source for extracting scene object parameters. Three types of core scene parameters are extracted from the map first: object category, spatial coordinates, and hierarchical relationship. These three types of parameters completely define the basic spatial structure and object attributes of the virtual rehabilitation scene.

[0069] The object category parameter is used to distinguish between rehabilitation interactive objects and environmental objects with different functions in the virtual scene, adapting to different training scenarios such as limb rehabilitation, balance rehabilitation, and motor function rehabilitation. Common object categories include rehabilitation training balls, limb assistive handrails, movable obstacle barriers, rehabilitation stepping platforms, and spatial limit poles. Different categories of objects correspond to different rehabilitation training actions and interaction logics. In this example, the rehabilitation scene selects three core objects commonly used in upper limb function rehabilitation: elastic training balls, fixed assistive handrails, and movable obstacle barriers, to form a basic rehabilitation interactive scene.

[0070] The spatial coordinate parameters are the three-dimensional spatial positioning data of all scene objects, using a unified three-dimensional Cartesian coordinate system, including three coordinate values: x-axis, y-axis, and z-axis. The coordinate unit is meters, and the numerical precision is uniformly retained to two decimal places. The origin of the coordinate system is the center reference point of the virtual rehabilitation scene. The x-axis and y-axis form the horizontal ground plane of the scene, and the z-axis is the vertical height direction of the ground. The absolute spatial position of each object in the virtual scene is precisely defined. In the example, the initial spatial coordinates of the elastic training ball are (2.15m, 1.80m, 1.20m), the initial spatial coordinates of the auxiliary handrail are (1.50m, 1.80m, 0.90m), and the initial spatial coordinates of the obstacle barrier are (2.15m, 2.50m, 0.75m).

[0071] The hierarchy parameter is used to define the spatial hierarchy and stacking priority of multiple objects in the scene, distinguish the hierarchical order of basic environmental objects, core interactive objects, and auxiliary obstacle objects, and avoid confusion in the spatial logic of objects. The smaller the hierarchy value, the higher the priority of the object and the stronger the basic attributes of the scene. Level 1 is the basic fixed environmental object, level 2 is the core rehabilitation interactive object, and level 3 is the auxiliary adjustment obstacle object. In the example, the auxiliary handrail is a level 1 basic fixed object, the elastic training ball is a level 2 core interactive object, and the obstacle barrier is a level 3 auxiliary obstacle object, which clarifies the scene hierarchy logic of the three.

[0072] Patient kinematic parameters are core baseline data for personalized rehabilitation scenarios. These parameters, obtained through pre-treatment data collection and calibration using rehabilitation equipment, include key parameters such as maximum limb extension length, joint range of motion, limb radius of motion, and trunk movement deviation threshold. All parameters are personalized values ​​adapted to the patient's physical capabilities, eliminating scenario adaptation biases caused by generic parameters. The maximum limb extension length is measured in meters, and the joint range of motion is measured in degrees. In this example, the patient is undergoing rehabilitation for upper limb dysfunction. The maximum unilateral upper limb extension length is 0.90 m, the shoulder joint flexion-extension range is 0 to 170 degrees, the elbow joint rotation range is 0 to 145 degrees, and the limb horizontal radius of motion is 0.95 m, comprehensively representing the patient's limb movement capabilities.

[0073] The initial scene layout point cloud is a set of three-dimensional discretized spatial data generated by fusing scene object parameters and human kinematic parameters. It samples the surface spatial points of all objects in the scene with millimeter-level precision, while simultaneously matching the spatial boundary points of the patient's range of motion, thus fusing and mapping the object distribution space with the human's movable space. The point cloud sampling precision is set to 0.005m, meaning one spatial point is collected every 0.005m, ensuring a refined restoration of the scene layout. The final generated initial scene layout point cloud completely covers the contour space of all scene objects and the human's movable space, accurately restoring the original rehabilitation scene spatial structure before safety optimization.

[0074] Apply safety boundary constraints to each object in the initial scene layout point cloud, calculate the minimum safe distance between the reachable space of human joints and the object position, identify object layouts that violate the constraints, and generate a list of non-compliant layouts. The core of this step is to establish safety verification standards for rehabilitation scenarios. By quantitatively calculating the distance parameters between the human movement space and objects in the scenario, object layouts with potential safety hazards are identified. This allows for the identification of target objects in subsequent scenario optimization and adjustments, ensuring that there are no safety issues such as limb collisions or exceeding movement limits during rehabilitation training. The specific implementation method is as follows: Safety boundary constraints are safety protection rules for human movement set up for virtual rehabilitation scenarios. They are exclusive spatial constraints adapted to rehabilitation training scenarios, and their core function is to limit the minimum distance between objects in the scene and the human body's movable area, preventing dangerous situations such as limbs touching objects, joint movement restriction, and limb collisions or compressions during dynamic rehabilitation training. Safety boundary constraints include two types of parameters: fixed safety thresholds and dynamic adaptation thresholds. The fixed safety threshold is a general basic parameter, set at 0.15m, representing the minimum allowable distance between the edge of the space accessible to human joints and the surface of an object. The dynamic adaptation threshold is fine-tuned according to the patient's degree of limb impairment. The dynamic threshold is 0.12m for patients with mild impairment, 0.15m for patients with moderate impairment, and 0.18m for patients with severe impairment. In the example, the patient has moderate upper limb dysfunction, and a safety boundary threshold of 0.15m is used as the verification standard.

[0075] The human joint reachable space is a three-dimensional spatial envelope region generated by fitting the patient's human kinematic parameters. It completely replicates all reachable ranges of motion of the patient's upper limbs and trunk during training. The space is centered on the patient's virtual standing center of gravity and is constructed by combining parameters such as joint movement angle, limb extension length, and movement radius. The inside of the surface is the patient's safe movement area, the outside of the surface is a free area without movement risk, and the edge of the surface is the movement safety threshold.

[0076] The minimum safe distance is the core calculation parameter in this verification. It refers to the shortest straight-line distance between the critical edge of the accessible space of the human joint and the outer surface contour of the scene object. The unit of calculation is meters, and the precision is retained to three decimal places. This parameter directly determines whether the object layout meets the rehabilitation safety standards. For each object in the initial scene layout point cloud, all point cloud contour points of the object are traversed, and the boundary points of the accessible space of the human are matched one by one. The distance between all points is calculated by traversing all points using the spatial distance calculation formula. The minimum value selected is the minimum safe distance of the object.

[0077] After calculating the distances to all objects, the minimum safe distance for each object is compared with a preset safety boundary threshold to determine compliance. If the minimum safe distance is greater than or equal to 0.15m, the object's layout is considered compliant and will not affect the patient's rehabilitation movements. If the minimum safe distance is less than 0.15m, the object's layout is considered non-compliant, posing a safety hazard of limb collision or restricted movement. In the example, the minimum safe distance for the auxiliary handrail is 0.22m, meeting the safety constraint standard; the minimum safe distance for the elastic training ball is 0.16m, also meeting the safety constraint standard; and the minimum safe distance for the obstacle barrier is 0.09m, less than the preset safety threshold, thus constituting a non-compliant layout.

[0078] After completing the full-scene object verification, information such as the category, original spatial coordinates, hierarchical relationship, minimum safe distance, and violation difference of all non-compliant objects is summarized and integrated to generate a standardized list of non-compliant layouts. The list fully records the detailed parameters of each non-compliant layout, clarifies the objects that need to be optimized and adjusted in the future, and provides accurate target data for iterative optimization and adjustment.

[0079] For objects in the list of non-compliant layouts, the gradient descent method is used to iteratively adjust the three-dimensional coordinates and orientation angles of the objects until all safety boundary constraints are met, generating corrected scene layout data. The core of this step is to dynamically and iteratively correct the spatial position and orientation of non-compliant objects using a gradient descent optimization algorithm, gradually eliminating safety hazards in the scene layout, ensuring that all scene objects meet the rehabilitation safety constraints, and outputting compliant scene layout data. The specific implementation method is as follows: Gradient descent is an iterative numerical optimization algorithm used for scene layout parameter optimization. Its core principle is to take the scene safety distance error as the optimization target, solve the gradient descent direction of the error function, and continuously update the spatial parameters of the object along the direction of error reduction until the error converges to the compliant range. It is suitable for the fine adjustment needs of 3D scene coordinates and pose angles, and has the advantages of stable convergence, high adjustment accuracy, and adaptability to continuous spatial optimization. It is fully suitable for the layout correction scenario of virtual rehabilitation scene.

[0080] This algorithm implementation uses two core hyperparameters: the learning rate and the convergence threshold. The learning rate is set to 0.02, which controls the adjustment range of the object's coordinates and orientation angle during a single iteration. Its value directly impacts optimization efficiency and stability; the 0.02 value strikes a balance between iteration speed and adjustment accuracy, avoiding parameter oscillations due to excessively large step sizes and low optimization efficiency due to excessively small step sizes. The convergence threshold is set to 0.001m, meaning that optimization is considered complete and constraints are met when the difference between the minimum safe distance of the object and the standard safe threshold is less than this value.

[0081] The object adjustment parameters include two categories: three-dimensional coordinates and orientation angles. The three-dimensional coordinates are the spatial position parameters of the object along the x, y, and z axes, with an adjustment accuracy of 0.01m, enabling millimeter-level fine-tuning of spatial position. The orientation angles include three attitude parameters: pitch angle, roll angle, and yaw angle, in degrees, with an accuracy controlled within 0.1 degrees. These correspond to the rotation angles of the object around the x, y, and z axes, respectively, and are used to correct attitude violations such as tilting and deflection, ensuring that the object's attitude is regular and does not intrude into the human body's safe movement space.

[0082] The algorithm's iterative optimization process is a continuous closed-loop adjustment procedure. First, it reads the original parameters and error values ​​of the target objects in the list of objects violating layout rules, calculates the safe error gradient direction of the current layout, and calculates the coordinate and orientation angle adjustments for this iteration based on the learning rate. The object parameters are then updated and corrected. Next, the minimum safe distance of the corrected object is recalculated, and compared to the safety threshold to determine if it meets the standard. If violation errors still exist, the iterative process of gradient calculation, parameter adjustment, and distance verification continues. If the error is less than the convergence threshold, the iteration terminates.

[0083] In the example, the error between the original minimum safe distance and the standard threshold for the illegal obstacle barrier was 0.06m. After eight iterations of gradient descent optimization, the y-axis coordinate and horizontal yaw angle were gradually fine-tuned. Each iteration slightly increased the distance between the object and the human movement space. Finally, the minimum safe distance of the obstacle barrier was optimized to 0.151m, with an error of 0.001m, satisfying the convergence condition and safety constraint standard. After all illegal objects were iteratively optimized, the corrected 3D coordinates, orientation angles, and hierarchical relationship parameters of all objects were integrated. Illegal layout data was removed, and the data was stored in a standardized format to generate corrected scene layout data that fully complies with safety boundary constraints.

[0084] Based on the corrected scene layout data, each object is assigned physical properties including mass, friction coefficient, and elastic coefficient, and kinematic connections between objects are constructed to ultimately generate a scene physics model that conforms to physical rationality.

[0085] The core of this step is to endow the scene space layout, which lacks physical attributes, with realistic physical and mechanical characteristics and motion-related logic. This allows the motion state and interaction effects of virtual scene objects to conform to real-world physical rules, eliminating the physical distortion problem of virtual scenes and constructing a standardized scene physical model that can be adapted to rehabilitation dynamics training. The specific implementation method is as follows: The corrected scene layout data is standardized spatial data that has undergone safety verification and optimization. It accurately records the compliant three-dimensional coordinates, orientation, hierarchical distribution and spatial positional relationships of all scene objects. It is the basic carrier for assigning physical attributes and constructing motion relationships. All spatial parameters meet the safety standards for human rehabilitation and do not require further spatial verification. It can be directly used for physical modeling.

[0086] Physical properties are the core parameters that determine the kinematic performance of virtual objects. These include three core parameters: mass, coefficient of friction, and elastic coefficient. Different types of rehabilitation objects have different parameter values, and all parameters are consistent with the physical characteristics of real-world rehabilitation equipment to ensure the realism of virtual interaction. The mass parameter, measured in kilograms, defines the object's inertia and directly determines the amplitude of the patient's limb's motion response when touching or pushing the object. In the example, the auxiliary handrail is a fixed heavy-duty device with a mass parameter set to 8.0 kg, the elastic training ball is a lightweight interactive device with a mass parameter set to 0.8 kg, and the obstacle barrier is a medium-sized adjustable device with a mass parameter set to 3.5 kg.

[0087] The coefficient of friction refers to the sliding friction parameter of an object's surface, ranging from 0 to 1. A higher value indicates greater surface friction, stronger sliding resistance, and better static stability. 0 represents a frictionless, smooth surface, while 1 represents a surface with maximum frictional roughness. In the example, the auxiliary handrail is made of a non-slip, hard material with a coefficient of friction set to 0.6 to effectively prevent patients from slipping. The elastic training ball is made of a flexible material with a coefficient of friction set to 0.3, ensuring the ball can slide and roll, adapting to dynamic training movements. The obstacle barrier is made of a hard, non-slip material with a coefficient of friction set to 0.5, ensuring the stability of its fixed placement.

[0088] The elasticity coefficient defines an object's ability to deform and rebound after being impacted or compressed by external forces. Its value ranges from 0 to 1; a higher value indicates a more pronounced elastic deformation and rebound effect. 0 represents a rigid object with no elasticity, while 1 represents a fully elastic flexible object. In the example, the metal support handrail is a rigid object with an elasticity coefficient set to 0.1, exhibiting almost no deformation or rebound effect. The flexible rubber training ball has an elasticity coefficient set to 0.7, demonstrating a significant impact rebound effect, suitable for dynamic rehabilitation training. The hard plastic obstacle barrier has an elasticity coefficient set to 0.2, showing no significant deformation upon minor impacts, consistent with the characteristics of real-world equipment.

[0089] Kinematic connectivity defines the motion relationships between multiple objects in a scene. It is divided into two categories: rigid connections and flexible dynamic connections. Rigid connections mean that the objects are in fixed positions with no relative motion, maintaining a unified motion state. Flexible dynamic connections mean that the objects are not subject to fixed constraints and can independently generate displacement, rotation, and deformation motion. In the example, the handrail is rigidly connected to the virtual scene ground, with its position and posture permanently fixed; the obstacle barrier is semi-rigidly connected to the ground, supporting only manual position adjustment and having no autonomous motion; the elastic training ball is dynamically connected to the scene environment, allowing it to freely roll, displace, and rebound as the patient's limbs touch or push it.

[0090] After assigning physical properties and constructing kinematic connections for all objects, the physical parameters, motion association logic, and corrected scene space layout data are deeply integrated and encapsulated. At the same time, the constraints of human rehabilitation dynamics are matched to verify that the physical motion logic of all objects is conflict-free and distortion-free. Finally, a scene physical model that fully conforms to real-world physical rules, is adapted to personalized rehabilitation training, and has safe and controllable interactive capabilities is generated, providing accurate physical scene input for subsequent neural rendering pipelines.

[0091] S205, the scene physical model is input into the neural rendering pipeline, and combined with the preset material feature library and dynamic lighting model, a virtual rehabilitation training scene with immersiveness and interactivity is rendered and output in real time.

[0092] Specifically, it can parse the geometric information and material labels of objects in the scene's physical model, retrieve the corresponding color, roughness, metallicity, and normal map from the preset material feature library, and generate a material parameter mapping table. The core of this step is to decompose the standardized scene physical model data, extract the basic feature information of all virtual objects in the scene, match the corresponding physical rendering parameters based on the preset dedicated material database, complete the precise binding of objects and material parameters, build a standardized material mapping relationship, and provide fine-grained material data support for subsequent scene rendering. The specific implementation method is as follows: The scene physics model is a standardized 3D scene model that has undergone dynamic constraints, safety verification, and iterative optimization. It fully stores the geometric information and custom material labels of all virtual objects in the rehabilitation training scene. The geometric information includes the object's 3D contour structure, vertex spatial coordinates, surface topology, 3D size parameters, and fixed spatial pose. It is the basic data that determines the object's appearance. The material labels are preset classification identifiers for virtual objects in the rehabilitation scene. They are exclusive text identifiers used to distinguish rehabilitation objects with different functions and materials. The label classification is fully adapted to various training scenarios such as limb rehabilitation, balance rehabilitation, and fine motor rehabilitation of the hands. Common labels include metal rehabilitation handrails, soft training spheres, non-slip training floors, matte training walls, and flexible obstacle mats. Each virtual object corresponds to a unique material label to avoid material matching confusion.

[0093] The pre-built material feature library is a standardized material resource database specifically built for virtual rehabilitation training scenarios. All material parameters in the database are adapted to the texture of objects in real rehabilitation training environments, abandoning surreal special effects parameters and fully matching the material characteristics of real objects in hospital rehabilitation rooms and home rehabilitation spaces. For each type of material tag, the database pre-stores four corresponding core rendering parameters: color, roughness, metallicity, and normal map. These four parameters work together to restore the realistic material texture of virtual objects. All parameters are set with a unified precision standard to ensure the consistency and stability of the rendering effect.

[0094] Color parameters refer to the basic inherent color of virtual objects, defined using the RGB three-channel color mode. The values ​​of the red, green, and blue channels range from 0 to 255, with the values ​​corresponding to color saturation and brightness. By combining the values ​​of the three channels, the basic colors of various rehabilitation equipment and environmental objects can be accurately reproduced, adapting to the visual needs of rehabilitation scenarios for simplicity and eye protection. In the example, the color parameters of the metal rehabilitation handrail are set to RGB values ​​of 190, 190, 190, presenting a matte metallic silver-gray, and the color parameters of the soft training ball are set to RGB values ​​of 50, 170, 130, presenting an eye-friendly cyan-green, which meets the visual adaptation standards for rehabilitation training.

[0095] The roughness parameter is used to characterize the smoothness of the virtual object's surface. The parameter value ranges from 0 to 1, with a precision of 0.01. The closer the value is to 1, the rougher the surface texture and the stronger the diffuse reflection effect. The closer the value is to 0, the smoother the surface and the stronger the specular reflection effect. In the example, the anti-slip training floor has been used for patient walking training for a long time and has anti-slip particles on its surface. The roughness parameter is set to 0.78. The matte training wall is made of latex paint and the roughness parameter is set to 0.65. The metal rehabilitation handrail has been frosted and the roughness parameter is set to 0.12, accurately replicating the surface smoothness characteristics of different objects.

[0096] The metallicity parameter is used to define the proportion of metallic material properties in virtual objects. The parameter value ranges from 0 to 1, with a precision of 0.01. A value of 0 represents that the object is completely non-metallic, including materials such as fabric, plastic, latex paint, and rubber. A value of 1 represents that the object is purely metallic. Intermediate values ​​represent the mixed material characteristics of metallic and non-metallic materials. In the example, the metallicity parameter of the metal rehabilitation handrail is set to 0.92, reflecting the metallic material properties. The metallicity parameters of the soft training ball, non-slip floor, and training wall are all set to 0, perfectly matching the non-metallic material characteristics.

[0097] Normal maps are grayscale texture data used to simulate the microscopic bumps and depressions on the surface of an object. They can accurately reproduce the fine texture details of the object's surface without increasing the number of faces and vertices in the 3D model, which can greatly improve the realism of scene rendering while ensuring rendering efficiency. Different material tags correspond to exclusive normal maps. In the example, the anti-slip floor matches the particle texture normal map, the metal handrail matches the brushed metal texture normal map, the soft training ball matches the frosted soft rubber texture normal map, and the wall matches the smooth latex paint texture normal map.

[0098] After completing the retrieval and matching of all parameters, the system treats each virtual object in the scene as an independent unit, and associates the object's geometric identifier, material label, corresponding color parameters, roughness parameters, metallicity parameters, and normal map resource address one by one. It integrates the material matching data of all objects and generates a standardized material parameter mapping table. This mapping table has unique correspondence, which can ensure that each virtual object can accurately call its own material parameters during the subsequent rendering process, eliminating the problems of material confusion and parameter missingness, and providing accurate material data support for subsequent neural rendering.

[0099] Based on the time setting and environmental requirements of the rehabilitation scenario, load dynamic lighting model parameters, including the direction of the main light source, ambient light intensity, and shadow sampling rate, to generate dynamic lighting configuration data; The core of this step is to dynamically match the corresponding core lighting parameters by combining the simulated time environment of the virtual rehabilitation scenario with the patient's training environment requirements, construct a lighting system adapted to the rehabilitation training scenario, restore the light and shadow effects of a real indoor space, and improve the environmental realism and visual comfort of the virtual scenario. The specific implementation method is as follows: The rehabilitation scenario time settings are divided into three basic scenario dimensions: daytime indoor, dusk indoor, and nighttime indoor. Each corresponds to different basic natural lighting conditions. The daytime indoor scenario simulates the lighting effect of natural light shining through the windows into the rehabilitation room during the day, with the overall light being uniform and bright. The dusk indoor scenario simulates the lighting effect of weak natural light in the evening combined with indoor supplementary lighting, with the light being soft and dim. The nighttime indoor scenario simulates the lighting effect that relies entirely on indoor artificial lighting, with the light being concentrated and having a strong sense of layering. The three time settings cover the scenario simulation needs of most rehabilitation training.

[0100] The environmental requirements for rehabilitation settings are categorized based on the patient's visual tolerance and the type of training program. These include three types: standard bright training environment, soft eye-protecting training environment, and low-light immersive training environment. The standard bright training environment is suitable for routine limb strength and joint mobility training, ensuring clear visual field. The soft eye-protecting training environment is suitable for rehabilitation training of patients with eye sensitivity and dizziness susceptibility, reducing light stimulation. The low-light immersive training environment is suitable for balance training and attention training, weakening environmental interference and improving training focus.

[0101] The direction of the main light source is the spatial illumination angle parameter of the core lighting light in the scene. It is defined by two dimensions: horizontal azimuth and vertical pitch. The horizontal azimuth ranges from 0 to 360 degrees with an accuracy of 0.5 degrees. It is used to characterize the circumferential illumination direction of the main light source on the horizontal plane. The vertical pitch ranges from 0 to 90 degrees with an accuracy of 0.5 degrees. It is used to characterize the vertical illumination height of the main light source from the horizontal ground. The larger the angle value, the higher the light source is and the wider the illumination range. In the example, the direction of the main light source in the standard bright rehabilitation scene during the day is set to a horizontal azimuth of 60 degrees and a vertical pitch of 55 degrees to simulate the illumination angle of natural light above the side of the room, which is consistent with the lighting layout of a real rehabilitation room.

[0102] Ambient light intensity is used to characterize the global diffuse illumination intensity of the entire virtual scene. The parameter unit is nits, and the value ranges from 0 nits to 1000 nits. The larger the value, the higher the overall basic brightness of the scene and the more uniform the light coverage. 0 nits means there is no global diffuse light, and the scene only retains the direct light from the main light source. 1000 nits is the maximum illumination of the scene. In the example, the ambient light intensity of the soft eye-protection training environment is set to 380 nits, and the light is soft and not dazzling. The ambient light intensity of the low-light immersive training environment is set to 180 nits, and the light is dim and focused. The ambient light intensity of the standard bright training environment is set to 480 nits, and the light is sufficient and uniform.

[0103] The shadow sampling rate controls the level of detail in scene shadow generation. The parameter ranges from 16 to 128 and is an integer parameter. A larger value means more pixels are sampled for the shadow, resulting in smoother shadow edge transitions, richer details, and higher realism of light and shadow. A smaller value means fewer sampling points, lower rendering computation, and a more stable frame rate, but slight jagged edges may appear on the shadow edges. In the example, the shadow sampling rate for a typical rehabilitation training scene is set to 64, which balances shadow realism with real-time rendering efficiency and meets the frame rate requirements of virtual reality interaction. For high-precision detail training scenes, it can be set to 96 to optimize shadow texture to the extreme.

[0104] After determining the scene time setting and environmental requirements, the system automatically matches the corresponding main light source direction, ambient light intensity, and shadow sampling rate parameters, binds and integrates the scene environment identifier with the three types of lighting parameters, and encapsulates them into structured dynamic lighting configuration data. The data completely records all the core parameters of scene lighting, and the parameter logic is mutually compatible. It can be directly input into the subsequent rendering pipeline to complete global lighting and shadow calculation, realizing dynamic adaptive adjustment of the lighting effect of the rehabilitation scene.

[0105] The scene physical model, material parameter mapping table and dynamic lighting configuration data are input into the neural rendering pipeline, and a differentiable renderer is used to generate color values ​​and depth information pixel by pixel, and output the initial rendering frame. The core of this step is to integrate scene geometric and physical data, material data, and lighting data. Relying on the intelligent rendering capabilities of the neural rendering pipeline, it uses a differentiable rendering algorithm to perform precise pixel-level calculations to generate an initial rendered image with complete object shapes, material textures, and basic lighting effects. This lays the foundation for subsequent image optimization. The specific implementation method is as follows: The Neural Rendering Pipeline is an end-to-end intelligent rendering process adapted to real-time interactive virtual reality scenarios. It is specifically optimized for rehabilitation training scenarios, eliminating the high latency defects of traditional offline rendering. It can achieve millisecond-level real-time rendering and stably maintain a rendering frame rate of 60 frames per second, fully meeting the needs of patients' real-time interactive training. The pipeline can simultaneously receive three types of core input data: scene physical model, material parameter mapping table, and dynamic lighting configuration data, and complete three types of pre-processing work: geometric analysis, material mapping, and lighting calculation, respectively, ensuring the continuity and efficiency of the rendering process.

[0106] The scene physics model provides the rendering pipeline with the 3D geometry, spatial pose, and physical constraint boundary data of all objects, ensuring that the rendered object shapes and spatial layouts fully conform to the rules of human kinematics and physical dynamics, and eliminating problems such as scene object interpenetration, misalignment, and shape distortion; the material parameter mapping table accurately matches exclusive color, roughness, metallicity, and texture parameters for each virtual object in the pipeline, ensuring accurate and consistent reproduction of object surface texture; dynamic lighting configuration data provides a standardized lighting and shadow calculation benchmark for the entire scene, unifying the light angle, overall brightness, and shadow detail, and ensuring the coordination of lighting and shadow effects throughout the scene.

[0107] The differentiable renderer is the core computational unit in this step, possessing pixel-level differential solution and iterative calculation capabilities. Unlike traditional fixed rendering algorithms, it can adjust pixel output results in real time according to subtle changes in scene parameters, adapting to the dynamic rendering needs of personalized rehabilitation scenes. The renderer uses a single pixel of the virtual image as the smallest computational unit, traversing and calculating all pixels within the resolution range one by one. Combining the object material parameters, global illumination parameters, and 3D spatial position corresponding to the pixel, it simultaneously solves for the core image data of each pixel.

[0108] Pixel color values ​​are the final color data of a pixel, defined using normalized floating-point values. The three RGB channels all range from 0 to 1, with a precision of up to 0.001. This accurately reproduces the light and shadow color differences and gradations produced by light superimposed on the surface of an object's material, solving the problem of harsh and layered colors in traditional rendering. In the example, the pixel color values ​​of the metal handrail under direct light are 0.74, 0.74, and 0.74, while the pixel color values ​​of the handrail in the diffuse reflection area are 0.68, 0.68, and 0.68, naturally presenting the light and shadow levels.

[0109] Pixel depth information is the straight-line distance between the current pixel in the 3D scene space and the virtual viewing angle, in meters, with an accuracy of 0.001 meters. It is used to define the spatial hierarchy of objects in the scene, realizing the realistic reproduction of object occlusion and distance. In the example, the pixel depth value of the training ball at close range is 2.150 meters, the pixel depth value of the handrail at medium range is 3.620 meters, and the pixel depth value of the wall at far range is 5.890 meters, accurately distinguishing the spatial distance relationship of different objects.

[0110] After calculating the color value and depth information of all pixels, the system integrates and arranges all pixel data according to the screen resolution matrix, and splices them together to form a complete two-dimensional screen frame, which is the initial rendering frame. This screen frame has completely restored the object structure, material texture and basic lighting effects of the rehabilitation scene. The screen layout conforms to physical rules, but there are problems such as jagged edges, harsh light and shadow transitions and weak spatial hierarchy. These issues need to be optimized and upgraded through subsequent post-processing algorithms.

[0111] Temporal anti-aliasing and post-processing effects are applied to the initial rendered frame, including depth-of-field blur and ambient occlusion, to ultimately output an immersive and interactive virtual rehabilitation training scene.

[0112] The core of this step is to repair the image defects of the initial rendering frame through multiple post-processing algorithms, optimize image details, spatial hierarchy, and lighting texture, while fully preserving the physical interaction logic of the scene. This achieves both visual immersion and interactive practicality, outputting a virtual scene that can be directly used for rehabilitation training. The specific implementation method is as follows: Temporal anti-aliasing is a pixel optimization algorithm adapted to dynamic virtual interactive scenarios. Its core function is to eliminate pixel jaggedness, dynamic flickering, and edge blurring at the edges of objects in the image. Compared with static anti-aliasing algorithms, this algorithm can dynamically optimize by linking multiple consecutive frames of image data. It collects edge pixel data from the current initial rendering frame and the previous two historical rendering frames, and uses a temporal interpolation algorithm to smooth the sub-pixel areas of object outlines, scene boundaries, and equipment edges. It accurately corrects the jaggedness and flickering problems caused by dynamic viewpoint rotation and object interaction, ensuring that the image remains clear and smooth for patients during dynamic training without visual jitter interference.

[0113] The depth-of-field blur effect simulates the natural visual characteristics of the human eye by relying on the pixel depth information of the initial rendering frame. It optimizes the spatial hierarchy of the image by following the visual rule of near objects being sharp and far objects being blurry. The algorithm divides the core focus area, near-field area, and far-field area according to the pixel depth value. For the core object area of ​​the patient training interaction, the pixel sharpness is kept fixed and no blurring is applied. For the near ground and corner areas that are too close to the observation point, and the far wall and peripheral obstacle areas that are too far away, different intensity of blur effect is superimposed according to the depth difference gradient. The larger the depth difference, the higher the blur intensity. In the example, the hand training equipment area of ​​the core interaction is kept in ultra-clear focus state, and the far-field scene with a depth greater than 4 meters is lightly blurred to effectively focus the training target and enhance the visual immersion of the scene.

[0114] Ambient occlusion effects are used to optimize the realistic details of local lighting and shadows in a scene, compensating for the defects of transparent lighting and shadows and lack of layering at the junctions of objects in the basic rendering. The algorithm calculates the spatial occlusion relationship between adjacent objects in the scene, identifies areas where light cannot directly hit, such as the joints of objects, corner gaps, and contact surfaces, and calculates the light attenuation of pixels in the corresponding areas to automatically generate natural local shadows. In the example, the contact position between the training ball and the training table, the connection gap between the handrail support and the ground, and the corners where the wall and the ground meet can all generate delicate occlusion shadows through this effect, completely solving the problem of flat lighting and shadows and lack of three-dimensionality in the image, making the overall scene lighting and shadow effects fit the real physical environment.

[0115] After completing temporal anti-aliasing optimization and applying dual post-processing effects such as depth-of-field blurring and ambient occlusion, all image defects in the initial rendered frame were fully repaired. The image edges are smooth and regular, the spatial hierarchy is clear, and the lighting and shadow details are delicate and rich, providing a strong sense of visual immersion. At the same time, all post-processing optimizations only adjust the visual effects of the image, fully preserving the dynamic constraints, object interaction logic, and safety boundary rules of the scene's physical model. All virtual objects in the scene can accurately respond to the patient's hand touch, limb movement, positional movement, and other interactive actions, supporting real-time interactive feedback for various rehabilitation training actions. The final generated virtual rehabilitation training scene balances a highly immersive visual experience with stable interactive performance and can be directly applied to various personalized virtual reality rehabilitation training tasks.

[0116] Another embodiment of the present invention provides an automatic construction system for virtual reality rehabilitation scenes based on semantic understanding, see [link to relevant documentation]. Figure 3 The system may include: The receiving module 301 is used to receive multimodal data containing rehabilitation training needs. The multimodal data includes voice commands, text descriptions, or demonstration videos. The different modal data are mapped to a unified semantic space through a cross-modal semantic encoder to generate a rehabilitation intention vector. The construction module 302 is used to construct an initial semantic graph of the rehabilitation scene based on the rehabilitation intention vector using a graph attention network. The nodes of the initial semantic graph represent virtual objects or environmental elements, the edges represent the interaction relationships between nodes, and the semantic graph is converted into a three-dimensional scene layout heatmap using a conditional generative adversarial network. The search module 303 is used to learn a personalized preference reward function from the patient's historical rehabilitation data through inverse reinforcement learning based on the three-dimensional scene layout heatmap, and to dynamically adjust the object complexity and interaction challenge in the scene using Monte Carlo tree search to generate a personalized difficulty adaptation map. The verification module 304 is used to apply rehabilitation dynamics constraints to the personalized difficulty adaptation map, verify the spatial position and relative distance of objects in the scene according to human kinematic parameters and safety boundaries, and adjust the pose and motion trajectory of objects through iterative optimization to generate a scene physical model that conforms to physical rationality. The output module 305 is used to input the scene physical model into the neural rendering pipeline, and combine it with a preset material feature library and dynamic lighting model to render and output an immersive and interactive virtual rehabilitation training scene in real time.

[0117] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0118] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0119] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.

[0120] The above description, based on the embodiments shown in the figures, details the structure, features, and effects of the present invention. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for automatically constructing virtual reality rehabilitation scenes based on semantic understanding, characterized in that, The method includes: Receive multimodal data containing rehabilitation training needs, including voice commands, text descriptions, or demonstration videos, and map different modal data to a unified semantic space through a cross-modal semantic encoder to generate a rehabilitation intention vector; Based on the rehabilitation intention vector, an initial semantic graph of the rehabilitation scene is constructed using a graph attention network. The nodes of the initial semantic graph represent virtual objects or environmental elements, and the edges represent the interaction relationships between nodes. The initial semantic graph is topologically optimized to remove redundant nodes and low-weight edges, and necessary spatial constraints are added to generate a simplified rehabilitation scene semantic graph. The simplified rehabilitation scene semantic graph is then converted into a three-dimensional scene layout heatmap using a conditional generative adversarial network. Based on the aforementioned 3D scene layout heatmap, a personalized preference reward function is learned from the patient's historical rehabilitation data through inverse reinforcement learning, and Monte Carlo tree search is used to dynamically adjust the object complexity and interaction challenge in the scene to generate a personalized difficulty adaptation map. Rehabilitation dynamics constraints are applied to the personalized difficulty adaptation map. The spatial position and relative distance of objects in the scene are verified according to human kinematic parameters and safety boundaries. The pose and motion trajectory of the objects are adjusted through iterative optimization to generate a scene physical model that conforms to physical rationality. The physical model of the scene is input into the neural rendering pipeline, and combined with a preset material feature library and dynamic lighting model, a virtual rehabilitation training scene with immersive and interactive effects is rendered and output in real time.

2. The method according to claim 1, characterized in that, The process involves receiving multimodal data containing rehabilitation training needs, including voice commands, text descriptions, or demonstration videos. A cross-modal semantic encoder maps different modal data to a unified semantic space, generating a rehabilitation intention vector, including: The system receives user voice commands through a voice acquisition module, converts the voice signal into a text sequence using a voice recognition algorithm, and simultaneously receives text descriptions through a text input interface to generate original text data of the same modality. The video acquisition module receives demonstration videos, and the pose estimation algorithm is used to extract key frames and human skeleton sequences from the videos. The skeleton sequences are then encoded into motion feature vectors to generate motion modal feature data. The original text data and action modality feature data are input into the cross-modal semantic encoder, which consists of a pre-trained language model and a graph convolutional network, to extract text semantic embeddings and action semantic embeddings respectively, generating heterogeneous semantic embedding vectors. By using a cross-modal attention fusion mechanism, heterogeneous semantic embedding vectors are projected onto a unified semantic space and the semantic representations of different modalities are aligned, ultimately generating a rehabilitation intention vector that represents the intention of rehabilitation training.

3. The method according to claim 1, characterized in that, Based on the rehabilitation intention vector, an initial semantic graph of the rehabilitation scene is constructed using a graph attention network. Nodes in the initial semantic graph represent virtual objects or environmental elements, and edges represent the interaction relationships between nodes. Topology optimization is performed on the initial semantic graph to remove redundant nodes and low-weight edges, and necessary spatial constraints are added to generate a simplified rehabilitation scene semantic graph. A conditional generative adversarial network is then used to convert the simplified rehabilitation scene semantic graph into a 3D scene layout heatmap, including: The key entities and action requirements in the rehabilitation intention vector are analyzed, and matching node candidate sets are retrieved from the preset virtual object library and environmental element library to generate a node candidate set. The candidate set of nodes is input into the graph attention network. The semantic relevance weights between nodes are calculated through the attention mechanism, and the interaction relationship edges between nodes are constructed based on the weights to generate a weighted initial semantic graph. The initial semantic graph is topologically optimized to remove redundant nodes and low-weight edges, and necessary spatial constraints are added to generate a simplified semantic graph of the rehabilitation scene. The simplified rehabilitation scene semantic graph is input into a conditional generative adversarial network. The simplified rehabilitation scene semantic graph is used as a conditional generator to generate a probability density field of object distribution in three-dimensional space, and finally outputs a three-dimensional scene layout heatmap.

4. The method according to claim 3, characterized in that, Based on the 3D scene layout heatmap, a personalized preference reward function is learned from the patient's historical rehabilitation data through inverse reinforcement learning. Monte Carlo tree search is then used to dynamically adjust the object complexity and interaction challenge in the scene, generating a personalized difficulty adaptation map, including: Historical rehabilitation data, including the trajectory of completed movements, error rate, and subjective difficulty score, is extracted from the patient's electronic rehabilitation record to generate a patient historical behavior dataset. An initial state space is constructed based on a 3D scene layout heatmap. An inverse reinforcement learning algorithm is used to inversely deduce the implicit personalized preference reward function parameters from the patient's historical behavior dataset, thereby generating a preference reward function model. By combining the preference reward function model with the Monte Carlo tree search algorithm, the tree is expanded with the current scene layout as the root node. The cumulative reward value under different combinations of object complexity and interaction challenge is evaluated, and the search tree evaluation result is generated. Based on the search tree evaluation results, select the scene configuration with the highest reward value, adjust the number of object types, obstacle density, and motion sequence difficulty in the scene, and finally output a personalized difficulty adaptation map.

5. The method according to claim 4, characterized in that, The process involves applying rehabilitation dynamics constraints to the personalized difficulty adaptation map, verifying the spatial position and relative distance of objects in the scene based on human kinematic parameters and safety boundaries, and iteratively optimizing and adjusting the pose and trajectory of the objects to generate a physically plausible scene physical model, including: Extracting the categories, spatial coordinates, and hierarchical relationships of objects from the personalized difficulty adaptation map, and combining them with the patient's human kinematic parameters to generate an initial scene layout point cloud; Apply safety boundary constraints to each object in the initial scene layout point cloud, calculate the minimum safe distance between the reachable space of human joints and the object position, identify object layouts that violate the constraints, and generate a list of non-compliant layouts. For objects in the list of non-compliant layouts, the gradient descent method is used to iteratively adjust the three-dimensional coordinates and orientation angles of the objects until all safety boundary constraints are met, generating corrected scene layout data. Based on the corrected scene layout data, each object is assigned physical properties including mass, friction coefficient, and elastic coefficient, and kinematic connections between objects are constructed to ultimately generate a scene physics model that conforms to physical rationality.

6. The method according to claim 5, characterized in that, The process of inputting the scene's physical model into the neural rendering pipeline, combining it with a pre-defined material feature library and a dynamic lighting model, and rendering and outputting an immersive and interactive virtual rehabilitation training scene in real time includes: The system parses the geometric information and material labels of objects in the scene's physical model, retrieves the corresponding color, roughness, metallicity, and normal map from the preset material feature library, and generates a material parameter mapping table. Based on the time setting and environmental requirements of the rehabilitation scenario, load dynamic lighting model parameters, including the direction of the main light source, ambient light intensity, and shadow sampling rate, to generate dynamic lighting configuration data; The scene physical model, material parameter mapping table and dynamic lighting configuration data are input into the neural rendering pipeline, and a differentiable renderer is used to generate color values ​​and depth information pixel by pixel, and output the initial rendering frame. Temporal anti-aliasing and post-processing effects are applied to the initial rendered frame, including depth-of-field blur and ambient occlusion, to ultimately output an immersive and interactive virtual rehabilitation training scene.

7. A virtual reality rehabilitation scene automatic construction system based on semantic understanding, characterized in that, The system includes: The receiving module is used to receive multimodal data containing rehabilitation training needs, including voice commands, text descriptions or demonstration videos, and to map different modal data to a unified semantic space through a cross-modal semantic encoder to generate a rehabilitation intention vector. The construction module is used to construct an initial semantic graph of the rehabilitation scene based on the rehabilitation intention vector using a graph attention network. The nodes of the initial semantic graph represent virtual objects or environmental elements, and the edges represent the interaction relationships between nodes. The initial semantic graph is topologically optimized to remove redundant nodes and low-weight edges, and necessary spatial constraints are added to generate a simplified rehabilitation scene semantic graph. The simplified rehabilitation scene semantic graph is then converted into a three-dimensional scene layout heatmap using a conditional generative adversarial network. The search module is used to learn a personalized preference reward function from the patient's historical rehabilitation data through inverse reinforcement learning based on the three-dimensional scene layout heatmap, and to dynamically adjust the complexity of objects and the interactive challenge in the scene using Monte Carlo tree search to generate a personalized difficulty adaptation map. The verification module is used to apply rehabilitation dynamic constraints to the personalized difficulty adaptation map, verify the spatial position and relative distance of objects in the scene according to human kinematic parameters and safety boundaries, and adjust the pose and motion trajectory of objects through iterative optimization to generate a scene physical model that conforms to physical rationality. The output module is used to input the scene physical model into the neural rendering pipeline, and combine it with a preset material feature library and dynamic lighting model to render and output an immersive and interactive virtual rehabilitation training scene in real time.

8. The system according to claim 7, characterized in that, The receiving module is specifically used for: The system receives user voice commands through a voice acquisition module, converts the voice signal into a text sequence using a voice recognition algorithm, and simultaneously receives text descriptions through a text input interface to generate original text data of the same modality. The video acquisition module receives demonstration videos, and the pose estimation algorithm is used to extract key frames and human skeleton sequences from the videos. The skeleton sequences are then encoded into motion feature vectors to generate motion modal feature data. The original text data and action modality feature data are input into the cross-modal semantic encoder, which consists of a pre-trained language model and a graph convolutional network, to extract text semantic embeddings and action semantic embeddings respectively, generating heterogeneous semantic embedding vectors. By using a cross-modal attention fusion mechanism, heterogeneous semantic embedding vectors are projected onto a unified semantic space and the semantic representations of different modalities are aligned, ultimately generating a rehabilitation intention vector that represents the intention of rehabilitation training.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1-6 when it is run.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1-6.

Images