Integrated Design Digital Interaction Methods and Systems Based on Virtual Reality
By introducing causal consistency ordering and dynamic latency compensation that periodically couples logical frames and rendering frames into the virtual reality system, and combining graph attention networks and reinforcement learning agents, the temporal alignment and computational conflict problems of multimodal interaction in virtual reality are solved, realizing highly immersive and low-latency intelligent design interaction.
Patent Information
- Application Number
- CN202611009995.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-08
- Publication Date
- 2026-11-10
AI Technical Summary
Existing virtual reality-based integrated design digital interaction technologies suffer from inconsistent rendering frame rates, significant differences in eye-tracking, gesture, and voice latency characteristics in multi-person collaborative indoor lighting and air conditioning design scenarios, network-introduced disorder, and real-time inference of high-dimensional physical field coupled graph networks and reinforcement learning agents crowding out rendering computing power, leading to interaction latency or frame rate reduction.
By periodically coupling logical frames and rendering frames, causal consistency sorting, dynamic latency compensation, and frame drift filtering are performed to generate a multimodal fusion data stream that is time-sequentially aligned with the rendering frames. High-level design semantics are parsed through a graph attention network, and combined with a physical perception design coupled graph network and reinforcement learning agent, the optimal design parameter adjustment scheme is deduced in real time to achieve closed-loop iteration within a single frame period.
It achieves highly immersive and low-latency intelligent design interaction, solves the problems of timing alignment and computing power conflict, and quickly adapts to personalized preferences through counterfactual reasoning and meta-learning mechanisms, reducing computational complexity and ensuring rendering frame rate.
Smart Images

Figure CN123067702A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of virtual reality and human-computer interaction technology, specifically to a digital interaction method and system based on the integrated design of virtual reality. Background Technology
[0002] Digital interaction based on virtual reality integrated design refers to the use of terminal devices such as virtual reality head-mounted displays, gesture sensors, eye trackers, microphone arrays, and physiological signal sensors, combined with centralized event sequencing servers, rendering engines, and physical field simulations, to form a complete hardware and software system that supports designers to conduct immersive design modifications, parameter adjustments, and scheme comparisons in a virtual three-dimensional space, and to achieve natural collaboration between humans and the system through multimodal interaction.
[0003] Currently, existing integrated design digital interaction technologies based on virtual reality mainly adopt the following approach: the system collects multimodal input data from users, synchronizes the data to the rendering frame through simple timestamp alignment or fixed buffering mechanisms, and then identifies the user's simple operation intentions, such as selection, movement, and scaling, based on rules or pre-trained models, and provides preset visual feedback or parameter adjustment suggestions.
[0004] In scenarios such as collaborative indoor lighting and air conditioning design, the system needs to collect eye movement, gestures, voice, and physiological signals in real time, and then deduce the optimal parameter combination after fusion. Existing technologies have some problems in the application of such scenarios: the rendering frame rate of different terminals is inconsistent, the latency characteristics of eye movement, gestures, and voice are significantly different, the network introduces out-of-order signals, and relying solely on physical timestamps cannot handle cross-terminal causal dependencies, resulting in distortion of intent recognition input; the real-time inference of high-dimensional physical field coupled graph networks and reinforcement learning agents crowds out rendering computing power, the computational complexity of all nodes is high, and it is impossible to complete the closed loop within a single frame period, causing interaction delay or frame rate drop. Summary of the Invention
[0005] To achieve the above objectives, the present invention provides the following technical solution:
[0006] A virtual reality-based integrated design approach for digital interaction includes the following steps:
[0007] Multimodal raw data from multiple user terminals, including eye movements, gestures, speech, and physiological signals, are coupled periodically with virtual reality rendering frames. Causal consistency sorting, dynamic latency compensation, and frame drift filtering are then applied to obtain a multimodal fusion data stream aligned with the rendering frame's temporal sequence. Based on this multimodal fusion data stream, the current design micro-process stage is identified through temporal behavior pattern mining. High-level design semantics are then analyzed using a graph attention network. The identification results from this stage are combined with the semantic classification results to obtain a design intent vector. A physical perception design coupling graph network is constructed, incorporating the environmental physical field, user perception thresholds, and design parameters. The current physical field is determined based on the gaze coordinates and gesture spatial range in the multimodal fusion data stream. The active region is simulated, and the physical field variables and user perception threshold variables of the nodes within the active region are extracted as feature vectors. The feature vectors are concatenated with the design intent vector and input into the reinforcement learning agent to deduce the adjustment amount of adjustable design parameters. At the beginning of each frame, frame adaptive dynamic pruning is performed on the coupled graph network, and cold-start prior bias is injected into the reinforcement learning agent in atypical scenarios to obtain the optimal design parameter adjustment scheme. The optimal design parameter adjustment scheme is transformed into visual guidance information and superimposed and rendered into the virtual reality scene. User feedback gestures are received, and parameter locking, physical field updating, experience storage, and policy network fine-tuning are completed within a single frame cycle, outputting the updated design parameters.
[0008] Furthermore, the causal consistency sorting includes: arranging multimodal raw data in ascending order of physical timestamps; when multiple data packets have the same timestamp, sequentially checking semantic dependencies of operation types, spatial overlap dependencies, and inter-user dependencies to determine the order; when cyclic dependencies are detected, determining the priority according to the order of physical timestamps, and allocating global logical frame numbers according to the final order; the dynamic latency compensation includes: establishing independent latency estimation models for eye movement, gestures, and speech respectively; calculating the predicted latency in real time based on the historical arrival time deviation sequence; and subtracting the predicted latency from the data packet timestamp to achieve preliminary timeline alignment.
[0009] Furthermore, the frame drift filtering process includes: maintaining a historical offset queue, calculating a weighted moving average to obtain the predicted drift amount, and triggering cubic spline interpolation resampling when the absolute value of the predicted drift amount reaches a preset drift amount threshold to generate virtual data points aligned with the frame period; obtaining the multimodal fusion data stream aligned with the rendering frame time sequence also includes: dynamically calculating the confidence coefficient of each modality based on the head rotation angular velocity, hand occlusion ratio, and environmental signal-to-noise ratio, and normalizing and weighting the eye movement, gesture, and speech data within the same global logical frame according to the confidence coefficient to obtain the multimodal fusion data stream.
[0010] Furthermore, the process of identifying the current design micro-process stage through temporal behavior pattern mining includes: mapping the multimodal fusion data stream into an event sequence with spatiotemporal labels, searching for frequent pattern fragments within an adaptive sliding window and matching them with a preset design micro-process dictionary, and outputting a one-hot encoded vector as the micro-process stage identification result.
[0011] Furthermore, the process of parsing high-level design semantics through graph attention network includes: defining the semantic calculation focus area with the gaze point as the center, organizing the multimodal features in the area into a spatiotemporal tensor to construct graph neural network nodes and edges, and aggregating multi-hop neighbor information through stacked graph attention layers to output a five-dimensional probability vector as the semantic classification result.
[0012] Furthermore, the construction process of the physical perception design coupled graph network includes: adaptively discretizing the virtual indoor space into a parameterized grid, associating each grid node with physical field variables, user perception threshold variables, and adjustable design parameter variables, and establishing connection edges between nodes based on spatial adjacency and physical influence radius; the frame adaptive dynamic pruning includes: determining the physical inference active region by the union of the gaze point hotspot region and the gesture interaction region, and only performing complete computation graph and reinforcement learning inference on active nodes located in the physical inference active region or connected to it by a high-weight physical influence edge, while transferring the computational load of inactive nodes to the background asynchronous queue.
[0013] Furthermore, the adjustment amount of the adjustable design parameters deduced by the reinforcement learning agent is specifically as follows: the feature vector and the design intent vector are concatenated and input into a three-layer fully connected policy network, and the ten-dimensional continuous action value is output as the adjustment amount of the adjustable design parameters; the reinforcement learning agent integrates counterfactual reasoning and meta-learning mechanisms, including: when a veto instruction is received, generating counterfactual candidate schemes that are adjacent in the parameter space and converting them into auxiliary reward signals and real feedback weighted fusion to update the policy network, and loading a probability diffusion prior model to generate an initial policy bias injection agent in atypical scenarios, and performing gradient updates through a small amount of online interactive decision-making to quickly adapt to personalized preferences.
[0014] Furthermore, the process of converting the optimal design parameter adjustment scheme into visual guidance information and overlaying it onto the virtual reality scene includes: generating a semi-transparent preset state model with preset transparency and a confidence heat bar representing the acceptance probability, and overlaying it onto the current virtual scene; the process of receiving user feedback gestures includes recognizing adoption gestures, fine-tuning gestures, or rejection gestures, wherein the fine-tuning gesture records the parameter values after dragging or rotating and calculates the maximum relative adjustment range, and assigns the corresponding explicit reward signal value according to the interval where the maximum relative adjustment range is located;
[0015] The process of completing parameter locking, physics field update, experience storage, and policy network fine-tuning within a single frame cycle includes: persistentizing the design parameter database through asynchronous pipeline parallel scheduling, triggering forward computation of the coupled graph network to update the full-space physics field variables, storing the interaction trajectory in the experience replay pool, and sampling historical trajectories from the experience replay pool to perform small-batch gradient updates on the policy network; when the graphics processor load continuously exceeds a preset threshold, a degradation protection mechanism is automatically triggered, adjusting the policy network fine-tuning and some physics field calculations to be performed once every preset number of frames, prioritizing the frame rate of the main rendering thread.
[0016] The integrated design digital interaction system based on virtual reality includes a multimodal data synchronization processing module, which processes raw interactive data from multiple terminals by performing causal consistency sorting, dynamic latency compensation, and frame drift filtering through periodic coupling of logical frames and rendering frames, outputting a multimodal fusion data stream aligned with the rendering frame time sequence; a design intent recognition module, which identifies the design stage through temporal behavior pattern mining based on the multimodal fusion data stream, and parses the design semantics through a graph attention network to output a design intent vector; a physical perception inference module, which constructs a graph network containing physical fields, user perception, and design parameters, determines the inference focus based on the multimodal fusion data stream and the design intent vector, and outputs the optimal design parameter adjustment scheme through a reinforcement learning agent integrating counterfactual reasoning and meta-learning mechanisms; and a real-time closed-loop update module, which transforms the adjustment scheme into visual information and renders it, receives user feedback gestures, and asynchronously triggers parameter updates, physical field recalculation, experience storage, and policy network fine-tuning within a single frame cycle to complete the closed-loop iteration.
[0017] Furthermore, the multimodal data synchronization processing module, when performing causal consistency ranking, determines the order based on the semantic type of operations, spatial overlap, and user dependencies, and establishes delay estimation models for eye-tracking, gesture, and speech modalities for dynamic compensation; the design intent recognition module identifies the design phase by mining frequent behavioral patterns in event sequences, and aggregates spatiotemporal features to analyze semantics within the semantic focus region centered on the gaze point through a graph attention network; the physical perception inference module, when constructing the graph network, associates physical field variables, user perception variables, and design parameter variables with spatial grid nodes, and... In each frame, dynamic pruning is performed on the graph network based on user gaze and interaction region to extract inference features; the policy network of the reinforcement learning agent outputs parameter adjustment based on inference features and design intent vector, the reward function integrates explicit user feedback and implicit physiological comfort feedback, and integrates counterfactual reasoning and meta-learning mechanisms to adapt to personalized preferences; the real-time closed-loop update module identifies the user's adoption, fine-tuning or rejection gestures, and performs physiological signal processing and reward calculation based on them in a low-frequency asynchronous process through asynchronous pipeline parallel scheduling of parameter persistence, physical field update, experience replay storage and policy network online learning operations.
[0018] This invention provides an integrated design digital interaction method and system based on virtual reality, which has the following beneficial effects:
[0019] 1. This invention achieves precise synchronization and hierarchical semantic mapping by periodically coupling logical frames and rendering frames, generating a local logical frame number for each frame and forcibly marking it. Through centralized causal consistency sorting and multimodal dynamic latency compensation, it outputs a strictly aligned fused data stream, solving the temporal alignment problem. It also introduces frame adaptive dynamic pruning, performing full calculation only on active nodes in the current gaze and operation area, significantly compressing the real-time calculation amount, releasing resources to ensure the rendering pipeline, and solving the problem of computing power conflict.
[0020] 2. This invention integrates counterfactual reasoning and meta-learning mechanisms into a reinforcement learning agent. When a user rejects a solution, it automatically generates counterfactual candidate solutions with multi-dimensional biases. The acceptance probability is converted into a weighted fusion of auxiliary rewards and real feedback through a response prediction network, and the preference boundary is learned from a single rejection. The model-independent meta-learning framework is used to pre-train the policy network, so new users only need a few key decisions to quickly adapt to personalized preferences, avoiding the inefficient exploration in the cold start phase.
[0021] 3. This invention achieves seamless iteration of decision-making, feedback, and learning through non-intrusive visualization and single-frame asynchronous parallel closed-loop rendering. It uses a semi-transparent model and confidence heatsink overlaid for rendering, allowing users to intuitively compare differences between solutions without interrupting their operations. Parameter locking, physics field updates, experience storage, and strategy fine-tuning are scheduled in parallel through asynchronous pipelines, reducing the total time consumption. When computing power is insufficient, the strategy fine-tuning frequency is automatically downgraded, so users are unaware of the background computing load, achieving highly immersive, low-latency, and adaptive intelligent design interaction. Attached Figure Description
[0022] Figure 1 This is a flowchart of a method according to an embodiment of the present invention.
[0023] Figure 2 This is a flowchart of the multimodal data synchronization and design intent recognition process of the present invention.
[0024] Figure 3 This is a flowchart of the physical perception deduction and parameter optimization process of the present invention.
[0025] Figure 4 This is a system framework diagram of Embodiment 2 of the present invention. Detailed Implementation
[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] Example 1:
[0028] Please see Figures 1 to 3 This embodiment provides an integrated design digital interaction method based on virtual reality, including steps 1 to 4.
[0029] The system referred to in this embodiment is an integrated design digital interaction system based on virtual reality, which is a complete hardware and software system consisting of virtual reality terminal devices, a centralized event sequencing server, a rendering engine, a physics simulation part, a reinforcement learning agent part, and a visualization feedback part; virtual reality terminal devices include head-mounted displays, gesture recognition sensors, eye trackers, microphone arrays, and physiological signal sensors; this method runs on this system to achieve multimodal data synchronization, design intent recognition, intelligent deduction, and real-time closed-loop iteration.
[0030] Step 1: Precise synchronization and purification of multimodal data based on the periodic coupling of logical frames and virtual reality rendering frames: For multimodal raw data such as eye movement, gestures, voice, and physiological signals from multiple user terminals, the system performs periodic phase-locking of logical frames and rendering frames, causal consistency sorting, dynamic delay compensation, and frame drift filtering to output a clean multimodal fusion data stream that is strictly aligned with the rendering frames, providing a precise timing reference for subsequent design intent recognition.
[0031] Step 1.1: Logical Frame Generation and Multimodal Data Marking: Generate a local logical frame number for each rendered frame and forcibly mark it on all sensor data collected within that frame period, establishing a local frame sequence number benchmark for subsequent server-side sorting.
[0032] In a virtual reality collaborative session, each user's terminal independently maintains a local logical frame counter with an initial value of 0. The system listens to the vertical synchronization signal of the virtual reality rendering engine. Whenever the graphics processor finishes drawing a new frame, the vertical synchronization signal emits a pulse or the terminal reads the rendering completion interrupt signal when running in variable refresh rate mode, and the local logical frame number is incremented by 1.
[0033] In multi-user collaborative scenarios, the rendering frame rates of different virtual reality terminals may differ. In this case, each terminal independently generates a local logical frame number without needing to maintain frame rate synchronization. Subsequently, the centralized event sorting server in step 1.2 globally sorts the data according to the physical timestamp and then reallocates the globally unique logical frame number to complete cross-terminal time sequence normalization.
[0034] Each user's terminal is equipped with a sensor array to collect multimodal raw data. The sensor array includes at least: an eye tracker for collecting the three-dimensional spatial coordinates of the user's gaze points, a gesture recognition sensor for collecting the positions and rotation angles of 21 key points on both hands, a microphone array for collecting voice command text and semantic anchoring parameters, and a physiological signal sensor for collecting heart rate variability and skin conductance. The 21 key points are set according to the hand skeleton model, covering 1 wrist, 5 metacarpophalangeal joints, 5 proximal interphalangeal joints, 5 distal interphalangeal joints, and 5 fingertips, totaling 21 points, which can completely describe hand posture and fine movements.
[0035] When the terminal collects raw data, it synchronously records a high-precision physical timestamp and forces the local logical frame number corresponding to the current rendering frame to be synchronously marked on each data record, forming a raw multimodal data packet with both local logical frame number and physical timestamp.
[0036] Step 1.2: Causal Consistency Ordering and Modal Differentiation Dynamic Delay Compensation: The centralized event ordering server receives all data packets, sorts them by physical timestamp, determines the order of data packets with the same timestamp using a causal consistency protocol, assigns global logical frame numbers according to the final order, and performs dynamic delay compensation for different modalities, outputting a preliminarily aligned multimodal data stream.
[0037] Step 1.2.1: Receive all raw multimodal data packets uploaded by user terminals through a centralized event sequencing server, sort them in ascending order of physical timestamps. When multiple data packets have the same physical timestamp, a causal consistency protocol is used to determine their order. The core rules are as follows:
[0038] Check operation dependencies:
[0039] Operation type semantic dependencies; for example, the operation to delete an object must occur after the object is created; the operation to modify the material must occur after the object is created.
[0040] Spatial overlap dependency: If the spatial ranges of two operations overlap, they are sorted by the default priority of the operation type, such as creation takes precedence over modification, and modification takes precedence over deletion.
[0041] Inter-user dependency: In the same collaborative session, when user B performs an operation on an object created by user A, that operation automatically depends on user A's creation operation.
[0042] When a dependency exists, the operation being depended on comes first, followed by the operation that depends on it. When no dependency exists, the two data packets can be processed in parallel to improve collaboration efficiency. For operation pairs whose dependency cannot be determined by the above rules, such as two users making design modifications independently in different regions, they are considered to have no dependency and are allowed to be processed in parallel. When a circular dependency is detected between operations, the priority is determined according to the order of the physical timestamps of the user operations.
[0043] In the final determined order, each data packet is assigned an incrementing global logical frame number starting from 0. This frame number is strictly consistent with the actual physical order and causal relationship of the events, serving as a unified time reference for all subsequent steps.
[0044] By using the above rules, the protocol can reconstruct a globally ordered sequence of events without relying on the order of physical timestamps, thus fundamentally immunizing against the problem of out-of-order data packets caused by network latency jitter.
[0045] Step 1.2.2: After completing the causal ordering, establish independent delay estimation models for each modality to address the differences in delay characteristics:
[0046] Eye-tracking delay estimation model: Assuming that the delay of eye-tracking data is mainly determined by the sensor sampling frequency, the delay is modeled as a fixed value of 15 milliseconds plus Gaussian noise with a mean of 0 and a standard deviation of 3 milliseconds; the parameters of this model are obtained by least squares fitting of historical data. 15 milliseconds is the typical hardware delay of mainstream VR eye trackers; 3 milliseconds is the statistical standard deviation of sampling jitter.
[0047] Gesture delay estimation model: Considering the continuity of gesture data, its delay is modeled as a first-order linear dynamic system. The delay is equal to the delay of the previous moment multiplied by the attenuation factor of 0.9 plus the current measurement deviation multiplied by the gain of 0.1. The value is selected to balance the smoothness of gesture movement and the agility of response. In order to make the delay estimation smoothly track the continuous changes of gesture movement and respond quickly to abrupt changes, simulation tests show that the attenuation factor of 0.9 and the gain of 0.1 are the optimal compromise combination.
[0048] Voice delay estimation model: Considering the suddenness and variable length of voice commands, the delay is modeled as the sum of the voice endpoint detection time and the semantic parsing time. Based on the end-to-end delay test range of mainstream speech recognition engines, the delay value ranges from 50 milliseconds to 200 milliseconds.
[0049] Each latency estimation model calculates the predicted latency of the current data packet based on the deviation sequence between the actual arrival time and the theoretical arrival time of the most recent 10 data packets. The window length is 10 to cover the typical network jitter period, which is about 0.5 to 1 second, and the computational overhead is small. The predicted latency of the current data packet is calculated in real time. Then, the timestamp of each data packet is subtracted from the predicted latency, so that the data of each modality is initially aligned to the same logical frame time on the time axis. After processing, the initially aligned multimodal data stream is output.
[0050] Step 1.3: Frame drift compensation filtering and cross-modal semantic confidence weighted fusion: For the initially aligned data stream, the time axis is finely aligned by predicting rendering drift and resampling. Then, the data quality of each modality is dynamically evaluated and weighted fusion is performed based on the interaction context to obtain the final fused stream.
[0051] Fine-grained resampling processing is performed on the initially aligned multimodal data stream: The system maintains a historical offset queue of length 10, which stores the time deviation between the actual rendering completion time and the theoretical rendering cycle time of the most recent 10 frames. The length of 10 can smooth short-term fluctuations without causing response lag, and covers the typical graphics processor rendering load change cycle.
[0052] For the current frame, the weighted moving average of these time deviation values is calculated as the predicted drift of the current frame. The deviation values closer to the current frame are given higher weights. The weights decay exponentially, with the decay factor ranging from 0.7 to 0.9 and the default value being 0.85. This ensures that the sum of the weights of the most recent 5 frames accounts for a significant proportion, highlighting the recent drift trend while retaining a certain historical smoothing effect.
[0053] Adaptive resampling of the data stream is performed based on the predicted drift: if the absolute value of the predicted drift is less than 5 milliseconds, it indicates that the time deviation of the current frame is within the allowable range, and the original sampled data points are directly retained; if the absolute value of the predicted drift is greater than or equal to 5 milliseconds, cubic spline interpolation is triggered to generate several virtual data points that are strictly aligned with the frame period between two adjacent sampling points, while retaining the extreme value features in the original sampling points, such as the peak position and peak speed when the gesture is quickly waved; the human eye's perception delay threshold for visual persistence is about 10 milliseconds, and a value of 5 milliseconds is taken here as a safety margin to ensure sub-frame level alignment.
[0054] The specific calculation method of cubic spline interpolation is as follows: a cubic polynomial function is constructed between two adjacent original sampling points. The first and second derivatives of this function are continuous at the nodes, thereby generating a smooth interpolation curve. The number of interpolation points is linearly determined according to the magnitude of the predicted drift. One interpolation point is added for every millisecond increase in drift, with a maximum of 10 interpolation points. The basis for setting 10 interpolation points is that when the drift exceeds 10 milliseconds, it exceeds the threshold of human visual perception. Too many interpolation points will increase the computational load without any practical benefit.
[0055] After completing the timeline resampling, a cross-modal semantic confidence weighting mechanism is run to dynamically calculate the confidence coefficient of each modality's data based on the current interaction context. The specific evaluation rules are as follows:
[0056] Eye movement confidence coefficient: When the detected head rotation angular velocity of the user is >60° / s, the eye movement confidence coefficient is set to the default value of 0.3; otherwise, it is set to 0.9. The setting basis is: 60° / s is the threshold for distinguishing between natural saccades and violent head turning. When the value is higher than this, the noise of the eye movement signal increases sharply, so the weighting is reduced.
[0057] Gesture confidence coefficient: When the gesture recognition sensor detects a hand occlusion ratio > 30%, the gesture confidence coefficient is set to 0.4; otherwise, it is set to 0.9. Basis for setting: Official accuracy tests of gesture recognition show that the recognition accuracy drops by about 60% when the occlusion exceeds 30%, hence the weighting is reduced.
[0058] Voice confidence coefficient: When the microphone array detects an ambient signal-to-noise ratio of <15 dB, the voice confidence coefficient is set to 0.2; otherwise, it is set to 0.8. The setting basis is that the effective lower limit of voice endpoint detection is about 15 dB. Below this value, wake-up and recognition are unreliable, and the weight is reduced to 0.2.
[0059] For physiological signals, such as heart rate variability and skin conductance response, since their frequency is much lower than the rendering frame rate, they do not participate in the real-time weighted fusion frame by frame. Instead, they are downsampled to 1Hz by the sliding window averaging method and then buffered independently. When calculating the implicit reward in step 3.3, the original sampled values are used directly. The reliability of this treatment is always set to 1.0.
[0060] The data of each modality are normalized and weighted according to their confidence coefficients: the fusion contribution of each modality is equal to the confidence coefficient of that modality divided by the sum of the confidence coefficients of all modalities; the multimodal data within the same global logical frame are weighted and summed according to their contribution to generate the fusion data record of that frame; physiological signals are not included in this summation and are kept separately as the original sequence.
[0061] The processed output is a high-fidelity, time-series-free multimodal fusion data stream, i.e., MMF data stream, with each record strictly corresponding to a global logical frame.
[0062] Step 2: Temporal Behavior Pattern Mining and Hierarchical Design Semantics Real-time Mapping: For the MMF data stream output from Step 1, the user's current design micro-process stage is identified through frequent pattern mining, and then the high-level design semantics are analyzed through graph attention network. Finally, a structured ten-dimensional design intent vector is output to accurately depict the user's cognitive state.
[0063] Step 2.1: Identification of Microprocess Stages: Map the MMF data stream into a sequence of events with spatiotemporal labels, and identify the current design stage by matching it with the historical knowledge base using a sliding window and frequent pattern mining algorithm.
[0064] The MMF data stream is mapped to the virtual reality 3D spatial coordinate system to generate a sequence of event tuples. Each event tuple contains at least the following fields: the global logical frame number where the event occurred, the 3D spatial coordinates where the event occurred, the modality type field of the event, and the specific parameter value of the event.
[0065] A frequent pattern mining algorithm with time constraints is used to process the event tuple sequence. The execution flow is as follows:
[0066] Set the initial default time window length, with a default value of 10 seconds. Based on the design stage results identified in the previous frame or time window, adaptively adjust the current window length: if the previously identified stage was the spatial exploration stage, set the current window length to 5 seconds; if the previously identified stage was the fine-tuning stage, such as material iteration or lighting parameter adjustment, set the current window length to 15 seconds; for the preliminary layout planning stage and the scheme comparison and confirmation stage, maintain the default window length of 10 seconds; when running for the first time or when the system cannot identify the previous stage, use the default 10-second window length; the spatial exploration stage involves rapid behavior changes, with an average cycle of 2-3 seconds for staring or turning the head, making 5 seconds sufficient for capturing the mode; the fine-tuning stage involves long operation repetition cycles, with material loops lasting approximately 10-15 seconds, where 15 seconds can fully encompass multiple loops, making 10 seconds a general compromise;
[0067] Search for recurring behavioral pattern fragments within a time window. A behavioral pattern fragment is defined as at least three event tuples arranged in chronological order.
[0068] The frequency of each behavioral pattern fragment in the entire session is calculated, and fragments that appear more than 5 times above the preset minimum support threshold are marked as frequent patterns. Based on the statistical analysis of 100 sets of experimental data, 5 times can effectively filter out random behaviors, while retaining statistically significant design patterns.
[0069] Frequent pattern fragments are matched with a pre-built design micro-process dictionary, which is constructed by offline mining of at least 1,000 sets of historical design session data. Each entry in the dictionary maps a behavioral pattern fragment to a micro-process stage type with clear design semantics. When multiple frequent patterns match at the same time, the design stage corresponding to the pattern with the highest frequency is selected. If the frequencies are the same, the pattern that is most recent in time is selected.
[0070] The microprocess stage types and their corresponding behavioral pattern characteristics are as follows:
[0071] Space exploration phase: characterized by five or more consecutive staring spells, each lasting more than 0.5 seconds, combined with a wide range of head rotations, with a cumulative rotation angle exceeding 90°;
[0072] Preliminary layout planning stage: characterized by alternating gesture drag operations and viewpoint switching operations, with the time interval between drag operations and viewpoint switching being less than 2 seconds;
[0073] Material iteration and debugging phase: Characterized by a loop pattern of "select material, observe effect, change material" repeating more than 5 times within 30 seconds;
[0074] Lighting parameter adjustment stage: characterized by staring at the light source followed by a numerical sliding operation, the duration of which is between 2 and 5 seconds;
[0075] Scheme comparison and confirmation stage: characterized by rapidly switching viewpoints between two or more preset schemes, with each scheme being viewed for 3 to 8 seconds, followed by a confirmation gesture.
[0076] It should be noted that the five micro-process stages mentioned above are not strictly unidirectional and progressive. Users may jump back and forth between different stages during the actual design process. For example, they may return from the material debugging stage to the preliminary layout stage to make structural adjustments. Step 2.1 determines the current stage in real time based on the behavior pattern within the current time window, supporting flexible switching between stages.
[0077] The above matching process determines the user's current design stage in real time and outputs a five-dimensional one-hot encoded vector, where the dimension corresponding to the current stage is 1 and the other dimensions are 0.
[0078] Step 2.2: Classification of high-level design semantics: Within the focal region of the user's current attention, multimodal data is organized into a spatiotemporal tensor and input into a pre-trained graph attention network to obtain the probability distribution of high-level design semantics.
[0079] The multimodal features within each rendered frame are organized into a three-dimensional tensor. The three dimensions of this three-dimensional tensor are time, space, and modality. The time dimension is indexed using global logical frames as units, with a time window length of 15 consecutive logical frames. The space dimension is indexed using grid coordinates in virtual reality space, with a grid size of 0.2 meters by 0.2 meters. The modality dimension is indexed using three modal types: eye movement, gesture, and speech. The 15-frame rule is based on the fact that at 72 frames per second, approximately 0.21 seconds is sufficient to capture pattern information of short-term action sequences such as grasping or gaze shifting, while the computational complexity of the graph attention network is controllable. The 0.2-meter grid rule is based on the fact that the spatial resolution of fine hand operations in ergonomics is approximately 0.2 meters, such as the minimum perceptual distance for fingertip movement, which is also a typical granularity for fine-tuning walls and furniture in interior design.
[0080] Since the number of spatial grid nodes in real-world scenarios is large, potentially exceeding a thousand, directly inputting all nodes as graph nodes into the graph attention network would result in excessive computation. Therefore, in actual calculations, based on the gaze coordinates and gesture spatial range in the multimodal fusion data stream output from step 1, a spherical region with a radius of 2 meters centered on the gaze point is defined as the semantic analysis focus area. Graph attention network calculations are performed only on the spatial grid nodes within this region. The 2-meter radius is based on the typical viewing distance of a designer examining a local space in virtual reality; nodes outside this region contribute very little to the current semantics. For nodes in inactive regions, the output features from the previous frame are directly reused as approximations. Through this localization strategy, the number of nodes actually participating in the graph attention network calculation is usually no more than 200.
[0081] The 3D tensor is input into a pre-trained graph attention network for processing.
[0082] Graph Neural Network Node Construction: All modal data at each time step and each spatial grid location in the 3D tensor are considered as a graph node. The initial feature vector of each graph node is composed of the original values of eye movement, gesture, and speech modalities at that location at that time step.
[0083] Edge connection rules: Establish edge connections between nodes in adjacent time steps, and also establish edge connections between nodes with a spatial adjacent distance of less than 0.5 meters; the basis for 0.5 meters: the spatial correlation of the physical field influence significantly decays within about half a meter; the edge weights are initialized as a function of spatial distance and time interval, and can be learned and adjusted during training.
[0084] Graph Attention Layer: Three stacked graph attention layers. Each graph attention layer calculates the association strength between each node and its neighboring nodes through an attention mechanism, gradually aggregating information from multi-hop neighbors. The association strength is calculated as follows: the feature vector of the current node is multiplied by a learnable weight matrix to obtain the query vector, the feature vector of the neighboring node is multiplied by another learnable weight matrix to obtain the key vector, the inner product of the query vector and the key vector is calculated, and then the attention coefficient, i.e., the association strength, is obtained after SoftMax normalization.
[0085] Output layer: After three layers of graph attention processing, the features of all nodes are aggregated into a fixed-length global feature vector through a global average pooling layer, and then the probability value of each design semantic category is output through a fully connected layer and SoftMax.
[0086] The offline training method for the graph attention network is as follows: at least 2000 sets of interactive conversation data with designed semantic categories already labeled by experts are used as the training set. The cross-entropy loss function and Adam optimizer are used for training, with the learning rate set to 1‰, the batch size set to 32, and the training epochs set to 100. This configuration is determined based on grid search and early stopping: on 2000 sets of labeled data, the cross-entropy loss can converge to the stationary region the fastest, and the validation set loss fluctuates by less than 0.1% for 10 consecutive epochs, with no risk of overfitting.
[0087] After training, the network can output the probability distribution of the high-level design semantic category to which the current interaction segment belongs. The design semantic category includes at least five categories: spatial layout planning semantic category, material and texture matching semantic category, lighting environment adjustment semantic category, furniture arrangement adjustment semantic category, and style atmosphere switching semantic category. The output layer of the graph attention network outputs a five-dimensional probability vector through the SoftMax function, representing the probability distribution of the design semantic category to which the current interaction segment belongs. The five dimensions correspond to the five semantic categories mentioned above, and the value of each dimension is between 0 and 1, and the sum of the probability values of the five dimensions is equal to 1.
[0088] The five-dimensional one-hot encoded vector output in step 2.1 is concatenated with the five-dimensional probability vector output in step 2.2 to obtain a ten-dimensional vector, which is called the design intent vector.
[0089] It should be noted that this method involves two different levels of graph structures: the graph attention network in step 2.2 is a graph neural network at the machine learning model level, used to identify high-level design semantic categories from multimodal data streams; the coupled graph network in step 3 is a system-level data structure used to characterize the dynamic interaction between the environmental physical field, user perception threshold, and design parameters. The two graph structures function independently, but in the closed-loop update in step 4, the design intent vector output in step 2.2 provides key information such as gaze coordinates and gesture operation space range for the coupled graph pruning in step 3.3.
[0090] Step 3: Construction and Enhanced Real-Time Inference of Physically Aware Design Coupled Graph Network: Construct a physically aware design coupled graph network that includes environmental physical fields, user perception thresholds, and design parameters. Employ a reinforcement learning agent that integrates counterfactual reasoning and meta-learning mechanisms, combined with frame adaptive computation graph dynamic pruning and cold-start prior injection, to output the optimal design parameter adjustment scheme in real time, solving the problems of computing power conflict and cold-start inefficiency.
[0091] Step 3.1: Construction of the physical perception design coupled graph network: The virtual indoor space is discretized into a parameterized grid, each grid node is associated with three types of state variables, and the connection edges between nodes are established according to the spatial adjacency and physical influence rules, outputting a dynamically updated graph network.
[0092] A physical perception design coupled graph network (hereinafter referred to as the coupled graph network) is constructed, which includes three elements: environmental physical field, user perception threshold, and design parameters. The virtual interior space is discretized into a dense parametric grid. The grid division precision is adaptively adjusted according to the size of the design scene: for spaces with an area of less than 50 square meters, the grid size is set to 0.3 meters × 0.3 meters; for spaces with an area between 50 and 150 square meters, the grid size is set to 0.5 meters × 0.5 meters; and for spaces with an area greater than 150 square meters, the grid size is set to 1 meter × 1 meter. The basis for this is: based on the space division granularity commonly used in the field of interior design, spaces under 50 square meters are considered small spaces, and 0.3 meters can preserve furniture details; spaces between 50 and 150 square meters are considered medium spaces, and 0.5 meters balances precision and computing power; spaces over 150 square meters are considered large spaces, and 1 meter ensures real-time performance.
[0093] Each node in the parameterized mesh is associated with three types of variables:
[0094] Physical field variables include: light intensity value, in lux, provided by the lighting calculation module of the virtual reality rendering engine, ranging from 0 to 20,000 lux; thermal comfort index value, dimensionless, calculated by the environmental physics simulation module based on the location of air conditioning vents, set temperature, and personnel distribution, ranging from 0 to 1, referring to the corresponding relationship of predicted average vote PMV in ASHRAE Standard 55 thermal comfort standard, below 0.5 indicates discomfort, 0.5 to 0.7 indicates acceptable, and above 0.7 indicates comfort; sound pressure level value, in decibels, calculated by the acoustic simulation module based on the sound source location and the sound absorption coefficient of the space materials, ranging from 0 to 120 decibels.
[0095] User perception threshold variables include: average dwell time, in seconds, obtained by statistically analyzing the user's historical interaction trajectory at the node location. A dwell time queue of length 100 is maintained for each node, and the arithmetic mean of the queue is calculated; gaze time entropy, dimensionless, obtained by calculating the information entropy of the user's gaze points near the node. The entropy value is equal to the sum of the logarithms of all negative grid areas where the gaze point falls in that area, multiplied by the probability of that probability to base 2. The larger the entropy value, the more scattered the user's attention is. The value ranges from 0 to 5.5; micro-attitude adjustment amplitude, in degrees, obtained by analyzing and accumulating the changes in the user's head pitch angle, yaw angle, and roll angle. The value ranges from 0 to 90 degrees.
[0096] Adjustable design parameter variables: Includes 10 preset adjustable design parameters, covering the most significant factors affecting user comfort and design effect in interior design. The 10 parameters are: main lighting illuminance, auxiliary ambient lighting illuminance, air conditioning set temperature, horizontal component of air supply angle, vertical component of air supply angle, wall diffuse reflection coefficient, floor diffuse reflection coefficient, furniture surface diffuse reflection coefficient, sound-absorbing material density, and curtain light transmittance. Each parameter has its own value range, determined by hardware specifications and design constraints.
[0097] Connection edges are established between nodes according to two rules:
[0098] Based on spatial adjacency: Edges are established between nodes in the grid with a Manhattan distance ≤ 2, and the edge weight is set to the spatial adjacency weight value of 1.0. The Manhattan distance is defined as the sum of the absolute values of the differences in the 3D grid coordinates. The basis for Manhattan distance ≤ 2 is to ensure that the diffusion of the physical field on the graph satisfies numerical stability. Each node has approximately 6 to 32 neighbors, and the computational cost is moderate. This rule ensures that the propagation and diffusion of the physical field in the spatial grid can be effectively modeled in the graph network.
[0099] Based on the physical radius of influence: For light source nodes, edges are established between all nodes within their light propagation distance, with edge weights set to the normalized value of light intensity decreasing with the square of distance; the influence radius of the light source is determined by its power, with the influence radius increasing by 1 meter for every 100 lumens increase in power, based on the inverse square law of light intensity. A 100-lumen point light source has an illuminance of approximately 80 lux at a distance of 1 meter, which meets the typical requirements of indoor lighting; For air conditioning vent nodes, edges are established between all nodes within their airflow influence radius of 3 meters, with edge weights set to decrease linearly with distance; the influence radius of the light source node is determined by its power, with the influence radius increasing by 1 meter for every 100 lumens increase in power; the 3-meter limit is based on the fact that the effective influence distance of air conditioning airflow in actual indoor environments is typically 3–5 meters, and the minimum value is taken to limit the scale of the graph.
[0100] Based on the above definitions of nodes and edges, a dynamic coupled network with a graph structure is formed. The initial state of the network is determined by the current indoor design parameters and the current physical field simulation results. The state of the network will be updated once per frame as the design parameters are adjusted and user behavior is fed back. The state update of the coupled graph network adopts a message passing mechanism. The new state of each node is obtained by weighted summation of its own old state and the states of its neighboring nodes, with the weights being the edge weights. The 50-dimensional representative feature vector extracted from the coupled graph network is called the coupled graph state vector.
[0101] Step 3.2: Construction and training of reinforcement learning agent integrating counterfactual reasoning and meta-learning: Construct reinforcement learning agent, train and optimize it by integrating reward functions of explicit and implicit user feedback, and combining two major mechanisms: counterfactual reasoning preference learning and meta-learning rapid adaptation.
[0102] The policy network of the reinforcement learning agent adopts a three-layer fully connected neural network structure. The first layer is the input layer with a dimension of 60. It corresponds to the concatenation of the 10-dimensional design intent vector output in step 2 and the 50 representative feature nodes extracted from the coupled graph network in step 3.1. The extraction of these 50-dimensional features depends on the physical inference active region defined in step 3.3.1. In actual operation, the pruning in step 3.3.1 is performed first, and then the feature vectors of all nodes in the active region are extracted. Each node is about 10-dimensional. Principal component analysis is used to reduce the dimension to 50, retaining 99% of the variance. The second layer is the hidden layer with a dimension of 128. The activation function is the ReLU function. The third layer is the output layer with a dimension of 10 consecutive action values, which correspond to the adjustment amount of the above 10 adjustable design parameters. The value of each action value ranges from -0.1 to 0.1, representing the relative change ratio of the parameter value.
[0103] The reward function of the reinforcement learning agent incorporates two types of signals:
[0104] Explicit reward signals: When a user confirms the system's recommended solution, the reward value is 1.0; when a user makes a minor adjustment to the system's recommended solution with an adjustment range of less than 5%, the reward value is 0.5; when a user makes a minor adjustment to the system's recommended solution with an adjustment range between 5% and 20%, the reward value is 0; when a user rejects the system's recommended solution, the reward value is -1.0. 5% is the minimum perceptible difference of parameter change for the human body, and 20% is the critical point for significant change. Experiments have verified that this grading best matches users' psychological ratings.
[0105] Implicit reward signal: The implicit reward consists of two parts: heart rate variability reward and dwell time reward, specifically: Implicit reward = (heart rate variability reward × skin conductance consistency coefficient) + dwell time reward, where:
[0106] Heart rate variability reward: When the high-frequency power component of heart rate variability (HF-HRV) increases by more than 10%, the reward value is 0.3; when it decreases by more than 10%, the reward value is -0.3; when the change is within ±10%, the reward value is 0; the 10% change range is based on the empirical threshold of significant change in physiological signal. In most research literature, a change exceeding 10% of the baseline is considered to be statistically and physiologically significant.
[0107] Skin conductance consistency coefficient: When the trend of skin conductance changes is consistent with the trend of HF-HRV changes, the coefficient is 1.0; if the two are opposite, the coefficient is 0.5; a change in skin conductance exceeding 5% of the baseline within 2 consecutive seconds is considered an increase or decrease, and the baseline is the moving average of the previous 10 seconds;
[0108] Dwell Time Reward: When the average dwell time of a user at a certain spatial grid node increases by more than 20% compared to the baseline value, the reward value is 0.2; when it decreases by more than 20%, the reward value is -0.2; otherwise, it is 0. The baseline value is the average value of the past 1 minute. The 20% threshold is based on the statistics of a pre-experiment with 50 designers: when users have a significant behavioral response to changes in comfort, the average change in dwell time is 18.7%, and 20% is taken as the safety threshold.
[0109] To avoid interference with physiological signals from non-design factors, the following conditions are considered only when calculating the heart rate variability reward: a) The user does not perform any major designed operation, such as dragging or voice input, for 3 consecutive seconds; b) The user's gaze falls within a circular area centered on the node coordinates with a radius equal to the grid size, such as 0.2 meters, and the gaze duration exceeds 2 seconds; the final implicit reward is the average of all active nodes.
[0110] The final reward value is the weighted sum of explicit and implicit rewards, with explicit rewards weighted at 0.7 and implicit rewards weighted at 0.3. The weighting is based on user research showing that explicit operational intentions are more reliable than physiological signals, and are therefore given higher weights.
[0111] To improve the learning efficiency of this reinforcement learning agent in scenarios with few samples, two enhancement mechanisms were integrated.
[0112] The first enhancement mechanism is counterfactual reasoning preference learning.
[0113] When a user rejects or modifies a design solution recommended by the system, a set of counterfactual candidate solutions that are adjacent to the rejected solution in the parameter space is automatically generated. The counterfactual candidate solutions are generated as follows: for each parameter dimension of the rejected solution, a preset small offset is added or subtracted. The absolute value of the offset is 5% of the parameter value of the rejected solution, but not less than 0.01 and not more than 0.1. These single-dimensional offsets are then combined. For 10 parameter dimensions, each dimension has two offset directions, namely addition and subtraction, generating a total of 20 counterfactual candidate solutions. The 5% offset can generate counterfactual solutions that are sufficiently similar to the original solution but also have differences. This can effectively learn the user's preference boundaries without deviating from the user's design intent. The 20 solutions cover all major single-dimensional change directions, and the computational load is controllable.
[0114] Each counterfactual candidate solution, along with the current design intent vector and coupling graph state vector, is input into a pre-trained user response prediction network. The user response prediction network adopts the same three-layer fully connected neural network structure as the reinforcement learning agent policy network, but its output layer is changed to a single output node. This output node, after passing through the Sigmoid activation function, obtains a value between 0 and 1, representing the predicted probability that the counterfactual candidate solution will be accepted by the user in the current state. The user response prediction network is trained by using all solutions that have been genuinely responded to by users in historical interaction data as training samples. Solutions that are adopted are labeled as positive samples (label 1), and solutions that are rejected are labeled as negative samples (label 0). The binary classification cross-entropy loss function is used for training.
[0115] During the system's cold start phase, when sufficient historical feedback data has not yet been accumulated, the user response prediction network cannot provide accurate predictions. Therefore, the following progressive initialization strategy is adopted: In the first 10 sessions, the initial weights of the user response prediction network are obtained through transfer learning, using an offline dataset from 500 anonymous users for pre-training to obtain a general prediction model. From the 11th to the 50th session, real user feedback data is gradually added to the training set, with incremental fine-tuning performed every 20 sessions. The learning rate is set to 0.0003, and the training epochs are set to 5. 20 sessions are sufficient to accumulate enough new data for effective fine-tuning, and the smaller learning rate and fewer training epochs prevent the network from forgetting previously learned general knowledge. After 50 sessions, the network enters a stable operating state. Each time real user feedback is received, including acceptance, rejection, and fine-tuning, the feedback data is immediately added to the experience pool, and incremental training is performed in the background using the closed-loop update mechanism in step 4.2.
[0116] The predicted probabilities of counterfactual candidate solutions are converted into auxiliary reward signals. The conversion method is: auxiliary reward = average predicted probability - 0.5, with its value ranging from -0.5 to 0.5. Then, the auxiliary reward is multiplied by the auxiliary weight of 0.5 and added to the real feedback reward to obtain the total reward used to update the policy network, that is: total reward = real feedback reward + 0.5 × (average predicted probability - 0.5). The basis for the auxiliary reward weight of 0.5 is: through A / B testing, comparing weights of 0.3, 0.5 and 0.7, the policy convergence speed is the fastest when the weight is 0.5 and it does not destroy the dominance of real feedback. This mechanism enables the system to learn the user's preference boundary in the multidimensional parameter space from the user's single rejection behavior.
[0117] The second enhancement mechanism is rapid personalized adaptation driven by meta-learning.
[0118] In the offline phase, design interaction log data from at least 500 different users were used, covering user groups of different ages, design experiences, and aesthetic preferences; the reinforcement learning agent was pre-trained using a Model-Independent Meta-Learning (MAML) training framework.
[0119] The training process of Model-Independent Meta-Learning is as follows: For each training task, all interaction trajectories of a user are randomly selected from the dataset. The user's trajectory is divided into a support set and a query set in chronological order. The support set contains the first 30% of the interaction records, and the query set contains the last 70% of the interaction records. The reinforcement learning agent first performs two gradient updates on the support set with a learning rate of 1‰ to obtain temporary parameters. Then, the loss value corresponding to the temporary parameters is calculated on the query set. The final optimization goal is to minimize the average loss value of all training tasks on the query set. This training process is repeated for 200 iterations. This setting follows the classic MAML framework: two in-step gradient updates are sufficient to capture the user's preference direction and are computationally lightweight; the 30% and 70% partitions ensure that the support set is sufficiently initialized; after 200 iterations, meta-validation and loss monitoring have reached a plateau.
[0120] After the above training, the agent's policy network parameters are initialized as meta-parameters. During the online interaction phase, when a new user uses the system for the first time, the system only needs to collect the user's first 5 to 8 key decisions, such as the adoption or rejection of material recommendations and the confirmation or modification of contrast adjustments. The policy network can then be adapted to the user's personalized preferences through two gradient updates, with the learning rate set to 0.0005. User experiment results show that 5 to 8 key decisions are sufficient to characterize the user's basic design preferences, and a small learning rate can prevent overfitting. The entire process is completed within the interval between two rendering frames.
[0121] The processed output is a reinforcement learning agent and user response prediction network, which can output the optimal action based on the current state.
[0122] Since the effective frequency of physiological signals such as heart rate variability is below 0.4Hz, their sampling and calculation cannot be completed within a single frame of 16.7ms. Therefore, the calculation of implicit rewards and the corresponding strategy fine-tuning adopt an asynchronous low-frequency update mechanism with a period of ≥1 second, and are placed in the background queue for execution, without blocking the 16.7ms main rendering and explicit interaction closed-loop pipeline.
[0123] Step 3.3: Frame-adaptive computation graph dynamic pruning and cold-start prior injection: At the beginning of each frame, the range of the graph network that needs to be calculated in real time is dynamically reduced according to the user's current gaze and operation area to ensure rendering computing power; and an initial bias based on empirical knowledge is injected for the reinforcement learning agent to start in unfamiliar scenes to improve convergence efficiency.
[0124] Step 3.3.1: In order to resolve the computational resource conflict between the real-time inference process of the reinforcement learning agent and the virtual reality rendering pipeline, frame adaptive computation graph dynamic pruning is performed at the beginning of each rendering frame.
[0125] Based on the real-time extracted 3D coordinates of the gaze point from the MMF data stream output in step 1, the user's current eye gaze hotspot area is identified. This area is defined as a spherical region with a radius of 0.5 meters centered on the gaze point. Simultaneously, based on the spatial range of gesture operation, the user's current hand interaction area is identified. This area is defined as a spherical region with a radius of 0.3 meters centered on the location of the hand key points. The union of these two areas is the physical inference active area of the current frame. The setting of 0.5 meters and 0.3 meters is based on the following: the spatial range corresponding to the high-resolution visual field of the human eye's fovea is approximately 0.5 meters in radius, and the fine operation area of the hand is approximately 0.3 meters, such as the range that the fingertips can reach when the arms are extended forward.
[0126] Traverse all nodes in the coupled graph network constructed in step 3.1, and determine whether each node is located within the active region of the physical deduction, or whether it has a direct physical influence edge connection with nodes within the active region. Only nodes that meet one of the following two conditions are marked as active nodes: Condition 1, the distance between the node coordinates and any point within the interactive active region is less than 0.1 meters; Condition 2, there is an edge connection between the node and any active node and the edge weight value is greater than 0.1. Here, 0.1 meters is the grid node tolerance to prevent boundary nodes from being missed, and the edge weight of 0.1 is the weak coupling filtering threshold, and physical influences below this value can be ignored.
[0127] Only nodes marked as active and their associated edges are subjected to full computation graph unfolding and reinforcement learning inference computation; for all nodes in inactive regions, computation graph topology pruning is performed, that is, the edge connections between these nodes and active nodes are disconnected, and the computational load of these nodes is transferred to the background asynchronous queue of the central processor for execution, and they do not participate in the real-time rendering and real-time inference loop of the current frame.
[0128] Through this pruning process, the number of nodes actually participating in real-time simulation calculations usually does not exceed 20% of the total number of nodes. The freed-up computing resources are prioritized for allocation to the virtual reality rendering pipeline to ensure stable rendering frame rates.
[0129] Step 3.3.2: To address the inefficiency of cold start exploration in atypical design scenarios for reinforcement learning agents, a prior knowledge injection mechanism based on spatial typology is introduced.
[0130] Maintain a pattern library of historically common design scenarios. This library is constructed by extracting spatial syntactic features and performing cluster analysis on at least 1000 completed interior design projects. The spatial syntactic features include: spatial accessibility index, which is the average shortest path length from any point in the space to the exit; visual connectivity index, which is the number of grid nodes reachable from any point in the space within its visible range; and spatial depth distribution statistics, which is the number of levels in the spatial hierarchy. The specific calculation methods are as follows: for the accessibility index, 1000 points are randomly sampled in the spatial grid, the shortest path length from each point to the nearest exit is calculated, and the average is taken; for the visual connectivity index, 1000 points are randomly sampled in the spatial grid, the number of grid nodes within the visual range of each point is calculated, and the average is taken; for the spatial depth distribution statistics, a spatial hierarchy tree is constructed with the entrance as the root node, and the average, maximum, and standard deviation of the depth of all nodes are calculated.
[0131] When the spatial syntactic features of the current design scenario are detected to have a feature distance exceeding the preset threshold of 0.3 from all known pattern types in the pattern library, the current scenario is determined to be an atypical scenario. At this time, a pre-trained lightweight probability diffusion prior model is automatically loaded, in which 0.3 is the optimal threshold for distinguishing typical clusters from noise points through k-means clustering contour coefficient analysis.
[0132] The training method of the probability diffusion prior model is as follows: The training data comes from the architectural space design code document and the spatial feature distribution of 1000 excellent design cases; First, a batch of samples is sampled from the real spatial feature distribution, and then Gaussian noise is gradually added until the sample distribution becomes a standard Gaussian distribution, with the diffusion step number set to 1000 steps; A neural network based on the Transformer architecture is trained, with the input being the sample after noise addition and the current step number encoding, and the output being the predicted noise value, and the loss function being the mean square error between the predicted noise and the actual added noise; After training, the model can start from a random noise distribution and gradually remove noise to restore the spatial feature distribution that conforms to the architectural code and empirical rules.
[0133] When loading the probabilistic diffusion prior model, a random vector is sampled from the standard Gaussian distribution, and a reasonable initial distribution of spatial features is generated through the model's 1000-step inverse denoising process. The initial distribution of spatial features is mapped to the initial bias of the policy network of the reinforcement learning agent. That is, before any interaction, the agent already knows which types of parameter combinations are more likely to be reasonable under building codes and rules of thumb, thereby avoiding inefficient random exploration in atypical scenarios. This bias is injected in the form of a mean offset of the output actions of the policy network, and the offset is 30% of the mean of the output actions of the initial policy network. Experiments show that 30% prior guidance can reduce invalid exploration by about 60% while retaining 70% of the personalized exploration space.
[0134] The processed output includes a pruned coupled graph computational graph and an initial policy for the reinforcement learning agent injected with prior bias.
[0135] Step 4: Predictive Visualization and Real-time Closed-Loop Iteration: The optimal design parameter adjustment scheme derived in Step 3 is transformed into a semi-transparent model and confidence heatsink, which is then superimposed on the virtual reality scene. User feedback gestures are received, and database updates, physical field recalculation, experience storage, and policy network fine-tuning are completed within a single frame cycle to achieve real-time closed-loop iteration.
[0136] Step 4.1: Generation and presentation of non-intrusive visual recommendations: The parameter adjustment scheme is mapped to a semi-transparent preview model and a visual indicator bar representing confidence level, which is then overlaid and rendered into the virtual reality scene to provide users with intuitive decision-making basis.
[0137] The optimal design parameter modification scheme output by the reinforcement learning agent in step 3 is transformed into visual guidance information and overlaid and rendered onto the current virtual reality scene; the visual guidance information includes the following two forms:
[0138] Semi-transparent preset model: This model is generated as follows: When it is recommended to adjust the position, angle, or size of a design element, first copy the original 3D model of the design element, and set the material transparency of the copied model to 60% semi-transparent; then render the semi-transparent model into the virtual reality scene with the recommended target position, target angle, or target size; at the same time, use a highlighted outline to outline the edge of the semi-transparent model, and the outline color can be set to cyan; users can see the real scene elements behind the semi-transparent model through it, so as to intuitively compare the visual effect differences between the two schemes;
[0139] Confidence Heatmap: This heatmap is a gradient bar attached next to the recommended item, 10 cm long and 1 cm wide, with its color gradient from green to red and its hue value gradient from green to red. A triangular pointer is displayed on the heatmap, and the position of the pointer is determined by the predicted probability value output by the user response prediction network in step 3.2: when the probability value is 0, the pointer corresponds to the red end; when the probability value is 1, the pointer corresponds to the green end; and when the probability value is 0.5, the pointer corresponds to the middle yellow-green position. The heatmap allows users to quickly assess the expected matching degree of the recommended solution.
[0140] Step 4.2: User Interaction Response and System Closed-Loop Update: Identify the user's gesture response to the visual recommendation and asynchronously and in parallel complete a series of operations such as parameter persistence, physics field update, experience storage, and policy network fine-tuning within one rendering frame cycle.
[0141] Users respond to recommendations using natural gestures in virtual reality; there are three possible response types:
[0142] Acceptance Gesture: The user uses their hand to perform a grasping action on the semi-transparent preset model, holds the grasp for more than 0.5 seconds, and then releases it; this gesture is recognized as an acceptance command; among them, 0.5 seconds is the anti-shake time to distinguish between unintentional touch and confirmation intention;
[0143] Fine-tuning gesture: After the user uses their hand to perform a grasping action on the semi-transparent preset model, dragging, moving, or rotating it to a new position or angle, and then releasing the hand, the gesture is recognized as a fine-tuning command, and the parameter values after fine-tuning are recorded. After recording the parameter values after fine-tuning, the fine-tuning amplitude is calculated. The fine-tuning amplitude is the maximum absolute value of the relative change of each adjustment parameter. The amplitude is calculated as follows: For position adjustment, amplitude = |new position - original recommended position| / original recommended region side length, where the original recommended region side length refers to the maximum side length of the axis-aligned bounding box of the adjusted model at the recommended position, in meters; For angle adjustment, amplitude = |new angle - original recommended angle| / 90° For size adjustments, the magnitude is calculated as: |new size - original recommended size| / original recommended size. The maximum relative change among all parameters is taken as the final fine-tuning magnitude value. Based on the magnitude value determination, corresponding reward values are assigned according to the explicit reward signal rules defined in step 3.2: 0.5 is awarded when the adjustment magnitude is less than 5%, 0 is awarded when the adjustment magnitude is between 5% and 20%, and -1.0 is awarded when the adjustment magnitude exceeds 20%. When the user drags the semi-transparent model, the system calculates the displacement, rotation, and scaling changes of the model in real time and maps them to the relative adjustment of the corresponding design parameters. The mapping ratio is 1:1, that is, moving the model 1 meter corresponds to adjusting the parameters by 1 meter.
[0144] No-show gesture: The user performs a rapid waving motion with their hand on the semi-transparent preset model, with a waving speed > 1 m / s; this gesture is recognized as a no-show instruction; where 1 m / s is the kinematic characteristic of the human body's natural rejection motion.
[0145] Once the user adopts or fine-tunes and confirms, the following four update operations are completed within a single rendering frame cycle:
[0146] Parameter locking: Permanently lock the confirmed design parameter values and synchronously update them to the building information model database in the background; the update operation includes writing to the database transaction to ensure data persistence; this operation is executed asynchronously in the background, taking approximately 0.5 milliseconds in actual time, and does not occupy the main rendering thread;
[0147] Physics Update: Triggers the forward computation of the coupled graph network in step 3.1. Using the updated design parameters as input, the physics variable values of all nodes in the entire space are recalculated. The specific calculation method is as follows: For the illumination field, ray tracing calculation is performed; for the temperature field, the finite difference solution of the thermal diffusion equation is performed; for the sound field, sound ray tracing calculation is performed. These three calculations are accelerated in parallel by the graphics processor. Ray tracing uses the OptiX or DXR acceleration library, the thermal diffusion solution uses CUDA kernel function parallelization, and sound ray tracing adopts a simplified model based on the image source method. It is executed concurrently through CUDA streaming, with a total time of about twelve milliseconds.
[0148] Experience storage: The complete trajectory data of this interaction is stored in the experience replay pool. Each trajectory data includes: the state vector before the interaction (60 dimensions); the action vector recommended by the system (10 dimensions); the user's feedback signal value (-1, 0, 0.5, or 1); and the new state vector after the interaction (60 dimensions). The experience replay pool is implemented using a circular queue with a maximum capacity of 10,000 records, covering approximately 2 hours of interaction data, while maintaining controllable memory usage. When the queue is full, the oldest record is discarded according to the first-in, first-out principle. This operation is a memory-level operation, taking approximately 0.2 milliseconds. The experience replay pool uses a uniform random sampling strategy, sampling 16 unique trajectory data each time.
[0149] Policy fine-tuning: Triggering online fine-tuning of the reinforcement learning agent in step 3.2; the specific operation is as follows: randomly sample 16 historical trajectory data from the experience replay pool. The 16 data are calculated at the optimal granularity of the graphics processor tensor, and the gradient estimation is kept stable; calculate the reward function value defined in step 3.2, and perform a mini-batch gradient update on the policy network parameters through the backpropagation algorithm; in order to speed up the fine-tuning, only the parameters of the last two layers of the policy network are updated, with about 5000 parameters updated. The calculation time for one backpropagation is about 3-4 milliseconds, and the learning rate is set to 0.0005; the last two layers contain most of the policy information, and updating only these two layers can significantly reduce the amount of computation. The small learning rate can ensure the stability of the policy update.
[0150] The four update operations mentioned above are scheduled in parallel via an asynchronous pipeline, with the total time controlled within 16.7 milliseconds. This ensures that the entire process from displaying the recommended solution, the user's response, to completing parameter updates and model learning will not cause any perceptible delay to the user. When the hardware computing power is continuously overloaded, a degradation protection mechanism is automatically triggered, degrading the strategy fine-tuning and some physics calculations to be executed once every N frames, where N = 2 to 5, prioritizing ensuring that the main rendering thread's frame rate is ≥60fps. The value of N is set based on a dynamic threshold of graphics processor utilization: N+1 when utilization is >90% for 3 consecutive frames, with an upper limit of 5; N-1 when utilization is <70% for 5 consecutive frames, with a lower limit of 2. This degradation strategy will be displayed to the user in the system status bar as an icon, ensuring transparency in the user experience.
[0151] When a user performs a veto operation, no parameter locking or physical field update is performed, but the vetoed solution and its context state are still stored in the experience replay pool, triggering the counterfactual reasoning preference learning mechanism in step 3.2. That is, 20 counterfactual candidate solutions are generated, the prediction probability of each solution is calculated, the average prediction probability is obtained by averaging, and then an auxiliary reward is generated according to the transformation in step 3.2. This auxiliary reward is then combined with the real veto reward in the form of total reward = real veto reward + 0.5 × auxiliary reward, and used to update the policy network.
[0152] The system outputs updated design parameters, an expanded experience replay pool, and optimized policy network parameters after processing. The updated design parameters and physical field states influence the deduction of the next frame through a coupled graph network. The optimized policy network parameters are used for the next round of decision-making. User feedback signals are used for system learning in real time, forming a complete closed loop of perception, understanding, deduction, presentation, response, and learning, realizing real-time iterative interaction where what you see is what you get.
[0153] In collaborative indoor lighting and air conditioning design scenarios involving multiple users, this method first couples logical frames with rendering frames periodically, marking local logical frame numbers for sensor data from each terminal. These logical frame numbers are then sorted and assigned globally by a centralized event sorting server according to a causal consistency protocol. Dynamic compensation models are then established for the differential delays of eye movement, gestures, and speech. Finally, frame drift filtering and confidence-weighted fusion are used to output a multimodal fusion data stream strictly aligned with the rendering frame, thus resolving cross-terminal temporal misalignment and causal inversion issues. Next, based on this data stream, the current design micro-process stage is identified through temporal behavior pattern mining. A graph attention network is then used to parse high-level design semantics within the focal region of the gaze point, concatenating them to obtain a ten-dimensional design intent vector. Simultaneously, a correlation is constructed between the environmental physical field, user perception thresholds, and design... The coupled graph network, which calculates parameters, determines the active region for physical deduction based on the real-time gaze point coordinates and gesture space range, and performs frame-adaptive dynamic pruning. Features are extracted only from nodes within this region and concatenated with the design intent vector before being input into the reinforcement learning agent. This process deduces the optimal adjustment amounts for parameters such as illumination and air conditioning air supply angle. In atypical scenarios, a cold-start prior bias is also injected. Finally, the adjustment scheme is transformed into a semi-transparent preset state model and a confidence heatsink, which is overlaid and rendered onto the virtual scene. Users provide feedback through acceptance, fine-tuning, or rejection gestures. Within a single frame cycle, the system asynchronously and in parallel completes parameter locking, full-space physical field updates, experience storage, and policy network fine-tuning, forming a real-time closed-loop iteration. Finally, the updated design parameters are output, achieving seamless collaborative design interaction with precise causal alignment and efficient allocation of computing power.
[0154] Example 2:
[0155] Please see Figure 4 Based on Embodiment 1, this embodiment also provides an integrated design digital interaction system based on virtual reality, including:
[0156] The multimodal data synchronization processing module is used to receive raw interactive data collected by multiple terminal sensors, establish a global timing benchmark by logical frame marking and phase locking with the rendering cycle, perform causal consistency sorting according to operation dependencies, establish independent delay estimation models for each modality's physical delay characteristics for dynamic compensation, and trigger cubic spline interpolation resampling and cross-modal confidence weighted fusion based on historical rendering drift prediction values, outputting a multimodal fused data stream that is strictly aligned with the rendering frame;
[0157] The design intent recognition module is used to receive multimodal fusion data streams, mine frequent behavior patterns through an adaptive sliding window and match them with a preset micro-process dictionary to identify design stages. At the same time, it constructs a spatiotemporal tensor input graph attention network to parse high-level design semantics by defining the semantic calculation focus area with the gaze point as the center, and concatenates the stage one-hot encoding with the semantic probability vector to output the design intent vector.
[0158] The physical perception coupled graph inference module is used to receive the design intent vector and the gaze coordinates in the multimodal fusion data stream, construct a graph mesh that associates physical field variables, user perception thresholds and design parameters, determine the active area of physical inference based on the spatial information in the multimodal fusion data stream, and perform frame-level computation graph dynamic pruning. The feature vector extracted after pruning is concatenated with the design intent vector and input into the reinforcement learning agent that integrates counterfactual reasoning and meta-learning mechanisms, and outputs the optimal design parameter adjustment scheme.
[0159] The closed-loop interaction and asynchronous scheduling module is used to receive the optimal design parameter adjustment scheme and generate a semi-transparent preset state model and confidence heat bar to be superimposed on the virtual scene. At the same time, it receives and recognizes the user's gesture of acceptance, fine-tuning or rejection of the visualization results. After recognizing the user's feedback gesture, it performs parameter persistence, full recalculation of physics field, storage of experience trajectory and gradient update of policy network in parallel through asynchronous pipeline within a single frame period, and outputs the updated design parameters and optimized network state.
[0160] Furthermore, when performing causal consistency sorting, the multimodal data synchronization processing module determines the order of data packets based on semantic dependencies of operation type, spatial overlap dependencies, and inter-user dependencies. When a circular dependency is detected, the loop is broken according to the order of physical timestamps. A fixed delay superimposed noise model is used for eye-tracking data, a first-order linear dynamic system model is used for gesture data, and an endpoint detection and semantic parsing time accumulation model is used for speech data. When weighting cross-modal semantic confidence, the modal weights are dynamically allocated based on head rotation angular velocity, hand occlusion ratio, and environmental signal-to-noise ratio. Physiological signals are downsampled and buffered independently and do not participate in frame-by-frame weighted summation.
[0161] The design intent recognition module dynamically adjusts the time window length based on the previous recognition stage. When multiple frequent patterns match simultaneously, it selects the pattern with the highest frequency or the most recent time. Within the semantic computing focus area, it constructs graph neural network nodes and connects them with edges, stacks graph attention layers to calculate the association strength, and then outputs a probability vector after global average pooling and a fully connected layer, which is then concatenated with the one-hot encoded vector.
[0162] The physical perception coupled graph inference module adaptively sets the grid division precision according to the scene area when constructing the spatial discrete grid. Each grid node is associated with light intensity, thermal comfort index, sound pressure level, average dwell time, gaze time entropy, micro-posture adjustment amplitude, and preset adjustable design parameters. When performing frame-level computational graph dynamic pruning, only nodes located in the active area of physical inference or connected to it by high-weight physical influence edges are calculated. The reinforcement learning agent adopts a multi-layer fully connected neural network structure. When a rejection instruction is received, a counterfactual candidate solution with a single-dimensional offset combination is generated. The acceptance probability is calculated through the user response prediction network and converted into an auxiliary reward. In the offline stage, the strategy network parameters are pre-trained using a meta-learning framework. In the online stage, gradient updates are performed based on interactive decisions. In atypical scenarios, a probability diffusion prior model is loaded to generate an initial bias injection strategy network.
[0163] When the closed-loop interaction and asynchronous scheduling module recognizes the adoption gesture, it determines the holding time threshold. When recognizing the fine-tuning gesture, it calculates the absolute maximum value of the relative change of each parameter as the final adjustment range. When recognizing the rejection gesture, it determines the waving speed threshold. The asynchronous pipeline parallel scheduling includes writing the confirmed parameters into the building information model database, triggering the coupled graph network to perform forward calculation to update the physical field variables of the entire space, storing the interaction trajectory containing the previous and subsequent states and feedback signals into the circular queue experience pool, sampling historical trajectories from the experience pool to perform small-batch backpropagation on the policy network, and adjusting the policy fine-tuning and part of the physical field calculation to be performed once every preset number of frames when the graphics processor load is continuously overloaded. When performing the rejection operation, it skips parameter locking and physical field update and directly triggers the counterfactual preference learning mechanism.
[0164] In a collaborative indoor lighting and air conditioning design scenario, the system first receives eye-tracking, gesture, voice, and physiological signals collected from various terminals via a multimodal data synchronization processing module. After logical frame phase-locking, causal consistency sorting, dynamic latency compensation, and resampling fusion, it outputs a multimodal fused data stream strictly aligned with the rendering frame. The design intent recognition module identifies the current design micro-process stage based on this data stream and parses high-level design semantics, outputting a ten-dimensional design intent vector. The physical perception coupling graph deduction module determines the active area for physical deduction based on the gaze coordinates and gesture spatial range in the fused data stream, and correlates parameters such as illumination and thermal comfort. The coupled graph network performs frame-adaptive dynamic pruning, extracts active region features and concatenates them with design intent vectors, and inputs them into a reinforcement learning agent that integrates counterfactual reasoning and meta-learning to deduce optimal adjustment schemes such as lighting intensity and air conditioning air supply angle. The closed-loop interaction and asynchronous scheduling module transforms the adjustment schemes into a semi-transparent preview model and a confidence heatsink overlaid on the virtual scene, recognizes the user's acceptance, fine-tuning or rejection gestures, and completes parameter locking, full-space physical field recalculation, experience storage and policy network fine-tuning in parallel within a single frame period, outputting updated design parameters to feed back to the coupled graph network, realizing a real-time collaborative design closed loop where what you see is what you get.
[0165] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.
[0166] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0167] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A digital interaction method based on virtual reality integrated design, characterized by: Includes the following steps: The raw data of eye movement, gesture, voice and physiological signals from multiple user terminals are coupled with the virtual reality rendering frame periodically through logical frames, and causal sorting, dynamic delay compensation and frame drift filtering are performed to obtain a time-aligned multimodal fusion data stream. Based on multimodal fusion data stream, the current design micro-process stage is identified by temporal behavior pattern mining, and the high-level design semantics are analyzed by graph attention network. The stage identification results and semantic classification results are spliced together to obtain the design intent vector. A coupled graph network containing environmental physics, user perception thresholds, and design parameters is constructed. The active region of the current physical deduction is determined based on the gaze coordinates and gesture space range in the multimodal fusion data stream. The physics variables and user perception threshold variables of the nodes in the active region are extracted as feature vectors. The feature vectors are concatenated with the design intent vector and input into the reinforcement learning agent to deduce the adjustment amount of the adjustable design parameters. Dynamic pruning is performed on the coupled graph network at the beginning of each frame, and cold start prior bias is injected into the reinforcement learning agent in atypical scenarios to obtain the optimal design parameter adjustment scheme. The optimal design parameter adjustment scheme is transformed into visual guidance information and overlaid on the virtual reality scene. It receives user feedback gestures and completes parameter locking, physical field updating, experience storage and policy network fine-tuning within a single frame cycle, and outputs the updated design parameters.
2. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The causal ordering process includes: receiving the original data packets with local logical frame numbers and physical timestamps uploaded by each terminal through a centralized event ordering server; sorting them by physical timestamps; determining the order of data packets according to operation type semantic dependencies, spatial overlap dependencies, and inter-user dependency rules; and assigning global logical frame numbers. The dynamic latency compensation process includes: establishing independent latency estimation models for eye movement, gestures, and speech respectively; calculating the predicted latency in real time based on the deviation between the actual and theoretical arrival times of the most recent data packets; and subtracting the predicted latency from the timestamp of each data packet to align the time axis of the multimodal data.
3. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The frame drift filtering process includes: maintaining a historical offset queue, calculating a weighted moving average to obtain the predicted drift amount, and triggering cubic spline interpolation resampling when the absolute value of the predicted drift amount reaches a preset drift amount threshold to generate virtual data points aligned with the frame period. The process of obtaining a time-aligned multimodal fusion data stream further includes: dynamically calculating the confidence coefficient of each modality based on the head rotation angular velocity, hand occlusion ratio, and environmental signal-to-noise ratio; and normalizing and weighting the eye movement, gesture, and speech data within the same global logical frame according to the confidence coefficient to obtain the multimodal fusion data stream.
4. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The process of identifying the current design microprocess stage through temporal behavior pattern mining includes: mapping the multimodal fusion data stream into an event tuple sequence, setting an adaptive time window, searching for recurring behavior pattern fragments, marking fragments whose frequency exceeds the preset minimum support as frequent patterns, matching frequent patterns with a pre-constructed design microprocess dictionary, and outputting a one-hot encoded vector as the microprocess stage identification result.
5. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The process of parsing high-level design semantics through a graph attention network includes: defining the semantic computation focus region centered on the gaze point; organizing the multimodal features of each frame within the semantic computation focus region into a three-dimensional tensor of time, space, and modality; inputting the three-dimensional tensor into a pre-trained three-layer graph attention network; and outputting a probability vector as the semantic classification result after global average pooling.
6. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The construction process of the coupled graph network includes: discretizing the virtual indoor space into a parameterized grid, associating each grid node with light intensity, thermal comfort index and sound pressure level as physical field variables, associating average dwell time, gaze time entropy value and micro-posture adjustment amplitude as user perception threshold variables, associating preset adjustable design parameters, and establishing connection edges between nodes according to spatial adjacency and physical influence radius; The dynamic pruning process includes: determining the active area of physical inference by the union of the gaze hotspot area and the gesture interaction area; performing complete computation graph and reinforcement learning inference only on active nodes located in the active area of physical inference or connected to it by a high-weight physical influence edge; and transferring the computational load of inactive nodes to the background asynchronous queue.
7. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The process of deriving the adjustment amount of the adjustable design parameters includes: concatenating the feature vector and the design intent vector and inputting them into a three-layer fully connected policy network, and outputting continuous action values as the adjustment amount of the adjustable design parameters; The reinforcement learning agent integrates counterfactual reasoning and meta-learning mechanisms, including: when a user rejects a recommended solution, generating offset combinations on each parameter dimension of the rejected solution to form counterfactual candidate solutions, predicting the acceptance probability through a user response prediction network and converting it into an auxiliary reward, updating the policy network after weighted fusion with real feedback, and using a model-independent meta-learning framework to pre-train the reinforcement learning agent in the offline stage, and performing gradient updates through a small amount of online interactive decision-making to quickly adapt to personalized preferences.
8. The integrated design digital interaction method based on virtual reality according to claim 1, characterized in that: The process of converting the optimal design parameter adjustment scheme into visual guidance information and overlaying it onto the virtual reality scene includes: generating a semi-transparent preset state model with preset transparency and a confidence heat bar representing the acceptance probability, and overlaying it onto the current virtual scene; The feedback gestures include acceptance gestures, fine-tuning gestures, and rejection gestures; The process of completing parameter locking, physical field updating, experience storage, and policy network fine-tuning within a single frame period includes: The design parameters after user feedback gesture confirmation are written into the database, and the coupled graph network is triggered to recalculate the physical field of the entire space. Store the interaction trajectory, which includes the current state, recommended action, feedback signal and new state, into the experience replay pool; Historical trajectories are sampled from the experience replay pool, and mini-batch gradient updates are performed on the policy network of the reinforcement learning agent through backpropagation. Parameter locking, physics field update, experience storage, and policy network fine-tuning are scheduled in parallel through an asynchronous pipeline, with the total time controlled within one frame period. When a veto gesture is executed, parameter locking and physics field update are not performed. Instead, the vetoed scheme is stored in the experience replay pool and counterfactual reasoning preference learning is triggered.
9. A digital interactive system based on virtual reality, characterized in that: include: The multimodal data synchronization processing module processes raw interactive data from multiple terminals by performing causal sorting, dynamic delay compensation, and frame drift filtering through periodic coupling of logical frames and rendering frames, and outputs a time-aligned multimodal fusion data stream. The design intent recognition module, based on multimodal fusion data stream, identifies the design stage through temporal behavior pattern mining, and parses the design semantics through graph attention network to output design intent vector; The physical perception inference module constructs a graph network that includes physical fields, user perception, and design parameters. It determines the inference focus based on multimodal fusion data streams and design intent vectors, and outputs the optimal design parameter adjustment scheme through a reinforcement learning agent that integrates counterfactual reasoning and meta-learning mechanisms. The real-time closed-loop update module transforms the adjustment scheme into visual information and renders it, receives user feedback gestures, and asynchronously triggers parameter updates, physical field recalculation, experience storage, and policy network fine-tuning within a single frame cycle to complete the closed-loop iteration.
10. The integrated design digital interaction system based on virtual reality according to claim 9, characterized in that: When performing causal consistency sorting, the multimodal data synchronization processing module determines the order based on the type semantics between operations, spatial overlap, and user dependencies, and establishes delay estimation models for eye movement, gesture, and voice modalities for dynamic compensation. The design intent recognition module identifies the design phase by mining frequent behavioral patterns in event sequences, and aggregates spatiotemporal features through a graph attention network to analyze semantics within the semantic focus area centered on the gaze point. When constructing the graph network, the physical perception inference module associates physical field variables, user perception variables, and design parameter variables with spatial grid nodes, and performs dynamic pruning on the graph network based on user gaze and interaction areas in each frame to extract inference features. The policy network of the reinforcement learning agent outputs parameter adjustments based on inference features and design intent vectors. The reward function integrates explicit user feedback and implicit physiological comfort feedback, and integrates counterfactual reasoning and meta-learning mechanisms to adapt to personalized preferences. The real-time closed-loop update module identifies the user's acceptance, fine-tuning, or rejection gestures, and performs physiological signal processing and reward calculation based on them in a low-frequency asynchronous process through asynchronous pipeline parallel scheduling parameter persistence, physical field update, experience replay storage, and online learning operations of the policy network.
