VR scene control method and system

Abstract

The invention discloses a VR scene control method and system, and belongs to the technical field of VR scene control, and the method specifically comprises the steps: deploying a global time controller, generating a continuously increasing global timestamp at a fixed frequency, and transmitting the global timestamp to a scene rendering, audio processing and touch driving module; the scene interaction monitoring unit continuously collects user body movement and virtual object movement state data, analyzes interaction event types and characteristic parameters and then binds the interaction event types and the characteristic parameters with a current global timestamp to form an interaction information packet with the timestamp; a multi-modal feedback instruction generation unit receives the information packet, and generates three types of instructions carrying the same global timestamp based on the interaction event type, the characteristic parameters and the bound global timestamp; the feedback instruction distribution unit forwards the three types of instructions to corresponding modules at corresponding moments according to global timestamps carried by the instructions; and after executing feedback, each module records delay and sends the delay to the global time controller, so that the actual execution moments of three types of instructions in the next round are consistent.

Description

Technical Field

[0001] This invention relates to the field of VR scene control technology, and specifically to a VR scene control method and system. Background Technology

[0002] Current virtual reality (VR) scene control technology has formed a basic architecture consisting of four subsystems working collaboratively: visual rendering, motion tracking, audio spatialization, and haptic feedback. Visually, it relies on high-refresh-rate stereoscopic displays and lighting / shading algorithms to present realistic scene deformations and object displacements; motion tracking acquires user posture and interaction actions through headsets, controllers, inertial measurement units (IMUs), and optical / magnetic positioning systems; the audio subsystem uses spatial audio rendering, echo modeling, and event-driven sound engines to enhance auditory immersion; haptic feedback uses hardware such as vibration motors, force feedback drivers, haptic gloves, or subcutaneous arrays, through a dedicated haptic renderer, to map physical quantities into impulses, torques, or vibration patterns. To achieve complex interactions, modern VR systems also introduce physics engines (for collision detection and mechanical response), middleware communication buses (for synchronizing visual / haptic / audio events), and timestamp-based event queues and prediction / interpolation mechanisms. In multi-user and remote collaboration scenarios, network synchronization and bandwidth management, as well as client-based prediction-based error correction strategies, have also become important components of system design. Overall, current technologies offer a high degree of diversity and scalability in terms of hardware types and rendering algorithms, providing the necessary conditions for highly interactive and immersive experiences.

[0003] However, existing technologies still have significant shortcomings in cross-sensory coordination and real-time consistency, directly impacting user immersion and interaction reliability. The lack of spatiotemporal synchronization between tactile and visual / auditory feedback is a major issue: even with tactile devices, the timing, intensity, and duration of tactile feedback often fail to precisely match visual or auditory effects when users interact in virtual scenes. For example, delayed tactile sensations of virtual collisions or force changes that are out of sync with visual displays significantly diminish the realism and credibility of the interaction. Furthermore, the causes of these phenomena include inconsistencies between subsystem update rates and processing chains, insufficient coupling between the physics engine and the tactile renderer, and inaccuracies in event timestamps and prediction mechanisms. Other problems include limited tactile resolution, a lack of a unified model for mapping tactile effects to auditory / visual semantics, calibration errors in sensors and actuators, and inconsistencies in remote feedback caused by jitter and packet loss in network transmission or distributed systems. These deficiencies not only reduce the credibility of the scene but can also lead to user misjudgments, decreased operational efficiency, and even motion sickness, thus becoming key technical bottlenecks that urgently need to be overcome through systematic control methods and collaborative rendering strategies. Summary of the Invention

[0004] The purpose of this invention is to provide a control method and system for VR scenes, solving the following technical problem: lack of spatiotemporal synchronization between tactile and visual / auditory feedback: even when equipped with tactile devices, when users interact in virtual scenes, the triggering timing, intensity, and duration of tactile feedback often cannot be accurately matched with visual or auditory effects.

[0005] The objective of this invention can be achieved through the following technical solutions: A method for controlling a VR scene includes the following steps: S1: Deploy a global time controller, which generates continuously increasing global timestamps at a fixed frequency and sends the global timestamps to the scene rendering module, audio processing module and haptic drive module respectively; S2: The scene interaction monitoring unit continuously collects user body movement data and virtual object motion state data, analyzes the interaction event type through kinematic calculations, and extracts the corresponding feature parameters of the interaction event; the scene interaction monitoring unit binds the interaction event type and feature parameters with the current global timestamp to form a timestamped interaction information package; S3: The multimodal feedback instruction generation unit receives the interaction information packet and generates three types of instructions based on the interaction event type and feature parameters, using the bound global timestamp as a reference: haptic feedback instruction, visual rendering instruction, and auditory output instruction; all three types of instructions carry the same global timestamp. S4: The feedback instruction distribution unit forwards the three types of instructions to the haptic drive module, scene rendering module, and audio processing module respectively at the corresponding time according to the global timestamp carried by the instruction; S5: Each module executes feedback based on the instruction timing parameters, records the feedback execution delay value and sends it to the global time controller. The global time controller adjusts the generation offset of the next global timestamp accordingly, so that the actual execution time of the three types of instructions in the next round is consistent.

[0006] As a further aspect of the present invention: In step S2, the scene interaction monitoring unit continuously collects user limb movement data and virtual object motion state data, and the process of parsing the interaction event type through kinematic calculation is as follows: Deploy a user limb movement acquisition submodule and a virtual object motion state acquisition submodule to collect motion data of key skeletal nodes of the user's limbs and motion state data of virtual objects, respectively. The collected data is preprocessed, the time axis of the two types of data is calibrated based on the global timestamp, the spatial coordinate system is unified through coordinate transformation, and the scale difference between the virtual environment and the physical collection space is removed. A user limb kinematics model and a virtual object kinematics model are established respectively. The preprocessed data is mapped to a limb kinematic chain and an object surface dynamic mesh model to form a continuous kinematic state sequence indexed by timestamps. Spatial indexing is performed on the limb kinematic chain and the dynamic mesh model of the object surface. The shortest spatial distance between the key points on the surface of the two is calculated in real time. When the distance is less than a preset threshold, it is determined to be a potential interaction state. The timestamp of the initial contact moment, the coordinates of the contact point, and the identification of the limb and object involved in the interaction are recorded. In the potential interaction state, the relative motion parameters at the contact point are analyzed, integrated into an interaction feature set, and bound to the contact time timestamp; An interactive event classifier and a basic event type library including touch, grab, push, hit, and friction are constructed. The classifier takes parameters from the interactive feature set as input and matches a unique interactive event type through rule reasoning.

[0007] As a further aspect of the present invention: in step S2, feature parameters corresponding to the interactive event are extracted simultaneously, including the interactive position coordinates, contact force value, and relative motion speed.

[0008] As a further aspect of the present invention: in step S3, the process of generating three types of instructions respectively: haptic feedback instructions, visual rendering instructions, and auditory output instructions is as follows: The multimodal feedback instruction generation unit receives the interaction information packet with timestamp, parses and extracts the interaction event type, feature parameters and global timestamp, and decomposes the feature parameters into three-dimensional components of the interaction position coordinates, scalar data of the contact force value and vector data of the relative motion velocity, and stores them separately. Establish a mapping rule library categorized by interaction event type, and store the parameter generation logic corresponding to each type of event, including the association between tactile, visual, and auditory parameters and feature parameters, as well as the duration matching rules; Each event calls the rule base logic for the corresponding interaction event type, calculates its own instruction parameters based on the contact force value and relative motion speed feature parameters, sets the start time based on the global timestamp and the offset relationship is fixed, and integrates them into their respective instruction core parameters. The three types of instructions are bound to the original global timestamp, and the time consistency and timing difference are verified. Once verified, a set of three types of instructions carrying the same global timestamp is formed.

[0009] As a further aspect of the present invention: in step S5, the specific method of integrating them into their respective instruction core parameters is as follows: The logic of the current event type in the rule base is called to calculate the intensity level based on the contact force value, and the duration of the event type is matched with the relative motion speed and the event type. The trigger time is set as a global timestamp plus a preset tactile offset, and integrated into tactile feedback instruction parameters. The logic of the current event type in the rule base is called to calculate the deformation amplitude parameter based on the contact force value. The relative motion speed is combined with the event type to match the duration. The deformation start time is set as a global timestamp plus a preset visual offset and integrated into the visual rendering instruction parameters. The logic of the current event type in the rule base is called to calculate the sound effect frequency parameter based on the relative motion speed modulus. The duration of the event is matched with the contact force value and the event type. The playback start time is set as a global timestamp plus a preset auditory offset and integrated into auditory output command parameters.

[0010] As a further aspect of the present invention: in step S3, the tactile feedback instruction includes a trigger time, intensity level, and duration; the visual rendering instruction includes a deformation start time, deformation amplitude parameters, and deformation duration; and the auditory output instruction includes a playback start time, sound effect frequency parameters, and sound effect duration.

[0011] As a further aspect of the present invention: In step S5, each module performs feedback according to the instruction timing parameter, records the feedback execution delay value, and sends it to the global time controller. The process by which the global time controller adjusts the offset for the next global timestamp generation is as follows: Each module deployment time recording subunit records the startup time when receiving instructions, and calculates the feedback execution delay value and stores it in the local delay cache when actually executing feedback; Each module binds the feedback execution delay value with the corresponding global timestamp to form a delay information packet, which is then sent to the global time controller through a preset sub-channel; The global time controller verifies the consistency of timestamps and the validity of delay values ​​in delay information packets, and categorizes and stores the delay values ​​that have passed the verification. The global time controller uses the maximum delay value as a reference to calculate the compensation offset of the other two modules, and sets the compensation offset of the reference module to zero. The global time controller updates the timestamp generator parameters and associates the compensation offset to ensure that the next global timestamp is corrected so that the actual execution time of the three types of instructions in the next round is consistent.

[0012] A control system for a VR scene includes: The global time controller generates continuously increasing global timestamps at a fixed frequency and sends the global timestamps to the scene rendering module, audio processing module, and haptic drive module respectively. The scene interaction monitoring unit is used to continuously collect user body movement data and virtual object motion state data, analyze the interaction event type through kinematic calculations, and extract the corresponding feature parameters of the interaction event. The scene interaction monitoring unit binds the interaction event type and feature parameters with the current global timestamp to form a timestamped interaction information package. The multimodal feedback instruction generation unit is used to receive interactive information packets and generate three types of instructions based on the interactive event type and feature parameters, with the bound global timestamp as the reference: haptic feedback instructions, visual rendering instructions, and auditory output instructions. The feedback instruction distribution unit is used to forward the three types of instructions to the haptic drive module, scene rendering module, and audio processing module at the corresponding time according to the global timestamp carried by the instruction. The offset feedback unit is used by each module to perform feedback based on the instruction timing parameters, record the feedback execution delay value and send it to the global time controller. The global time controller adjusts the generation offset of the next global timestamp accordingly to ensure that the actual execution time of the three types of instructions in the next round is consistent.

[0013] The beneficial effects of this invention are: This invention fundamentally solves the time misalignment problem of multimodal feedback by constructing a global time base and a command flow mechanism bound to timestamps. The global time controller unifies the timeline of scene rendering, audio processing, and haptic drive, ensuring that the capture of interactive events and the generation and distribution of feedback commands are all based on the same time base; the interactive information packets with timestamps and synchronous feedback commands strictly align the trigger time of haptic feedback, the start time of visual deformation, and the starting point of sound effect playback, effectively eliminating the delay or advance caused by independent timing of each module.

[0014] Meanwhile, the solution achieves continuous optimization of feedback effects through a dynamic calibration mechanism. Each module provides real-time feedback on execution delays, and the global time controller adjusts the timestamp offset for the next round accordingly. This not only matches the correspondence between parameters such as tactile intensity, visual deformation amplitude, and sound effect frequency, but also dynamically compensates for deviations caused by differences in hardware response, ensuring that multimodal feedback remains consistent in dimensions such as intensity and duration, significantly improving the realism and immersion of user perception in virtual interaction. Attached Figure Description

[0015] The invention will now be further described with reference to the accompanying drawings.

[0016] Figure 1 This is a flowchart illustrating a VR scene control method according to the present invention.

[0017] Figure 2 This is a schematic diagram of the structure of a VR scene control system according to the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 As shown, the present invention is a VR scene control method, comprising: S1: Deploy a global time controller, which generates continuously increasing global timestamps at a fixed frequency and sends the global timestamps to the scene rendering module, audio processing module, and haptic drive module respectively, so that the internal timelines of the three modules are consistent with the global timestamps.

[0020] Deploying a global time controller involves setting up a core unit within the system specifically responsible for time management. This unit is independent of the scene rendering, audio processing, and haptic feedback modules, serving as the baseline source for system-wide time synchronization. This global time controller generates continuously increasing global timestamps at a fixed frequency. The fixed frequency means that time counting progresses stably according to a preset system time interval, ensuring a consistent pace and preventing sudden changes. Continuously increasing timestamps mean that each new timestamp is larger than the previous one, reflecting the unidirectional flow of time and providing a unique and ordered identifier for each point in time. The principle is that the fixed frequency ensures the stability of time measurement, while the continuous increasing timestamp clearly distinguishes the sequence of different moments, providing a unified time reference standard for all modules. Subsequently, the global time controller transmits the generated global timestamps to the scene rendering, audio processing, and haptic feedback modules via internal system communication paths. This transmission method ensures that each module receives the same time information promptly, avoiding discrepancies caused by modules calculating time independently through direct information transmission. After receiving the global timestamp, the three modules will compare and adjust their internal timelines with the global timestamp. If there is a difference between the internal time of a module and the global timestamp, the module will update its time count according to the global timestamp. Subsequently, the module will also update its internal time according to the progress of the global timestamp. The principle is to make the time measurement of the three modules based on the same benchmark, thereby ensuring their coordination in the time dimension and avoiding operational or feedback errors caused by their time being out of sync.

[0021] S2: The scene interaction monitoring unit continuously collects user limb movement data and virtual object motion state data, analyzes the interaction event type through kinematic calculations, and extracts the corresponding feature parameters of the interaction event, including interaction position coordinates, contact force value, and relative motion speed. The scene interaction monitoring unit binds the interaction event type and feature parameters with the current global timestamp to form a timestamped interaction information package.

[0022] In step S2, the scene interaction monitoring unit continuously collects user limb movement data and virtual object motion state data, and the process of parsing the interaction event type through kinematic calculations is as follows: A multi-source data acquisition channel is constructed. Specifically, two dedicated data acquisition sections are set up in the scene interaction monitoring unit. The user limb movement acquisition section is equipped with both an inertial measurement unit (IMU) and an optical motion capture unit. The IMU can sense the acceleration and rotation state of the limb during movement, while the optical motion capture unit obtains the position of the limb in three-dimensional space by capturing the optical markers of key skeletal nodes. The two are combined to collect the three-dimensional position information of the key skeletal nodes of the user's limb, the angle change sequence of each joint, and the acceleration data during the movement in real time. The virtual object motion state acquisition section is connected to the core engine of the virtual environment and directly obtains the coordinates of the center of mass of the virtual object in three-dimensional space, the positional changes of each vertex on the object's surface, the angular velocity of the object during rotation, and rigid body dynamics data describing the object's motion from the engine. When collecting data, both acquisition components will associate the timestamps generated by the global time controller with each set of acquired data and temporarily store them. The principle behind this is that inertial measurement equipment and optical motion capture devices acquire limb information from different physical dimensions, and the combination of these information can comprehensively reflect the limb movement state. Direct connection with the virtual environment engine ensures that the most original and accurate motion data of virtual objects is obtained. Marking the data with global timestamps allows data from different sources to have a unified reference standard in the time dimension, which facilitates subsequent time comparison and analysis.

[0023] Next, the data preprocessing and spatiotemporal alignment process is initiated. First, the data cleaning section processes the collected raw data to remove interference information. For user limb data, Kalman filtering is used. This method continuously predicts the possible values ​​of the data and corrects them by combining them with newly collected data, thereby eliminating the drift error generated by the sensor during long-term operation. This is because sensor drift causes the measured value to gradually deviate from the true value, and the combination of prediction and correction can effectively offset this deviation. For virtual object data, moving average filtering is used. This method replaces a single data point by calculating the average of a certain number of adjacent data points, eliminating the jitter generated by the engine during real-time calculation. This is because the instantaneous calculation of the engine may have slight jumps due to fluctuations in computing load, and averaging can smooth out these fluctuations. After data cleaning is completed, the spatiotemporal alignment process is initiated. Based on the global timestamp, user limb data and virtual object data are calibrated in time, so that data with the same timestamp correspond to the same moment. At the same time, through coordinate transformation, the two types of data are converted into the same spatial coordinate system, eliminating the difference in size ratio between the virtual environment space and the actual physical acquisition space. The principle is that time calibration ensures the synchronization of data in time, while the same coordinate system provides a basis for spatial location comparison, avoiding position judgment errors caused by different time or spatial references.

[0024] Next, a kinematic representation model is constructed. When building the user's limb kinematic model, the preprocessed bone node coordinates and joint angle sequences are converted into limb motion chains. This involves connecting bone nodes into a chain-like structure based on the connections between bones, and then calculating the motion path and rate of posture change for each limb segment at each moment. Since limb movement is accomplished collaboratively by connected limb segments, the motion chain accurately reflects this collaborative relationship, while the motion trajectory and rate of posture change describe the dynamic characteristics of the movement. When building the virtual object kinematic model, a dynamic mesh model of the object's surface is constructed based on the object's centroid coordinates, vertex displacements, and rotation parameters. The mesh structure formed by connecting vertices on the object's surface changes with the object's movement, generating a motion vector field for each point on the object's surface—that is, the direction and velocity of each point's movement. Because the motion and deformation of a virtual object are reflected in the positional changes of its surface points, the dynamic mesh model and motion vector field visually demonstrate these changes. Both types of models use timestamps as indexes to form a continuous kinematic state sequence in chronological order. This is done because timestamps clearly identify the moment corresponding to each state, allowing the sequence to reflect the motion changes over time.

[0025] Then, dynamic collision detection and interaction determination are initiated. The collision detection module uses an octree space partitioning algorithm to divide the entire space into eight subspaces. Each subspace is then partitioned in the same way to form a hierarchical structure. This is used to index the position of the user's limb motion chain and the virtual object's dynamic mesh model in space. This indexing method can quickly locate the approximate area of ​​the two in space and reduce unnecessary computation because it is not necessary to compare all points in the entire space one by one.

[0026] Based on this, the shortest spatial distance between key points on the user's limb surface and the grid cells on the virtual object surface is calculated in real time. When this distance is less than a preset threshold, it is determined that an interaction may have occurred. At the same time, the timestamp of the initial contact, the coordinates of the contact point in three-dimensional space, and the identifiers of the limb segment and the object grid cell involved in the contact are recorded. The principle is that when the two are close enough, they can be regarded as having made contact in the virtual scene. Recording this information is for subsequent detailed analysis of the specific situation of the interaction.

[0027] After determining a potential interaction state, the kinematic calculation part analyzes the relative motion parameters at the contact point based on the two types of kinematic models previously established: calculating the relative velocity of the user's limb contact point and the virtual object contact point in linear and rotational motion; deriving the direction vector of the limb contact force by the rate of change of joint angle, since the change of joint angle is the result of force action, and the rate of change can reflect the direction of the force; calculating the deformation trend of the contact area based on the displacement change of the mesh unit on the surface of the virtual object, since contact will cause the object surface to deform, and the displacement change can reflect the degree and direction of this deformation.

[0028] These parameters are organized and combined into a set of interaction features and associated with the timestamp of the contact moment. This is done because these parameters collectively describe the key features of the interaction, and associating them with the timestamp clarifies the specific moment corresponding to these features.

[0029] Finally, the interaction event types are classified, and an interaction event classifier is constructed. A pre-set library of five basic interaction event types—touch, grasp, push, hit, and friction—serves as a reference standard for classification. The classifier takes the relative velocity threshold, the stability of the contact force direction, and the degree of deformation of the contact area from the interaction feature set as input parameters, and matches them through a rule-based reasoning mechanism: when the relative velocity is below the first threshold and the contact force direction remains stable, it is determined to be a touch, because touch is usually a slow and stable contact; when the degree of deformation of the contact area exceeds the set threshold and the relative velocity direction points inward, it is determined to be a grasp, because grasping causes obvious deformation of the object's surface and movement of the limb inward; and so on. Based on different parameter combinations and feature matching of various events, a unique corresponding interaction event type is output, and this type is associated with the corresponding timestamp and feature parameters. The principle is that different interaction events exhibit different parameter characteristics; rule-based reasoning can accurately identify the specific interaction type, and associating the timestamp and feature parameters forms complete interaction event information.

[0030] S3: The multimodal feedback instruction generation unit receives the timestamped interaction information packet and generates three types of instructions based on the interaction event type and feature parameters, using the bound global timestamp as a reference: haptic feedback instructions include the trigger time, intensity level, and duration; visual rendering instructions include the deformation start time, deformation amplitude parameters, and deformation duration; and auditory output instructions include the playback start time, sound effect frequency parameters, and sound effect duration. All three types of instructions carry the same global timestamp.

[0031] In step S3, the process of generating the three types of instructions is as follows: After receiving an interaction information packet with a timestamp, the multimodal feedback instruction generation unit activates the information parsing submodule to decompose the information packet, separating the specific type of the interaction event, various parameters describing the interaction characteristics, and a global timestamp bound to this information. The characteristic parameters are further decomposed into the components of the interaction position coordinates in three-dimensional space, scalar data describing the magnitude of the contact force, and vector data reflecting the direction and magnitude of the relative motion velocity. These three types of data are stored in their respective parameter cache areas. The principle behind this is that parsing the information packet is to obtain the basic information required to generate feedback instructions. Decomposing the characteristic parameters into different types and storing them in categories is because different feedback types (tactile, visual, auditory) require different parameters, and categorized storage facilitates subsequent instruction generation subunits to quickly and accurately obtain the required data.

[0032] Next, an event-instruction mapping rule base is constructed. This rule base categorizes and stores information according to the type of interactive event (including touch, grab, push, hit, and friction). For each type of event, corresponding parameter generation logic is pre-defined. Specifically, this includes the mapping method of how the intensity level of tactile feedback corresponds to the contact force value, the coefficient of how the deformation amplitude of visual feedback is related to the contact force value, the method of how the frequency parameter of auditory feedback is converted from relative motion speed, and the matching relationship between the duration of various instructions and the event type. The principle is that different interactive events will trigger different feedback requirements. For example, touch and hit require different tactile intensity and visual deformation. By establishing this correspondence, the rule base ensures that the generated feedback instructions can accurately match the characteristics of the interactive event.

[0033] Next, the haptic instruction generation subunit is activated. This subunit calls the logic in the rule base corresponding to the current interaction event type: using the contact force value as input, it calculates the intensity level of the haptic feedback through a preset mapping method. Since a greater contact force usually requires a correspondingly stronger haptic feedback, this mapping ensures that the feedback intensity matches the actual contact force. Based on the relative motion speed and the event type, it finds the corresponding duration from the rule base, as motion speed and event type affect the duration of the interaction; for example, rapid friction may last shorter than slow touch. The trigger time of the haptic feedback is set to a global timestamp plus a pre-set haptic response baseline offset. This offset is pre-configured based on the average delay recorded during past feedback processes. The purpose is to compensate for potential delays by setting the trigger time in advance, making the actual trigger time closer to the expectation. Finally, the calculated intensity level, duration, and trigger time are integrated to form the core parameters of the haptic feedback instruction.

[0034] Simultaneously, the visual instruction generation subunit is activated, invoking the logic corresponding to the current interaction event type in the rule base: taking the contact force value as input, the amplitude parameter of visual deformation is calculated through a preset correlation coefficient, because the greater the contact force, the more obvious the deformation of the virtual object should usually be, and the correlation coefficient can reflect this correlation; based on the relative motion speed and event type, the duration of deformation is matched from the rule base, because motion speed and event type affect the duration of deformation, for example, deformation caused by pushing may last longer than touching; the start time of deformation is set to a global timestamp plus a preset visual response baseline offset, and the difference between this offset and the tactile response baseline offset remains fixed. This is to ensure that the order or time interval between visual feedback and tactile feedback remains stable, without any sudden changes, thus enhancing the coordination between the two types of feedback. Finally, the deformation amplitude parameter, deformation duration, and deformation start time are integrated into the core parameters of the visual rendering instruction.

[0035] Next, the auditory instruction generation subunit is activated, calling the logic corresponding to the current interaction event type in the rule base: taking the magnitude of the relative motion speed (i.e., the speed magnitude) as input, the frequency parameters of the sound effect are calculated through a preset conversion method, because the magnitude of the relative motion speed affects the characteristics of the sound; for example, the faster the speed, the higher the corresponding sound effect frequency may be, which can more intuitively reflect the motion state. Based on the contact force value and event type, the duration of the sound effect is matched from the rule base, because the contact force and event type affect the duration of the sound; for example, hitting may produce a short but loud sound, while friction may produce a continuous sound. The playback start time is set to a global timestamp plus a preset auditory response baseline offset, and the difference between this offset and the tactile and visual response baseline offsets remains fixed. This is to ensure that auditory feedback is coordinated with tactile and visual feedback in time, forming a unified multi-sensory experience, and preventing any feedback from being significantly delayed or advanced. Then, the sound effect frequency parameters, sound effect duration, and playback start time are integrated into the core parameters of the auditory output instruction.

[0036] Finally, the instruction integration submodule is activated, binding the generated haptic feedback instructions, visual rendering instructions, and auditory output instructions to the global timestamp in the original interaction information package, ensuring that all three types of instructions can be traced back to the time of the corresponding interaction event. At the same time, the time consistency verification subunit is activated to check whether the trigger time, deformation start time, and playback start time of these three types of instructions are all generated based on the same global timestamp, and whether the time interval between them meets the preset limit. The principle behind this is that binding to the same timestamp ensures that the three types of instructions correspond to the same interaction event, while verifying the timing difference is to ensure the time coordination of the three types of feedback and avoid obvious asynchrony, thereby ensuring that the multimodal feedback received by the user is unified and coherent. After the verification is passed, a set of three types of instructions carrying the same global timestamp is formed.

[0037] S4: The feedback instruction distribution unit receives three types of instructions output by the multimodal feedback instruction generation unit. According to the global timestamp carried by the instruction, it forwards the tactile feedback instruction to the tactile driving module, the visual rendering instruction to the scene rendering module, and the auditory output instruction to the audio processing module at the corresponding time.

[0038] S5: After receiving the instruction, the haptic drive module, scene rendering module, and audio processing module start feedback execution according to the time parameter in the instruction, and record their respective feedback execution delay values ​​in real time during the execution process; each module sends the feedback execution delay value to the global time controller; the global time controller adjusts the generation offset of the next global timestamp according to the received feedback execution delay value, so that the actual execution time of the three types of instructions in the next round is consistent.

[0039] In step S5, each module performs feedback according to the instruction timing parameters, records the feedback execution delay value, and sends it to the global time controller. The global time controller then adjusts the offset for the next global timestamp generation accordingly. The haptic drive module, scene rendering module, and audio processing module each have their own dedicated time recording section to track the timing of feedback execution. When each module receives the corresponding feedback command, the time recording section first extracts the specified start time from the command (the trigger time for the haptic drive module, the deformation start time for the scene rendering module, and the playback start time for the audio processing module) and temporarily saves these times. When the module actually begins to execute the feedback operation, the time recording section obtains the current global timestamp generated by the global time controller, and then calculates the difference between the previously extracted start time and this current global timestamp. This difference represents the latency of the feedback execution, and then stores this latency value in the module's own local latency storage area. The principle behind this is that by recording the start time required by the command and the actual start time, the difference between the two can directly reflect the latency of the module when executing the feedback, and local storage facilitates subsequent processing and transmission of the latency value.

[0040] Next, each module activates its dedicated delay value transmission section, retrieves the recorded feedback execution delay value from its local delay storage area, and associates this delay value with the global timestamp carried by the feedback instruction. This forms a delay information combination that includes the module's identification information, the feedback execution delay value, and the corresponding global timestamp. Then, through a pre-defined dedicated channel for transmitting delay values, this delay information combination is sent in real time to the interface in the global time controller responsible for receiving delay information. The principle is that binding the delay value with the corresponding global timestamp clearly identifies which instruction execution the delay value corresponds to, while the module identification distinguishes the delay status of different modules. The dedicated transmission channel ensures that the delay information is accurately and timely delivered to the controller.

[0041] The global time controller initiates a delay information verification process, receiving combinations of delay information from various modules. First, it verifies whether the global timestamp included in the combination matches the controller's most recently generated global timestamp, thus confirming that the delay information corresponds to the latest instruction execution status. Simultaneously, it checks whether the feedback execution delay value is reasonable; that is, the value cannot be negative (because delay cannot be negative) and must be within a pre-defined reasonable range (avoiding abnormally large or small error values). For the feedback execution delay values ​​from the three modules that pass verification, they are categorized according to module type (haptic, visual, audio) and stored in the controller's dedicated delay data storage area. The principle behind this is that verifying the timestamp ensures that the latest delay data is being processed, checking the validity of the value prevents erroneous data from affecting subsequent adjustments, and categorized storage facilitates rapid retrieval and calculation of the delay status of each module.

[0042] Next, the global time controller initiates the offset calibration calculation process, extracting the latest set of feedback execution delay values ​​for the three modules from the delay data storage area. The largest of these three delay values ​​is determined as the baseline delay, because the largest delay value represents the slowest executing module among the three. To ensure that the actual execution times of the three modules are consistent, this slowest time must be used as the baseline. Then, the difference between the delay values ​​of the other two modules and this baseline delay is calculated. This difference is the offset that the corresponding module needs to compensate for in the next round. The compensation offset for the module corresponding to the baseline delay is set to zero. The principle is to make the modules with smaller delays start earlier by the corresponding time (i.e., compensate for the offset) to offset the delay difference between them and the slowest module, thereby making the final actual execution times of the three modules consistent.

[0043] Finally, the global time controller adjusts the parameters of the internal timestamp generation device, associating the calculated compensation offsets of the three modules with the reference of the next global timestamp to be generated. When generating the next global timestamp, the timestamp generation device adjusts the timestamps according to these compensation offsets, so that the start time of the next round of three types of instructions, after adding their respective compensation offsets, can keep the actual execution feedback time of the three modules consistent. The principle is to adjust the timestamps in advance so that the instruction start time of each module is advanced or remains unchanged, thereby offsetting their own execution delay differences and ultimately achieving the effect of synchronous execution.

[0044] Please see Figure 2 As shown, the present invention also provides a control system for a VR scene, comprising: The global time controller generates continuously increasing global timestamps at a fixed frequency and sends the global timestamps to the scene rendering module, audio processing module, and haptic drive module respectively. The scene interaction monitoring unit is used to continuously collect user body movement data and virtual object motion state data, analyze the interaction event type through kinematic calculations, and extract the corresponding feature parameters of the interaction event. The scene interaction monitoring unit binds the interaction event type and feature parameters with the current global timestamp to form a timestamped interaction information package. The multimodal feedback instruction generation unit is used to receive interactive information packets and generate three types of instructions based on the interactive event type and feature parameters, with the bound global timestamp as the reference: haptic feedback instructions, visual rendering instructions, and auditory output instructions. The feedback instruction distribution unit is used to forward the three types of instructions to the haptic drive module, scene rendering module, and audio processing module at the corresponding time according to the global timestamp carried by the instruction. The offset feedback unit is used by each module to perform feedback based on the instruction timing parameters, record the feedback execution delay value and send it to the global time controller. The global time controller adjusts the generation offset of the next global timestamp accordingly to ensure that the actual execution time of the three types of instructions in the next round is consistent.

[0045] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for controlling a VR scene, characterized in that, Includes the following steps: S1: Deploy a global time controller, which generates continuously increasing global timestamps at a fixed frequency and sends the global timestamps to the scene rendering module, audio processing module and haptic drive module respectively; S2: The scene interaction monitoring unit continuously collects user body movement data and virtual object motion state data, analyzes the interaction event type through kinematic calculations, and extracts the corresponding feature parameters of the interaction event; the scene interaction monitoring unit binds the interaction event type and feature parameters with the current global timestamp to form a timestamped interaction information package; S3: The multimodal feedback instruction generation unit receives the interaction information packet and generates three types of instructions based on the interaction event type and feature parameters, using the bound global timestamp as a reference: haptic feedback instruction, visual rendering instruction, and auditory output instruction; all three types of instructions carry the same global timestamp. S4: The feedback instruction distribution unit forwards the three types of instructions to the haptic drive module, scene rendering module, and audio processing module respectively at the corresponding time according to the global timestamp carried by the instruction; S5: Each module executes feedback based on the instruction timing parameters, records the feedback execution delay value and sends it to the global time controller. The global time controller adjusts the generation offset of the next global timestamp accordingly, so that the actual execution time of the three types of instructions in the next round is consistent.

2. The VR scene control method according to claim 1, characterized in that, In step S2, the scene interaction monitoring unit continuously collects user limb movement data and virtual object motion state data, and the process of parsing the interaction event type through kinematic calculations is as follows: Deploy a user limb movement acquisition submodule and a virtual object motion state acquisition submodule to collect motion data of key skeletal nodes of the user's limbs and motion state data of virtual objects, respectively. The collected data is preprocessed, the time axis of the two types of data is calibrated based on the global timestamp, the spatial coordinate system is unified through coordinate transformation, and the scale difference between the virtual environment and the physical collection space is removed. A user limb kinematics model and a virtual object kinematics model are established respectively. The preprocessed data is mapped to a limb kinematic chain and an object surface dynamic mesh model to form a continuous kinematic state sequence indexed by timestamps. Spatial indexing is performed on the limb kinematic chain and the dynamic mesh model of the object surface. The shortest spatial distance between the key points on the surface of the two is calculated in real time. When the distance is less than a preset threshold, it is determined to be a potential interaction state. The timestamp of the initial contact moment, the coordinates of the contact point, and the identification of the limb and object involved in the interaction are recorded. In the potential interaction state, the relative motion parameters at the contact point are parsed, integrated into an interaction feature set, and bound to the contact time timestamp; An interactive event classifier and a basic event type library including touch, grab, push, hit, and friction are constructed. The classifier takes parameters from the interactive feature set as input and matches a unique interactive event type through rule reasoning.

3. The VR scene control method according to claim 1, characterized in that, In step S2, the feature parameters corresponding to the interactive event are extracted simultaneously, including the interaction position coordinates, contact force value, and relative motion speed.

4. The VR scene control method according to claim 1, characterized in that, In step S3, the process of generating three types of instructions respectively: haptic feedback instructions, visual rendering instructions, and auditory output instructions is as follows: The multimodal feedback instruction generation unit receives the interaction information packet with timestamp, parses and extracts the interaction event type, feature parameters and global timestamp, and decomposes the feature parameters into three-dimensional components of the interaction position coordinates, scalar data of the contact force value and vector data of the relative motion velocity, and stores them separately. Establish a mapping rule library categorized by interaction event type, and store the parameter generation logic corresponding to each type of event, including the association between tactile, visual, and auditory parameters and feature parameters, as well as the duration matching rules; Each event calls the rule base logic for the corresponding interaction event type, calculates its own instruction parameters based on the contact force value and relative motion speed feature parameters, sets the start time based on the global timestamp and the offset relationship is fixed, and integrates them into their respective instruction core parameters. The three types of instructions are bound to the original global timestamp, and the time consistency and timing difference are verified. Once verified, a set of three types of instructions carrying the same global timestamp is formed.

5. The VR scene control method according to claim 4, characterized in that, In step S5, the specific method for integrating them into their respective instruction core parameters is as follows: The logic of the current event type in the rule base is called to calculate the intensity level based on the contact force value, and the duration of the event type is matched with the relative motion speed and the event type. The trigger time is set as a global timestamp plus a preset tactile offset, and integrated into tactile feedback instruction parameters. The logic of the current event type in the rule base is called to calculate the deformation amplitude parameter based on the contact force value. The relative motion speed is combined with the event type to match the duration. The deformation start time is set as a global timestamp plus a preset visual offset and integrated into the visual rendering instruction parameters. The logic of the current event type in the rule base is called to calculate the sound effect frequency parameter based on the relative motion speed modulus. The duration of the event is matched with the contact force value and the event type. The playback start time is set as a global timestamp plus a preset auditory offset and integrated into auditory output command parameters.

6. The VR scene control method according to claim 1, characterized in that, In step S3, the tactile feedback instruction includes the trigger time, intensity level, and duration; the visual rendering instruction includes the deformation start time, deformation amplitude parameter, and deformation duration; and the auditory output instruction includes the playback start time, sound effect frequency parameter, and sound effect duration.

7. The VR scene control method according to claim 1, characterized in that, In step S5, each module performs feedback according to the instruction timing parameters, records the feedback execution delay value, and sends it to the global time controller. The global time controller then adjusts the offset for the next global timestamp generation accordingly. Each module deployment time recording subunit records the startup time when receiving instructions, and calculates the feedback execution delay value and stores it in the local delay cache when actually executing feedback; Each module binds the feedback execution delay value with the corresponding global timestamp to form a delay information packet, which is then sent to the global time controller through a preset sub-channel; The global time controller verifies the consistency of timestamps and the validity of delay values ​​in delay information packets, and categorizes and stores the delay values ​​that have passed the verification. The global time controller uses the maximum delay value as a reference to calculate the compensation offset of the other two modules, and sets the compensation offset of the reference module to zero. The global time controller updates the timestamp generator parameters and associates the compensation offset to ensure that the next global timestamp is corrected so that the actual execution time of the three types of instructions in the next round is consistent.

8. The control system for a VR scene according to claim 1, characterized in that, Implementing a VR scene control method as described in any one of claims 1-7, comprising: The global time controller generates continuously increasing global timestamps at a fixed frequency and sends the global timestamps to the scene rendering module, audio processing module, and haptic drive module respectively. The scene interaction monitoring unit is used to continuously collect user body movement data and virtual object motion state data, analyze the interaction event type through kinematic calculations, and extract the corresponding feature parameters of the interaction event. The scene interaction monitoring unit binds the interaction event type and feature parameters with the current global timestamp to form a timestamped interaction information package. The multimodal feedback instruction generation unit is used to receive interactive information packets and generate three types of instructions based on the interactive event type and feature parameters, with the bound global timestamp as the reference: haptic feedback instructions, visual rendering instructions, and auditory output instructions. The feedback instruction distribution unit is used to forward the three types of instructions to the haptic drive module, scene rendering module, and audio processing module at the corresponding time according to the global timestamp carried by the instruction. The offset feedback unit is used by each module to perform feedback based on the instruction timing parameters, record the feedback execution delay value and send it to the global time controller. The global time controller adjusts the generation offset of the next global timestamp accordingly to ensure that the actual execution time of the three types of instructions in the next round is consistent.

Images