A virtual reality interaction method and system of multi-modal collaborative feedback

Abstract

The present application belongs to the field of virtual reality and human-computer interaction, and provides a kind of virtual reality interaction method and system of multi-modal collaborative feedback, obtains the real-time motion speed information of static rehabilitation bicycle pedal cycle, accurately synchronizes the speed of patient riding with the playing rate of panoramic video;Through scene slope visual recognition and dynamic damping feedback based on deep learning, analyze video terrain characteristics and adjust bicycle resistance, simulate real uphill and downhill sense of body;By integrating multiple pre-acquired branch routes, allow patients to turn the steering wheel at virtual intersections to choose paths independently, convert passive viewing into active exploration;Integrate heart rate monitoring bracelet, carbon dioxide detector, acetone detector, real-time evaluate patient rehabilitation training intensity and physical condition, provide data support for adaptive difficulty adjustment and safety warning.The present application enables users to ride vehicles in limited physical space to explore wider virtual space, achieving high-immersion rehabilitation VR experience.

Description

Technical Field

[0001] This invention belongs to the fields of virtual reality and human-computer interaction technology, and specifically relates to a virtual reality interaction method and system with multimodal collaborative feedback. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Stationary cycling equipment, such as medical rehabilitation bikes, home exercise bikes, or cycling trainers, is widely used due to its safety and high space utilization. However, whether for rehabilitation training for patients or daily fitness for ordinary users, traditional training methods are generally monotonous and tedious. In the rehabilitation field, this problem seriously affects patient adherence to treatment, thus limiting rehabilitation outcomes; in the public fitness field, it directly leads to decreased user interest in exercise, making it difficult to maintain long-term commitment.

[0004] Virtual Reality (VR) technology offers a new approach to solving this problem, enhancing user (including patients and regular cyclists) training motivation through immersive experiences. However, most existing virtual reality cycling applications still have significant limitations in terms of interactive realism, environmental adaptability, and personalized experiences, failing to meet the core requirements of an immersive experience. Specifically:

[0005] The virtual scene lacks real-time and accurate synchronization with the user's cycling intensity: the visual forward speed is often simply correlated with or completely disconnected from the user's actual cadence, which can easily lead to a disharmony between vision and vestibular proprioception, causing discomfort.

[0006] Environmental visual information was not effectively converted into real-time tactile feedback: Key terrain features in the virtual scene (such as slope and road surface type) were only presented as visual information and were not effectively converted into real-time, matching force feedback (such as changes in resistance). This resulted in a lack of effort when going uphill and a lack of inertia when going downhill, leading to a serious lack of immersion and realism.

[0007] The interaction mode is passive and linear, lacking the freedom of exploration: the existing cycling paths are mostly linear presets, and users can only passively follow them. They lack the ability to actively choose paths and explore in real time. The interaction mode is monotonous and it is difficult to maintain the novelty of long-term use.

[0008] Limited dimensions of user status perception and insufficient personalized adaptation: Most existing technologies only focus on basic exercise data (such as speed and mileage), lacking in-depth perception and fusion analysis of users' multimodal physiological status (such as heart rate and metabolic signals). It is difficult to dynamically adjust training load according to the user's real-time physical function status, and cannot provide users with different fitness levels (from rehabilitation patients to professional athletes) with a truly personalized, safe and efficient cycling experience.

[0009] Therefore, existing technologies struggle to create an integrated virtual reality cycling solution that combines high immersion, autonomous interactivity, personalized adaptability, and safety assurance. This has become a key technological bottleneck hindering the upgrade of user experience and widespread application of stationary cycling equipment. Summary of the Invention

[0010] To address the shortcomings of the prior art, this invention proposes a multimodal collaborative feedback virtual reality interaction method and system. By integrating virtual reality, deep learning, multimodal sensing, and adaptive control technologies, this invention constructs a highly immersive and intelligent interactive cycling experience. Its core lies in achieving deep collaboration between virtual scenes and real movement and physiological states, thereby comprehensively enhancing the immersion, freedom of interaction, and personalized adaptability of cycling.

[0011] According to some embodiments, the present invention adopts the following technical solution:

[0012] A multimodal collaborative feedback virtual reality interaction method includes the following steps:

[0013] Motion-sensory visual synchronization: a precise mapping relationship between cadence speed and visual forward speed is pre-established, the real-time cadence speed of the user on the fixed cycling equipment is obtained, and the panoramic video playback rate is dynamically adjusted according to the precise mapping relationship to solve the problem of visual-kinesthetic mismatch.

[0014] Real-time path generation and interaction based on intent recognition: Acquire steering data of the user's handlebars, identify the user's path selection intent at intersection nodes in the virtual environment based on the steering data, determine the target path from a pre-constructed multi-branch path grid based on the path selection intent, and generate or call corresponding visually smooth transition video clips to achieve seamless visual connection from the current path to the target path, giving the user the interactive freedom to actively explore.

[0015] Dynamic resistance feedback based on visual scene analysis: Based on a deep learning model, panoramic video frames are analyzed in real time to identify the data features of the terrain ahead (especially the road slope). The identified terrain slope information is converted into corresponding resistance control commands to adjust the resistance of the cycling equipment in real time, thereby simulating the feeling of going up and down hills in real cycling and realizing a closed loop of visual information to force feedback.

[0016] Adaptive optimization based on multimodal physiological perception: Real-time acquisition of users' multimodal physiological data, fusion and perception analysis of the multimodal physiological data, assessment of users' exercise intensity and physical function status during cycling, and dynamic optimization and adjustment of the basic resistance level of cycling equipment based on the assessment results, to achieve personalized adaptation of cycling difficulty and ensure the safety and effectiveness of training.

[0017] As an alternative implementation, the process of adjusting the panoramic video playback rate based on real-time cadence speed includes: constructing a playback rate coupling mechanism based on real-time pedal feedback; under this mechanism, the playback engine uses the currently measured speed of one pedal revolution as a reference coefficient for the playback rate, and the real-time playback rate... It is determined by the pedal cycle function; specifically, let the reference pedal cycle be... Corresponding standard playback speed At any time Real-time cadence speed As input, it is mapped to the real-time playback rate through the following relationship. :

[0018] = ;

[0019] A time series smoothing algorithm is used to adjust the real-time playback rate. Low-pass filtering and frame-level content adaptation are performed to maintain the visual coherence and temporal continuity of the video content during speed changes.

[0020] As an alternative implementation method, before acquiring the bicycle handlebar steering data and determining the target path selected by the user from multiple branch paths in the panoramic video based on the steering data, panoramic videos of each branch path are collected in different directions at key intersections of the real cycling road network. Transition videos of each steering connection segment are pre-synthesized using video generation technology, and the video segments are logically associated according to the real road network topology to form a graph-structured multi-path roaming network.

[0021] As a further defined implementation, the process of pre-synthesizing transition videos for each turning segment using video generation technology includes: for any two nodes... video clips connected and ,in, This is a video clip showing someone entering straight ahead. The video clip shows the vehicle turning and driving out. tail frame and The first frame As keyframe inputs, combined with structured text cues describing turning behavior, these are fed into a diffusion-based video generation engine. The engine uses a temporal attention mechanism to guide the generation of intermediate transition sequences that conform to both physical laws and visual continuity, using keyframes as strong visual conditions. :

[0022] ;

[0023] in, As a video generation engine based on a diffusion model, it uses a temporal attention mechanism to treat the input keyframes as strong visual conditions, ensuring that the start and end states of the generated sequence are consistent with them. Insert it into the generated transition video clip. and This enables a seamless visual transition from the current path to the turning path. To provide structured prompts, the video generation model is guided to generate panoramic videos and transition videos from straight ahead to left / right turns.

[0024] As an alternative implementation, acquiring the bicycle handlebar steering data and determining the target path selected by the user from multiple branch paths in the panoramic video based on the steering data includes: when the user rides to a node in the panoramic video... At that time, when the handlebars turn angle Exceeding the set threshold At that time, the system determines the next path based on the current node and the turning direction, and dynamically calls the pre-synthesized corresponding turning and exiting video clips. And the generated visually smooth intermediate sequence completes the smooth path switching.

[0025] As an alternative implementation, the process of visually recognizing terrain slope in panoramic video based on deep learning includes: Let the image of frame t be... The depth of the entire map is obtained through a depth estimation model. And obtain the road area mask through interactive segmentation. Define the region of interest mask Get the area of ​​the road within the region of interest. = Extract the depth value of the region and calculate its median. Calibration is performed using the first N frames to establish a reference depth. Calculate the relative depth change:

[0026] ;

[0027] Calculate the slope angle :

[0028] ;

[0029] Noise smoothing is achieved by using a sliding window averaging method. = , To smooth out window sizes.

[0030] As an alternative implementation method, the process of acquiring the user's multimodal physiological data, performing fusion and perceptual analysis on the multimodal physiological data, and assessing the user's training intensity and physical function during cycling includes: acquiring the user's heart rate, exhaled acetone, and carbon dioxide concentration physiological data; discretizing the heart rate into five damping levels, denoted as... Where i∈{1,2,3,4,5}, the higher the level, the greater the resistance;

[0031] The time before the start of riding is designated as the equipment warm-up period. After the warm-up period, the median acetone concentration within the predetermined time window is calculated. And classify the motion states according to them;

[0032] The user's metabolic state is estimated in real time using a model that couples heart rate and exhaled carbon dioxide signals. The model is based on the user's real-time heart rate. With weight Estimate its minute ventilation This is used as the upper limit of metabolic capacity; real-time monitoring of exhaled carbon dioxide concentration is used. With environmental background concentration The difference was used to calculate the change in alveolar carbon dioxide partial pressure, which reflects actual metabolic activity. ;

[0033] Through standard respiratory quotient Convert this to oxygen consumption per minute and calculate the real-time calorie consumption rate. .

[0034] As an alternative implementation method, terrain slope identification and adaptive optimization work together to affect resistance control, resulting in a final resistance control signal. Adaptive basic resistance based on physiological state Contextual resistance based on visual slope The superposition structure, its mathematical model is:

[0035] ;

[0036] in, Based on heart rate level Acetone in breath and carbon dioxide concentration The calculated adaptive resistance function, For mapping coefficients, To recognize slope based on vision The mapping function. That is, when going uphill, , It is a linear or nonlinear increasing function, which increases the drag, i.e. Simulates climbing load; during descent, , To be a decreasing function, thus reducing resistance or even providing assistance, i.e. This simulates the feeling of gliding. This design allows the resistance feedback to respond simultaneously to the user's internal state and the external virtual environment, forming a complete perception-feedback loop.

[0037] A multimodal collaborative feedback virtual reality interaction system, comprising:

[0038] The motion-sensing-based visual synchronization module is used to pre-establish a precise mapping relationship between cadence speed and visual forward speed, obtain the user's real-time cadence speed on a fixed cycling device, and dynamically adjust the panoramic video playback rate according to the precise mapping relationship to solve the visual-kinesthetic mismatch problem.

[0039] The real-time path generation and interaction module based on intent recognition is used to acquire the steering data of the user's handlebars, identify the user's path selection intent at intersection nodes in the virtual environment based on the steering data, determine the target path from the pre-constructed multi-branch path grid based on the path selection intent, and generate or call the corresponding visually smooth transition video clips to achieve seamless visual connection from the current path to the target path, giving the user the interactive freedom to actively explore.

[0040] The dynamic resistance feedback module based on visual scene analysis is used to analyze panoramic video frames in real time based on a deep learning model, identify the data features of the terrain ahead (especially the road slope), convert the identified terrain slope information into corresponding resistance control commands, and adjust the resistance of the cycling equipment in real time, thereby simulating the feeling of going up and down hills in real cycling and realizing a closed loop of visual information to force feedback.

[0041] The adaptive optimization module based on multimodal physiological perception is used to acquire users' multimodal physiological data in real time, perform fusion perception analysis on the multimodal physiological data, evaluate the user's exercise intensity and physical function status during cycling, and dynamically optimize and adjust the basic resistance level of cycling equipment based on the evaluation results to achieve personalized adaptation of cycling difficulty, ensuring the safety and effectiveness of training.

[0042] As an alternative implementation, it also includes a virtual reality device, a handlebar steering sensor, and a cadence period sensor. The physiological state monitoring and adaptive interaction module includes a heart rate monitoring wristband, a carbon dioxide detector, and an acetone detector, wherein:

[0043] Heart rate monitoring wristbands are used to detect a user's heart rate;

[0044] A carbon dioxide detector is used to detect carbon dioxide in exhaled breath.

[0045] An acetone detector for detecting acetone in exhaled breath;

[0046] Virtual reality devices are worn by users to display panoramic videos;

[0047] Handlebar steering sensors are used to capture the bicycle's steering angle;

[0048] The cadence period sensor is used to collect the number of complete pedal rotations in real time and calculate the real-time cadence speed.

[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0050] This invention enhances immersion and realism. By precisely synchronizing cadence speed with video playback rate, it addresses the core pain point of visual-motor and proprioceptive dissonance. Furthermore, through real-time scene analysis based on deep learning, it transforms visual information about terrain slope into real-time matched force-resistance feedback, constructing a "what you see is what you feel" visual-force closed loop. This allows users to experience an immersive sensation close to that of real outdoor cycling even on a fixed vehicle.

[0051] This invention breaks through the limitations of traditional linear path interaction, granting users a high degree of freedom in exploration. Its pioneering multi-path synthesis and real-time steering interaction mechanism revolutionizes virtual cycling from a passive, linear viewing mode to a non-linear exploration experience where users can actively choose branch paths. Users can autonomously determine their direction of travel through natural steering operations, greatly enhancing the fun, engagement, and replay value of the content.

[0052] This invention provides intelligent, personalized exercise adaptation and safety assurance based on multimodal perception. Moving beyond traditional solutions that rely solely on exercise data, this invention deeply integrates multi-dimensional physiological signals such as heart rate, exhaled carbon dioxide, and acetone to achieve precise and holistic perception of the user's exercise intensity and metabolic state. Based on this holistic perception, the system can adjust exercise load in real-time and adaptively, ensuring training efficiency and safety for users at different fitness levels (from rehabilitation users to professional athletes) while also providing scientific guidance for users pursuing specific fitness goals (such as efficient fat burning).

[0053] This invention boasts high integration and versatility. The system structure is clear, requires no large physical space, and allows for flexible deployment. Its technical solution not only perfectly meets the comprehensive needs of medical settings such as hospital rehabilitation departments and community rehabilitation centers for safety, effectiveness, and enjoyment, but can also be seamlessly extended to a wide range of fields including commercial gyms, home fitness, and professional sports training, providing various users with a next-generation virtual cycling solution that integrates immersion, interactivity, and intelligence.

[0054] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0055] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0056] Figure 1 This is a flowchart of a multimodal collaborative interaction according to an embodiment of the present invention; it clearly shows the complete closed-loop data flow from sensor data acquisition to core processing (path decision, resistance calculation) and then to actuator control (video playback, resistance adjustment);

[0057] Figure 2 This is a schematic diagram illustrating the working principle of slope information preprocessing and multi-path transition video synthesis in a panoramic video according to one embodiment of the present invention; it specifically demonstrates the data processing flow of visual slope recognition and the path transition segment generation logic based on AIGC technology.

[0058] Figure 3 This is a framework diagram of an overall system application scenario according to an embodiment of the present invention; it depicts the interaction relationship between the user, hardware devices (bicycle, sensor, VR headset) and software system (processing logic, virtual environment);

[0059] Figure 4 This is a hardware system control and communication architecture diagram according to an embodiment of the present invention; it specifically illustrates the connection relationship and control signal flow between various sensors, actuators and central processing unit. Detailed Implementation

[0060] This embodiment uses a virtual cycling system as an example to explain in detail how the present invention constructs an immersive interactive experience through two core technologies: real-time path generation and multimodal resistance feedback. The invention will be further described below with reference to the accompanying drawings and embodiments.

[0061] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0062] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0063] Where there is no conflict, the embodiments and features described in this application may be combined with each other.

[0064] Example 1

[0065] This embodiment uses a virtual cycling system as an example to describe in detail a multimodal collaborative feedback virtual reality interaction method according to the present invention. The method integrates virtual reality, deep learning, multimodal sensing, and adaptive control technologies to construct a highly immersive and intelligently interactive cycling experience. Its core lies in achieving deep collaboration between the virtual scene and real movement and physiological state, thereby comprehensively enhancing the immersion, freedom of interaction, and personalized adaptability of cycling. Its overall workflow is as follows: Figure 3 The multimodal collaborative interaction flowchart shows that it constitutes a complete closed loop of perception-decision-execution.

[0066] (I) Overall System Workflow

[0067] like Figure 1 As shown, the operation of this system begins with the collection of multi-source data by the perception layer, including bicycle cadence, handlebar steering angle, user heart rate, and exhaled acetone and carbon dioxide concentrations. This data is then transmitted to the decision layer (server) for fusion processing. At the decision layer, the system uses algorithms to deeply couple and collaboratively calculate the user's motion behavior, real-time physiological state, and virtual scene information, aiming to achieve a balance between immersion, freedom of interaction, and personalized adaptation. Its overall workflow can be broken down into the following four core stages:

[0068] 1) Virtual Scene Preprocessing: Before the interaction begins, the system preprocesses the virtual environment content. This includes pre-collecting and synthesizing panoramic videos of multi-branch paths based on the real road network, and using video generation technology to generate smooth visual transition segments between paths, constructing a graph-like roaming network that can be freely explored.

[0069] 2) Two-way interactive feedback: The system establishes a real-time two-way perception-feedback loop. In the "user → virtual environment" direction, the user's pedal speed drives the playback rate of the panoramic video in real time, achieving synchronous interaction of motion driving visual progress. In the "virtual environment → user" direction, the system analyzes the road slope information in the video in real time and converts it into matching resistance control commands to adjust the bicycle's damping, thereby completing the force feedback from the virtual content to the user. The combination of these two aspects constitutes the core two-way closed loop of speed synchronization and force simulation.

[0070] 3) Active Decision-Making: By recognizing the user's steering data from maneuvering the handlebars, the system transforms the traditional linear, passive video playback mode into a user-driven path selection experience. When the user makes a turning maneuver at a virtual intersection, the system seamlessly switches to the corresponding target path video, giving the user the interactive freedom to explore autonomously in the virtual world.

[0071] 4) Multimodal physiological adaptive adjustment: The system continuously monitors and integrates the user's multimodal physiological data to dynamically assess their real-time exercise intensity and physical load status. Based on this assessment result, the system adaptively adjusts the basic resistance level of cycling to achieve personalized adjustment of training difficulty, while ensuring the safety and effectiveness monitoring of the training process.

[0072] Ultimately, the collaborative control commands generated by the decision-making layer are sent to the execution layer, driving the VR device to perform visual rendering and scene switching, and the vehicle resistance system to perform real-time damping adjustment, thus forming a complete, dynamic, and personalized "perception-decision-execution" interactive closed loop.

[0073] Step 1: Motion-Aware Visual Synchronization. The system acquires the user's real-time cadence speed on the stationary cycling equipment and dynamically adjusts the panoramic video playback rate accordingly, establishing a precise mapping between cadence speed and visual forward speed. This process forms the core interaction link between the user and the virtual environment, resolving the visual-kinesthetic mismatch problem.

[0074] Step 2: Real-time path generation and interaction based on intent recognition. Acquire steering data of the user's handlebars; based on the steering data, identify the user's path selection intent at intersection nodes in the virtual environment; based on the intent, determine the target path from a pre-constructed multi-branch path grid, and generate or call corresponding visually smooth transition video clips to achieve seamless visual connection from the current path to the target path, giving the user the interactive freedom to actively explore.

[0075] Step 3: Dynamic Resistance Feedback Based on Visual Scene Analysis. A deep learning model is used to analyze panoramic video frames in real time, identifying data features of the terrain ahead (especially road slope). The identified terrain slope information is converted into corresponding resistance control commands, adjusting the resistance of the cycling equipment in real time. This process constitutes the core force feedback link from "virtual environment → user," simulating the sensation of going uphill and downhill in real cycling, achieving a closed loop from visual information to force feedback.

[0076] Step 4: Adaptive Optimization Based on Multimodal Physiological Perception. Real-time acquisition of the user's multimodal physiological data; fusion and perceptual analysis of the physiological data to assess the user's exercise intensity and physical function during cycling; based on the assessment results, dynamically optimizing and adjusting the basic resistance level of the cycling equipment to achieve personalized adaptive cycling difficulty, ensuring the safety and effectiveness of training.

[0077] (II) Real-time path generation and interaction

[0078] To enable users to actively explore the virtual environment, the system has constructed a path network that can be freely roamed. Figure 2 The pretreatment and synthesis principles were demonstrated.

[0079] During the content preparation phase, panoramic videos of branch routes are captured in different directions at key intersections of the real road network. For any two video clips that connect at node N... (Go straight in) and (Turn and drive out), take tail frame and The first frame As keyframes, combined with a structured Prompt representing "panoramic video, transitioning from straight ahead to left / right turn," a visually smooth intermediate transition sequence is generated by a commercial video generation engine based on a diffusion model. This achieves seamless visual connections between paths. All these video clips are logically linked based on the real road network topology, forming a graph-like multi-path roaming network.

[0080] During the interaction phase, when the user rides to the virtual intersection node N, the handlebar steering sensor continuously monitors the steering angle. Once the angle exceeds the set threshold... The system then determines that the user intends to turn and decides on the next path based on the current node and the turning direction. Subsequently, the system dynamically calls the pre-synthesized target path video. and transition video It enables smooth path switching and gives users the freedom to explore independently.

[0081] (III) Dynamic resistance feedback of multimodal fusion

[0082] One of the core innovations of this invention lies in the resistance control signal. The synthesis method integrates environmental visual perception and user physiological state, and the calculation formula is as follows:

[0083] ;

[0084] The formula contains two key parts:

[0085] 1. Personalized basic resistance (·): This section is based on the user's multimodal physiological data. Heart rate is discretized into multiple damping levels. The higher the level, the greater the base resistance. Simultaneously, the system calculates the median acetone concentration in exhaled breath within a time window (e.g., 60 seconds). It is used to assess exercise intensity (such as regular exercise, fat-burning status, etc.). In addition, it is combined with heart rate. The minute ventilation (VE) was estimated using body weight (W) and real-time monitoring of expiratory carbon dioxide concentration. With environmental background concentration The difference is used to calculate the real-time calorie expenditure rate using the standard respiratory quotient (RQ). . The (·) function integrates these parameters to calculate the basic resistance that adapts to the user's current physical condition, ensuring the safety and effectiveness of training.

[0086] 2. Contextual resistance This section is based on real-time analysis of the virtual scene. For example... Figure 1 As shown, for panoramic video frames Depth estimation is obtained And obtain the road area mask through interactive segmentation. Focus on the area of ​​interest ahead. Extract the median depth of the road region and with reference depth Comparison, through formula The visual slope angle was calculated. And further obtained through a sliding window .function Map this visual slope to a resistance adjustment amount. When going uphill ( ), S(·) is an increasing function, and the increase in resistance simulates the climbing load; when going downhill ( S(·) is a decreasing function, and the decrease in resistance simulates the feeling of gliding.

[0087] Final resistance control signal It is the superposition of the two mentioned above, through Figure 4The hardware system control and communication architecture shown is sent to the adjustable resistance system of the bicycle for execution, thereby realizing the direct and realistic conversion of visual information into force feedback.

[0088] (iv) Pedal frequency synchronization and system integration

[0089] To ensure immersion, the system establishes a synchronization mechanism between cadence and video playback. This involves real-time playback rate (vt) and real-time cadence. They are directly proportional, and the relationship is: = ,in As the baseline cadence, Standard playback rate. A time-series smoothing algorithm is used. Filtering is performed to ensure visual continuity of the video during speed changes.

[0090] In application scenarios, users wear VR devices and ride on a fixed bicycle. All their operations and physiological states are mapped to the virtual environment in real time, forming a highly consistent immersive experience.

[0091] (v) Hardware System Composition

[0092] The hardware control and communication architecture of this system is as follows: Figure 4 As shown, it mainly includes:

[0093] Sensing units: cadence cycle sensor, handlebar steering sensor, heart rate monitoring wristband, carbon dioxide and acetone detectors.

[0094] The core of computation and decision-making: the server, which is responsible for core logic such as path generation and resistance calculation.

[0095] Execution units: VR rendering host (used to display visual content), and the bicycle's adjustable resistance system.

[0096] Communication module: Responsible for data transmission between units.

[0097] Example 2

[0098] A virtual reality interaction system for implementing the method described in Embodiment 1, the system comprising:

[0099] The cadence-adaptive video playback control module is configured to acquire the real-time cadence of the bicycle and adjust the panoramic video playback rate accordingly, achieving precise synchronization between cycling speed and video playback.

[0100] The multi-path synthesis and interactive roaming module is configured to acquire the steering data of the bicycle handlebars, determine the target path selected by the user based on the steering data, and generate or call visually smooth transition video clips to achieve seamless path switching.

[0101] The visual slope recognition and dynamic resistance feedback module is configured to recognize the terrain slope in panoramic video based on deep learning, and generate corresponding resistance control commands based on the recognized slope.

[0102] The physiological state monitoring and adaptive interaction module is configured to acquire the user's multimodal physiological data, perform fusion analysis on the data to assess the user's state, and optimize and adjust the bicycle resistance accordingly.

[0103] The system also includes hardware entities such as virtual reality devices, handlebar steering sensors, cadence period sensors, heart rate monitoring wristbands, carbon dioxide detectors, and acetone detectors. Each module and hardware works together to implement the above-mentioned method and process.

[0104] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal collaborative feedback virtual reality interaction method, characterized in that, Includes the following steps: Motion-sensing-based visual synchronization: a precise mapping relationship between cadence speed and visual forward speed is pre-established, the real-time cadence speed of the user on the fixed cycling equipment is obtained, and the panoramic video playback rate is dynamically adjusted according to the precise mapping relationship. Real-time path generation and interaction based on intent recognition: acquire steering data of the user's handlebars, identify the user's path selection intent at intersection nodes in the virtual environment based on the steering data, determine the target path from a pre-constructed multi-branch path grid based on the path selection intent, and generate or call corresponding visually smooth transition video clips to achieve seamless visual connection from the current path to the target path. Dynamic resistance feedback based on visual scene analysis: Based on a deep learning model, panoramic video frames are analyzed in real time to identify the data features of the terrain ahead. The identified terrain slope information is converted into corresponding resistance control commands to adjust the resistance of the cycling equipment in real time. The process of visually recognizing terrain slope in panoramic video based on deep learning includes: Let the image of frame t be... The depth of the entire map is obtained through a depth estimation model. And obtain the road area mask through interactive segmentation. Define the region of interest mask Get the area of ​​the road within the region of interest. = Extract the depth value of the region and calculate its median. Calibration is performed using the first N frames to establish a reference depth. Calculate the relative depth change: ； Calculate the slope angle : ； Noise smoothing is achieved by using a sliding window averaging method. = , To smooth out window sizes; Adaptive optimization based on multimodal physiological perception: real-time acquisition of users' multimodal physiological data, fusion and perception analysis of multimodal physiological data, assessment of users' exercise intensity and physical function status during cycling, and dynamic optimization and adjustment of the basic resistance level of cycling equipment based on the assessment results; Final resistance control signal Adaptive basic resistance based on physiological state Contextual resistance based on visual slope The superposition structure, its mathematical model is: ； in, Based on heart rate level 100% exhaled acetone and carbon dioxide concentration The calculated adaptive resistance function, For mapping coefficients, To recognize slope based on vision The mapping function, that is, when going uphill, , It is a linear or nonlinear increasing function, which increases the drag, i.e. Simulates climbing load; during descent, , To be a decreasing function, thus reducing resistance or even providing assistance, i.e. To simulate the feeling of gliding.

2. The virtual reality interaction method with multimodal collaborative feedback as described in claim 1, characterized in that, The process of adjusting the panoramic video playback rate based on real-time cadence speed includes: constructing a playback rate coupling mechanism based on real-time pedal feedback. Under this mechanism, the playback engine uses the currently measured speed of one pedal revolution as the baseline coefficient for the playback rate. It is determined by the pedal cycle function; specifically, let the reference pedal cycle be... Corresponding standard playback speed At any time Real-time cadence speed As input, it is mapped to the real-time playback rate through the following relationship. : = ； A time series smoothing algorithm is used to adjust the real-time playback rate. Low-pass filtering and frame-level content adaptation are performed to maintain the visual coherence and temporal continuity of the video content during speed changes.

3. The virtual reality interaction method with multimodal collaborative feedback as described in claim 1, characterized in that, Before acquiring the bicycle handlebar steering data and determining the target path selected by the user from multiple branch paths in the panoramic video based on the steering data, panoramic videos of each branch path are collected in different directions at key intersections of the real cycling road network. Transition videos of each steering connection segment are pre-synthesized using video generation technology, and the video segments are logically linked according to the real road network topology to form a graph-structured multi-path roaming network.

4. The virtual reality interaction method with multimodal collaborative feedback as described in claim 3, characterized in that, The process of pre-synthesizing transition videos for each turning segment using video generation technology includes: for any two points at the node video clips connected and ,in, This is a video clip showing someone entering straight ahead. The video clip shows the vehicle turning and driving out. tail frame and The first frame As keyframe inputs, combined with structured text cues describing turning behavior, these are fed into a diffusion-based video generation engine. The engine uses a temporal attention mechanism to guide the generation of intermediate transition sequences that conform to both physical laws and visual continuity, using keyframes as strong visual conditions. : ； in, As a video generation engine based on a diffusion model, it uses a temporal attention mechanism to treat the input keyframes as strong visual conditions, ensuring that the start and end states of the generated sequence are consistent with them. Insert it into the generated transition video clip. and This enables a seamless visual transition from the current path to the turning path. To provide structured prompts, the video generation model is guided to generate panoramic videos and transition videos from straight ahead to left / right turns.

5. The virtual reality interaction method with multimodal collaborative feedback as described in claim 1, characterized in that, The process of acquiring bicycle handlebar steering data and determining the target path selected by the user from multiple branch paths in the panoramic video based on this data includes: when the user rides to a node in the panoramic video... At that time, when the handlebars turn angle Exceeding the set threshold At that time, the system determines the next path based on the current node and the turning direction, and dynamically calls the pre-synthesized corresponding turning and exiting video clips. And the generated visually smooth intermediate sequence completes the smooth path switching.

6. The virtual reality interaction method with multimodal collaborative feedback as described in claim 1, characterized in that, The process of acquiring users' multimodal physiological data, performing fusion and perceptual analysis on this data, and assessing the user's training intensity and physical function during cycling includes: acquiring the user's heart rate, exhaled acetone, and carbon dioxide concentration data; discretizing the heart rate into five damping levels, denoted as... Where i∈{1,2,3,4,5}, the higher the level, the greater the resistance; The time before the start of riding is designated as the equipment warm-up period. After the warm-up period, the median acetone concentration within the predetermined time window is calculated. And classify the motion states according to them; The user's metabolic state is estimated in real time using a model that couples heart rate and exhaled carbon dioxide signals. The model is based on the user's real-time heart rate. With weight Estimate its minute ventilation This is used as the upper limit of metabolic capacity; real-time monitoring of exhaled carbon dioxide concentration is used. With environmental background concentration The difference was used to calculate the change in alveolar carbon dioxide partial pressure, which reflects actual metabolic activity. ; Through standard respiratory quotient Convert this to oxygen consumption per minute and calculate the real-time calorie consumption rate. .

7. A multimodal collaborative feedback virtual reality interaction system, characterized in that, include: The motion-sensing-based visual synchronization module is used to pre-establish a precise mapping relationship between cadence speed and visual forward speed, obtain the user's real-time cadence speed on a fixed cycling device, and dynamically adjust the panoramic video playback rate according to the precise mapping relationship to solve the visual-kinesthetic mismatch problem. The real-time path generation and interaction module based on intent recognition is used to acquire the steering data of the user's handlebars, identify the user's path selection intent at intersection nodes in the virtual environment based on the steering data, determine the target path from the pre-constructed multi-branch path grid based on the path selection intent, and generate or call the corresponding visually smooth transition video clips to achieve seamless visual connection from the current path to the target path, giving the user the interactive freedom to actively explore. The dynamic resistance feedback module based on visual scene analysis is used to analyze panoramic video frames in real time based on a deep learning model, identify the data features of the terrain ahead, convert the identified terrain slope information into corresponding resistance control commands, and adjust the resistance of the cycling equipment in real time, thereby simulating the feeling of going up and down hills in real cycling and realizing a closed loop of visual information to force feedback. The process of visually recognizing terrain slope in panoramic video based on deep learning includes: Let the image of frame t be... The depth of the entire map is obtained through a depth estimation model. And obtain the road area mask through interactive segmentation. Define the region of interest mask Get the area of ​​the road within the region of interest. = Extract the depth value of the region and calculate its median. Calibration is performed using the first N frames to establish a reference depth. Calculate the relative depth change: ； Calculate the slope angle : ； Noise smoothing is achieved by using a sliding window averaging method. = , To smooth out window sizes; The adaptive optimization module based on multimodal physiological perception is used to acquire users' multimodal physiological data in real time, perform fusion perception analysis on the multimodal physiological data, evaluate the user's exercise intensity and physical function status during cycling, and dynamically optimize and adjust the basic resistance level of cycling equipment based on the evaluation results to achieve personalized adaptation of cycling difficulty and ensure the safety and effectiveness of training. Final resistance control signal Adaptive basic resistance based on physiological state Contextual resistance based on visual slope The superposition structure, its mathematical model is: ； in, Based on heart rate level 100% exhaled acetone and carbon dioxide concentration The calculated adaptive resistance function, For mapping coefficients, To recognize slope based on vision The mapping function, that is, when going uphill, , It is a linear or nonlinear increasing function, which increases the drag, i.e. Simulates climbing load; during descent, , To be a decreasing function, thus reducing resistance or even providing assistance, i.e. To simulate the feeling of gliding.

8. A multimodal collaborative feedback virtual reality interaction system as described in claim 7, characterized in that, It also includes virtual reality devices, handlebar steering sensors, and cadence sensors. The physiological state monitoring and adaptive interaction module includes a heart rate monitoring wristband, a carbon dioxide detector, and an acetone detector, among which: Heart rate monitoring wristbands are used to detect a user's heart rate; A carbon dioxide detector is used to detect carbon dioxide in exhaled breath. An acetone detector for detecting acetone in exhaled breath; Virtual reality devices are worn by users to display panoramic videos; Handlebar steering sensors are used to capture the bicycle's steering angle; The cadence period sensor is used to collect the number of complete pedal rotations in real time and calculate the real-time cadence speed.

Images