VR scenarized application method based on mixed multi-modal interaction

By using a hybrid multimodal interaction method, combining gesture trackers, eye-tracking sensors, and speech recognition units, the problems of fragmented operation and insufficient immersion in traditional VR scenarios are solved, improving the smoothness of operation and the accuracy of intent recognition, and enhancing scene adaptability and experience iteration capabilities.

CN121957334APending Publication Date: 2026-05-01PIMAX TECH (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202512027682.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-30
Publication Date
2026-05-01

AI Technical Summary

Technical Problem

Traditional single-modal interaction in VR scenarios suffers from fragmented operation, high learning cost, and insufficient immersion. Gesture tracking is prone to loss of positioning due to occlusion or rapid movements, leading to operation lag. Intent recognition is easily affected by the ambiguity of single-modal information, resulting in misjudgment.

Method used

A hybrid multimodal interaction method is adopted, which combines a gesture tracker, an eye-tracking sensor and a speech recognition unit. The skeletal topology prediction algorithm completes the joint positions under occlusion or rapid movements, and the eye-tracking data is combined to identify the focus target. The interaction priority rules are dynamically set to ensure the accuracy and real-time performance of the operation, and the interaction data is recorded to optimize the algorithm and rules.

Benefits of technology

It significantly improves the smoothness of operation and the accuracy of intent recognition in VR scenarios, enhances scene adaptability and the ability to continuously iterate the experience, reduces the learning cost for beginners, and improves the user's sense of immersion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121957334A_ABST
    Figure CN121957334A_ABST
Patent Text Reader

Abstract

The invention discloses a VR scenarized application method based on hybrid multi-modal interaction. According to the method, the operation fluency and the intention recognition accuracy in the VR scene are remarkably improved through mixed multi-modal interaction. The gesture topology pre-judgment algorithm can complement joint positions in shielding or rapid actions, maintain continuity of instrument positioning and avoid operation lagging caused by interruption of single gesture tracking; eye movement data and gesture tracks are combined, so that the system captures a focusing target of a user in advance, an intention recognition module is assisted to more accurately understand user requirements, and misjudgment caused by fuzzy single-mode information is reduced. The cooperation of the voice instruction and the visual action further supplements the details of the interaction intention, so that the complex operation can be quickly responded by the system, the user does not need to repeatedly adjust the action or repeat the instruction, and the operation process is smoother.
Need to check novelty before this filing date? Find Prior Art

Description

A VR Scene Application Method Based on Hybrid Multimodal Interaction Technical Field

[0001] This invention belongs to the field of VR scene simulation technology, specifically a VR scene application method based on hybrid multimodal interaction. Background Technology

[0002] In VR scenarios, medical simulations (such as surgical training), immersive games, and vocational skills training (such as industrial equipment operation) demand extremely high levels of precision, naturalness, and real-time performance in interaction. VR scene simulation is a technological application that utilizes virtual reality technology to reconstruct and dynamically simulate the real world or imagined environments in a three-dimensional computer space with high fidelity. By integrating 3D modeling, physics engines, real-time rendering, and multi-sensor interaction technologies, it constructs immersive digital scenes with visual, auditory, and even tactile feedback, making users feel as if they are actually there. This technology can not only accurately replicate complex spaces such as urban buildings and industrial production lines, but also simulate extreme environments or special working conditions such as fires and earthquakes. Compared to traditional simulations, VR scene simulation emphasizes strong interactivity and a first-person experience, and is widely used in fields such as education and training, military exercises, urban planning, psychological therapy, and gaming, providing users with a low-cost, zero-risk, and repeatable experimental and training environment.

[0003] However, traditional single-modal interaction (such as gamepad buttons) suffers from problems such as fragmented operation, high learning costs, and insufficient immersion. Gesture tracking is prone to loss of positioning due to occlusion or rapid movements, leading to operation lag, and intent recognition is susceptible to misjudgment due to the ambiguity of single-modal information. Summary of the Invention

[0004] The purpose of this invention is to provide a VR scenario-based application method based on hybrid multimodal interaction in order to solve the problems mentioned above.

[0005] The technical solution adopted in this invention is as follows: A VR scene-based application method based on hybrid multimodal interaction, the method comprising the following steps: S1: Loading the target VR scene and initializing the multimodal interaction perception module, including a gesture tracker, an eye-tracking sensor, and a speech recognition unit, to prepare for subsequent real-time collection of user gestures, eye gaze points, and voice commands; S2: Real-time collection of user gesture trajectories, eye-tracking focus positions, and voice input data in the VR scene, and synchronous transmission of these multimodal data to the fusion processing module; S3: Fusion processing of the collected multimodal data, wherein a skeletal topology prediction algorithm is used for gesture tracking to complete occlusions or fast movements based on the user's historical interaction trajectory. S4: Based on the current VR scene type, dynamically set the priority rules for multimodal interaction to ensure that operational conflicts in different scenes are reasonably resolved; S5: Execute the user's interaction intent based on the priority rules, and trigger the highlighting or annotation feedback of the focused target through the eye-tracking-virtual scene linkage engine to ensure the accuracy and real-time performance of the operation; S6: Record all operation data of the user during the interaction process, including gesture trajectory, eye tracking duration and voice command response, to optimize the completion accuracy of the skeletal topology prediction algorithm and the rationality of scene priority rules, so as to achieve continuous iterative upgrades of the interactive experience.

[0006] In a preferred embodiment, in step S1, when loading the target VR scene, the system first reads the 3D model, texture map, and interaction logic script in the scene resource package to complete the spatial layout and rendering parameter configuration of the scene elements; at the same time, the system calibrates the coordinate system of the VR device to ensure that the virtual scene is aligned with the physical space, and defines the trigger area and response rules of each interactive object in the scene to provide a positioning reference for the subsequent tracking target of the perception module; after the scene is loaded, the system preloads auxiliary resources related to the interaction, including the skeletal topology model, eye-tracking gaze point mapping table, etc., to ensure that the perception module can directly call the required data when initializing, reducing the delay in subsequent real-time interaction.

[0007] In a preferred embodiment, in step S2, the gesture trajectory acquisition relies on a high sampling rate sensor component to capture the spatial coordinates and motion vectors of each joint of the hand in real time. The joint position calculation is combined with the skeletal topology model to optimize the accuracy and ensure the continuity of the trajectory under rapid movements. The eye-tracking focus position is mapped to the three-dimensional gaze point of the VR scene by tracking eye feature points and simultaneously eliminating head movement offset to maintain positioning accuracy. The voice input uses a microphone array to collect signals, and the continuous voice stream is segmented into fixed-length frames by filtering background noise and echo through algorithms.

[0008] In a preferred embodiment, in step S2, all acquired data frames are marked with a high-precision timestamp generated by a unified clock source to avoid the impact of clock deviations between devices on the synchronization effect; the transmission adopts a low-latency protocol, and each modal data is transmitted independently but maintains a coordinated rhythm; the receiving end fusion module verifies the timestamps of each modality, calculates the data deviation value at the same time point, and dynamically adjusts the processing rhythm if the deviation exceeds the threshold.

[0009] In a preferred embodiment, in step S3, the received gesture trajectory, eye-tracking focus, and voice command data are first timestamped to ensure that the data of each modality are analyzed in the same time sequence. For the gesture tracking part, the skeletal topology prediction algorithm will refer to the joint movement patterns accumulated in the user's historical interaction trajectory. When the current gesture is detected to be occluded or the joint position information is interrupted due to rapid movement, the algorithm will combine the joint coordinates of adjacent frames and the preset skeletal topology model to fill in the missing joint position data. At the same time, the system will extract the focus position and duration information from the eye-tracking data, perform spatial matching with the interactive objects in the VR scene, identify the user's current focus target, and then combine the movement direction and speed characteristics of the gesture trajectory to preliminarily judge the user's potential interaction intention.

[0010] In a preferred embodiment, in step S4, the current VR scene type is first identified through scene metadata, and then a predefined scene priority library is invoked. In a medical scene, voice commands are given the highest priority due to the need for precise and rapid response, followed by gesture operations, with eye-tracking aiming as an auxiliary priority. In a game scene, gesture trigger operations, which are directly associated with the core gameplay, are given the highest priority, followed by eye-tracking aiming, with voice commands having the lowest priority. When the scene changes, the system updates the priority rules in real time to ensure that they match the interaction logic of the current scene. The dynamic priority score calculation formula is as follows: In the formula: P(m,s) represents the final priority score of interaction modality m in scenario s; B(m,s) represents the basic priority value of modality m in scenario s; Ws represents the scenario weight coefficient; R(m,s) represents the real-time behavior score of modality m in scenario s; Wr represents the real-time behavior weight coefficient.

[0011] In a preferred embodiment, in step S4, the priority weight is fine-tuned by combining real-time user interaction data during the dynamic priority adjustment process of real-time behavior feedback; the adjustment process needs to refer to the gesture completion data in S3 and the interaction feedback results in S5.

[0012] In a preferred embodiment, in step S5, the dynamic priority sorting table generated in S4 is first called to determine the priority of the currently triggered multimodal interaction intent; in the medical surgery scenario, when the voice command and the gesture cutting operation are triggered at the same time, the system executes the voice amplification command first and then processes the gesture cutting according to the medical scenario rules preset in S4; when the gesture operation is executed, the system calls the joint data completed in S3 to ensure that the cutting path matches the user's real action intent.

[0013] In a preferred embodiment, in step S5, the eye-tracking-virtual scene linkage engine binds the focused target identified in S3 to virtual scene elements, triggering highlighting or annotation feedback; the feedback intensity is dynamically adjusted based on real-time user interaction data: the longer the eye-tracking gaze on the target and the closer the hand gesture joint is to the target, the more obvious the feedback; in a game scene, when the user's eye tracks an enemy and the hand gesture trigger is pressed, a yellow highlight circle appears around the enemy; the feedback result is synchronized to S6 in real time for subsequent optimization of the skeletal topology prediction algorithm and scene priority rules; wherein the real-time feedback intensity calculation formula is: In the formula: H(o) represents the feedback intensity of virtual object o; T(o) represents the duration of eye gaze of the user on object o; Seye(o) represents the eye focus score of object o; D(g,o) represents the three-dimensional distance between gesture joint g and object o; R(g) represents the reliability score of gesture joint g.

[0014] In a preferred embodiment, in step S6, the system will completely record all operation data during the user interaction process, including the joint movement sequence of the gesture trajectory, the cumulative duration and interval of eye focus, the recognition accuracy and response delay of voice commands, and other details; for the skeletal topology prediction algorithm, the system compares the deviation between the completed joint position and the actual user action, adjusts the joint constraint parameters in the model, and optimizes the fusion prediction logic by combining the correlation data of eye movement and gesture; for the scene priority rule, the system counts the user correction frequency when different interaction intentions are triggered at the same time, adjusts the priority weight, and also judges the user's attention level based on the eye focus duration to optimize the intent judgment criteria.

[0015] In summary, due to the adoption of the above technical solutions, the beneficial effects of this invention are as follows: 1. In this invention, the smoothness of operation and the accuracy of intent recognition are significantly improved in VR scenes through hybrid multimodal interaction. The gesture topology prediction algorithm can complete the joint positions in occlusion or rapid movements, maintain the continuity of instrument positioning, and avoid operation lag caused by interruption of single gesture tracking; the combination of eye-tracking data and gesture trajectory allows the system to capture the user's focus target in advance, assisting the intent recognition module to understand user needs more accurately and reducing misjudgments caused by ambiguity of single-modal information. The coordination of voice commands and visual actions further supplements the details of interaction intent, allowing complex operations to be responded to quickly by the system, eliminating the need for users to repeatedly adjust actions or repeat commands, making the operation process smoother.

[0016] 2. This invention enhances scene adaptability and the ability to continuously iterate on the user experience. The priority determination module, based on comprehensive analysis of multimodal data, can quickly filter out the user's current core needs from multiple potential intentions, ensuring that high-priority operations receive timely responses and avoiding irrelevant interactions that interfere with the user's focus. Simultaneously, the interaction data recorded by the system is used to optimize the skeletal topology prediction algorithm and scene priority rules. As the number of times the user uses the system increases, the algorithm gradually adapts to the user's operating habits, and the interaction response better matches individual expectations. Precise calibration of the virtual scene and physical space, as well as the reasonable setting of the trigger areas for interactive objects, also enhance the user's immersion, making virtual operations closer to the behavioral logic of the real world, reducing the learning cost for novice users, and allowing users with different operating habits to obtain a smooth interactive experience. Attached Figure Description

[0017] Figure 1 is a schematic diagram of the overall method flow of the present invention; Figure 2 is a flowchart of the multimodal interaction scenario application in the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Example: Referring to Figures 1-2, a VR scene-based application method based on hybrid multimodal interaction includes the following steps: S1: Load the target VR scene and initialize the multimodal interaction perception module, including a gesture tracker, an eye-tracking sensor and a speech recognition unit, to prepare for subsequent real-time collection of the user's gestures, eye gaze points and voice commands.

[0020] S2: Real-time acquisition of user gesture trajectories, eye-tracking focus positions, and voice input data in VR scenes, and synchronous transmission of this multimodal data to the fusion processing module to ensure the temporal consistency of the data to support subsequent intent understanding.

[0021] S3: The collected multimodal data is fused and processed. For gesture tracking, a skeletal topology prediction algorithm is used to complete the joint positions under occlusion or rapid movements based on the user's historical interaction trajectory, so as to avoid the loss of instrument positioning during operation. At the same time, eye-tracking data is combined to initially identify the user's focus target, providing a basis for subsequent priority determination.

[0022] S4: Dynamically set the priority rules for multimodal interaction based on the type of the current VR scene. For example, in medical scenes, voice commands have higher priority than gesture operations, and in game scenes, gesture trigger operations have higher priority than eye-tracking aiming, to ensure that operation conflicts in different scenes are resolved reasonably.

[0023] S5: Execute the user's interaction intent based on priority rules. For example, in a medical scenario, respond to the voice amplification command first and then execute the gesture cutting operation. At the same time, trigger the highlighting or annotation feedback of the focused target through the eye-tracking-virtual scene linkage engine to ensure the accuracy and real-time performance of the operation. This step requires calling the gesture data completed by S3 and the priority rules set by S4.

[0024] S6: Records all user operation data during the interaction process, including gesture trajectory, eye-tracking focus duration, and voice command response, to optimize the completion accuracy of the skeletal topology prediction algorithm and the rationality of scene priority rules, so as to achieve continuous iterative upgrades of the interactive experience.

[0025] In step S1, when loading the target VR scene, the system first reads the 3D model, texture maps, and interaction logic scripts from the scene resource package to complete the spatial layout and rendering parameter configuration of the scene elements. Simultaneously, the system calibrates the coordinate system of the VR device to ensure alignment between the virtual scene and the physical space, and defines the trigger areas and response rules for each interactive object within the scene, providing a positioning reference for the subsequent tracking of the target by the perception module. After the scene is loaded, the system preloads auxiliary resources related to interaction, including skeletal topology models and eye-tracking gaze mapping tables, ensuring that the perception module can directly access the required data during initialization, reducing latency in subsequent real-time interactions.

[0026] When initializing the multimodal interaction perception module, the gesture tracker first loads a preset human skeletal topology model, adjusts the sensitivity threshold for joint recognition, and completes the initial localization and tracking range setting of the hand area. The eye-tracking sensor initiates a calibration process, guiding the user to gaze at multiple reference points within the virtual scene to generate mapping parameters between eye features and virtual gaze points, and sets the sampling frequency and smoothing filter coefficients for the gaze points. The speech recognition unit loads a scene-specific acoustic model and semantic dictionary, configures wake-word detection rules and background noise reduction parameters, and establishes a real-time transmission channel for the speech signal. Furthermore, the system simultaneously initializes the timestamp generation mechanism for the three modules to ensure that subsequently collected gesture, eye-tracking, and speech data are aligned on the same timeline, laying the foundation for multimodal data fusion.

[0027] In step S2, gesture trajectory acquisition relies on high-sampling-rate sensor components to capture the spatial coordinates and motion vectors of each joint of the hand in real time. Joint position calculation is combined with skeletal topology model to optimize accuracy and ensure trajectory continuity under rapid movements. Eye-tracking focus position is achieved by tracking eye feature points and mapping two-dimensional eye-tracking data to three-dimensional gaze points in the VR scene, while simultaneously eliminating head movement offset to maintain positioning accuracy. Voice input uses a microphone array to collect signals, and the algorithm filters background noise and echoes, segmenting the continuous speech stream into fixed-length frames to ensure the clarity and coherence of the speech data, providing a reliable foundation for subsequent semantic recognition.

[0028] In step S2, all acquired data frames are marked with a high-precision timestamp generated by a unified clock source to avoid clock deviations between devices affecting synchronization. A low-latency protocol is used for transmission, with each modality's data transmitted independently but maintaining a coordinated rhythm. The receiving end's fusion module verifies the timestamps of each modality and calculates the data deviation value at the same time point. If the deviation exceeds a threshold, the processing rhythm is dynamically adjusted—delaying the processing of subsequent frames or using interpolation to fill in missing frames, keeping the timing alignment error within milliseconds. The synchronized multimodal data forms a coherent temporal stream, ensuring that subsequent intent understanding can accurately associate the synchronized behavioral logic of gestures, eye movements, and speech.

[0029] In step S3, the received gesture trajectory, eye-tracking focus, and voice command data are first timestamped to ensure that the data from each modality are analyzed in the same time sequence. For the gesture tracking part, the skeletal topology prediction algorithm refers to the joint movement patterns accumulated in the user's historical interaction trajectory. When it detects that the current gesture is occluded or that a rapid movement causes an interruption in the joint position information, the algorithm combines the joint coordinates of adjacent frames and the preset skeletal topology model to fill in the missing joint position data, maintain the continuity of device positioning, and avoid the loss of device positioning during operation. At the same time, the system extracts the focus position and duration information from the eye-tracking data, performs spatial matching with interactive objects in the VR scene, identifies the user's current focus target, and then combines the movement direction and speed characteristics of the gesture trajectory to preliminarily determine the user's potential interaction intention. This provides a key basis for the application of subsequent scene priority rules, ensuring the accuracy and timeliness of the interaction response.

[0030] In step S4, the current VR scene type (medical simulation, gaming, industrial training, etc.) is first identified through scene metadata (such as preset tags and core object features). Then, a predefined scene priority library is invoked. This library presets the basic priorities of core modalities for different scene interaction needs: In medical scenes, voice commands (such as "magnify the lesion") are given the highest priority due to the need for precise and rapid response, followed by gesture operations (such as instrument holding), with eye-tracking aiming as an auxiliary; in gaming scenes, gesture trigger operations (such as shooting) are directly associated with the core gameplay and are given the highest priority, followed by eye-tracking aiming (such as locking onto a target), with voice commands (such as chatting) having the lowest priority. When a scene changes (such as switching from medical surgery training to post-operative equipment debugging), the system updates the priority rules in real time to ensure matching with the interaction logic of the current scene; the dynamic priority score calculation formula is: In the formula: P(m, s) represents the final priority score of interaction modality m in scenario s; B(m, s) represents the basic priority value of modality m in scenario s (preset, such as B=0.8 for voice and B=0.5 for gesture in medical scenarios); Ws represents the scenario weight coefficient (fixed value, such as 0.7-0.7, reflecting the dominance of basic rules); R(m, s) represents the real-time behavior score of modality m in scenario s (calculated based on the success rate and frequency of the most recent NN interactions, ranging from 0 to 10-1); Wr represents the real-time behavior weight coefficient (fixed value, such as 0.3-0.3, balancing the impact of real-time feedback); In step S4, the priority weight is fine-tuned in combination with real-time user interaction data during the dynamic priority adjustment of real-time behavior feedback. For example, in a medical scenario, if a user completes a fine operation with gestures three times in a row without error, the system will increase the priority weight of the gesture operation; in a game scenario, if a user frequently switches weapons with voice commands, the system will increase the priority score of the voice commands. The adjustment process needs to refer to the gesture completion data of S3 (to verify the validity of the gesture) and the interaction feedback results of S5 (to judge the success rate of the operation). Finally, a dynamic priority sorting table is generated to resolve multimodal conflicts (such as when the voice "stop" and the gesture trigger are triggered at the same time, the higher priority instruction is executed according to the sorting table).

[0031] In step S5, the dynamic priority sorting table generated in S4 is first invoked to determine the priority of the currently triggered multimodal interaction intent. For example, in a medical surgery scenario, when the voice command "magnify the lesion" and the gesture cutting operation are triggered simultaneously, the system, according to the medical scenario rules preset in S4 (voice has higher priority than gesture), prioritizes executing the voice magnification command and then processes the gesture cutting. When executing the gesture operation, the system calls the joint data completed in S3 (such as the precise position of the wrist and fingers) to ensure that the cutting path matches the user's actual action intent; for each interaction intent, the system verifies its validity (such as whether the gesture conforms to the topological rules of the human skeleton and whether the voice command is clear and recognizable), and only includes valid intents in the execution queue to avoid misoperation interfering with the scene process.

[0032] In step S5, the eye-tracking-virtual scene linkage engine binds the focused target (such as the lesion area or game character) identified in S3 to virtual scene elements, triggering highlighting or annotation feedback. The feedback intensity is dynamically adjusted based on real-time user interaction data: the longer the eye-tracking focus on the target and the closer the hand gesture joint is to the target, the more obvious the feedback (e.g., increased highlight brightness, thicker annotation lines). For example, in a medical scenario, when the user's eye-tracking focus on the lesion exceeds 2 seconds and the hand gesture joint is less than 3 cm from the lesion, the system automatically generates a red annotation line at the edge of the lesion; in a game scenario, when the user's eye-tracking locks onto an enemy and the hand gesture trigger is pressed, a yellow highlight circle appears around the enemy. The feedback results are synchronized to S6 in real time for subsequent optimization of the skeletal topology prediction algorithm and scene priority rules. The formula for calculating the real-time feedback intensity is: In the formula: H(o) represents the feedback intensity of the virtual object o (range 0-1, the higher the value, the more obvious the feedback); T(o) represents the duration of the user's eye gaze on object o (unit: seconds); Seye(o) represents the eye focus score of object o (from eye-gesture fusion recognition in S3, range 0-1); D(g, o) represents the three-dimensional distance between the gesture joint g and object o (unit: centimeters); R(g) represents the reliability score of the gesture joint g (from skeletal topology completion verification in S3, range 0-1); In step S6, the system will completely record all operation data during the user interaction process, including the joint motion sequence of the gesture trajectory, the cumulative duration and interval of eye focus, the recognition accuracy and response delay of voice commands, and other details. After these data are classified and stored, they are analyzed in multiple dimensions to optimize the completion accuracy of the skeletal topology prediction algorithm and the rationality of the scene priority rules. For the skeletal topology prediction algorithm, the system compares the deviation between the completed joint positions and the actual user movements, adjusts the joint constraint parameters in the model, and optimizes the fusion prediction logic by combining eye-tracking and gesture correlation data. For scene priority rules, the system statistically analyzes the frequency of user corrections when different interaction intentions are triggered simultaneously, adjusts the priority weights, and also judges the user's attention level based on eye-tracking focus duration to optimize the intent judgment criteria. The system regularly updates the optimized algorithm and rules to the interaction module, and verifies the effect through subsequent interaction data, forming a closed loop of data collection, analysis, optimization, and verification iteration to continuously improve the smoothness and accuracy of the interaction experience.

[0033] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the term "comprising" or any other variations thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0034] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A VR scenario-based application method based on hybrid multimodal interaction, characterized in that: The method Includes the following steps: S1: Load the target VR scene and initialize the multimodal interaction perception module, including a gesture tracker, eye movement sensor and voice recognition unit, to prepare for subsequent real-time collection of user gestures, eye gaze points and voice commands; S2: Real-time acquisition of user gesture trajectories, eye-tracking focus positions, and voice input data in the VR scene, and synchronous transmission of this multimodal data to the fusion processing module; S3: Fusion processing of the acquired multimodal data, including the use of a skeletal topology prediction algorithm for gesture tracking to complete the joint positions under occlusion or rapid movements based on the user's historical interaction trajectory, and the initial identification of the user's focus target by combining eye-tracking data to provide a basis for subsequent priority determination; S4: Dynamically set the priority rules for multimodal interaction based on the type of the current VR scene to ensure that operational conflicts in different scenes are resolved reasonably; S5: Execute the user's interaction intent based on the priority rules, and trigger the highlighting or annotation feedback of the focused target through the eye-tracking-virtual scene linkage engine to ensure the accuracy and real-time performance of the operation; S6: Records all user operation data during the interaction process, including gesture trajectory, eye-tracking focus duration, and voice command response, to optimize the completion accuracy of the skeletal topology prediction algorithm and the rationality of scene priority rules, so as to achieve continuous iterative upgrades of the interactive experience.

2. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S1, when loading the target VR scene, the system first reads the 3D model, texture map, and interaction logic script in the scene resource package to complete the spatial layout and rendering parameter configuration of the scene elements. At the same time, the system calibrates the coordinate system of the VR device to ensure that the virtual scene is aligned with the physical space, and defines the trigger area and response rules of each interactive object in the scene to provide a positioning reference for the subsequent tracking target of the perception module. After the scene is loaded, the system preloads auxiliary resources related to the interaction, including the skeletal topology model and eye-tracking gaze point mapping table, to ensure that the perception module can directly call the required data during initialization and reduce the delay in subsequent real-time interaction.

3. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S2, the gesture trajectory acquisition relies on a high sampling rate sensor component to capture the spatial coordinates and motion vectors of each joint of the hand in real time. The joint position calculation is combined with the skeletal topology model to optimize the accuracy and ensure the continuity of the trajectory under rapid movements. Eye-tracking focusing position maps two-dimensional eye-tracking data to three-dimensional gaze points in the VR scene by tracking eye feature points, and simultaneously eliminates head movement offset to maintain positioning accuracy; voice input uses a microphone array to collect signals, and filters background noise and echoes through algorithms to segment the continuous voice stream into fixed-length frames.

4. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S2, all acquired data frames are marked with a high-precision timestamp generated by a unified clock source to avoid the impact of clock deviation between devices on the synchronization effect; the transmission adopts a low-latency protocol, and each modal data is transmitted independently but maintains rhythm coordination; the receiving end fusion module verifies the timestamp of each modality, calculates the data deviation value at the same time point, and dynamically adjusts the processing rhythm if the deviation exceeds the threshold.

5. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S3, the received gesture trajectory, eye tracking and voice command data are first timestamped to ensure that the data of each modality are analyzed in the same time sequence. For the gesture tracking part, the skeletal topology prediction algorithm refers to the joint movement patterns accumulated in the user's historical interaction trajectory. When it detects that the current gesture is occluded or that the joint position information is interrupted due to rapid movement, the algorithm will combine the joint coordinates of adjacent frames and the preset skeletal topology model to fill in the missing joint position data. At the same time, the system will extract the focus position and duration information from the eye-tracking data, perform spatial matching with interactive objects in the VR scene, identify the user's current focus target, and then combine the movement direction and speed characteristics of the gesture trajectory to make a preliminary judgment on the user's potential interaction intention.

6. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S4, the current VR scene type is first identified through scene metadata, and then a predefined scene priority library is called. In medical scenarios, voice commands are prioritized for their need for precise and rapid response, followed by gestures, with eye-tracking aiming serving as a secondary priority. In gaming scenarios, gesture triggers, directly linked to core gameplay, are prioritized, followed by eye-tracking aiming, with voice commands having the lowest priority. When scenarios change, the system updates the priority rules in real time to ensure they match the interaction logic of the current scenario. The dynamic priority score calculation formula is as follows: ; In the formula: P(m,s) represents the final priority score of interaction modality m in scenario s; B(m,s) represents the basic priority value of modality m in scenario s; Ws represents the scenario weight coefficient; R(m,s) represents the real-time behavior score of modality m in scene s; Wr represents the real-time behavior weight coefficient.

7. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S4, the priority weights are fine-tuned by combining real-time user interaction data during the dynamic priority adjustment process of real-time behavior feedback. The adjustment process requires reference to the gesture completion data of S3 and the interaction feedback results of S5.

8. The VR scene-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S5, the dynamic priority sorting table generated in S4 is first called to determine the priority of the currently triggered multimodal interaction intent. In the medical surgery scenario, when voice commands and gesture cutting operations are triggered simultaneously, the system executes the voice amplification command first and then processes the gesture cutting according to the medical scenario rules preset in S4. When the gesture operation is executed, the system calls the joint data completed in S3 to ensure that the cutting path matches the user's real action intent.

9. A VR scenario-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S5, the eye-tracking-virtual scene linkage engine binds the focus target identified in S3 with virtual scene elements, triggering highlighting or annotation feedback. The feedback intensity is dynamically adjusted based on real-time user interaction data: the longer the eye movement gazes at the target and the closer the hand gesture joint is to the target, the more obvious the feedback. When a user locks onto an enemy with their eyes and presses the trigger in a game scene, a bright yellow circle appears around the enemy; the feedback is synchronized to S6 in real time for subsequent optimization of the skeletal topology prediction algorithm and scene priority rules. The formula for calculating the real-time feedback intensity is: In the formula: H(o) represents the feedback intensity of virtual object o; T(o) represents the duration of eye gaze of the user on object o; Seye(o) represents the eye focus score of object o; D(g,o) represents the three-dimensional distance between gesture joint g and object o; R(g) represents the reliability score of gesture joint g.

10. A VR scenario-based application method based on hybrid multimodal interaction as described in claim 1, characterized in that: In step S6, the system will fully record all operation data during the user interaction process, including the joint movement sequence of the gesture trajectory, the cumulative duration and interval of eye focus, the recognition accuracy and response delay of voice commands, and other details. For the skeletal topology prediction algorithm, the system compares the deviation between the completed joint position and the actual user action, adjusts the joint constraint parameters in the model, and optimizes the fusion prediction logic by combining the correlation data of eye movement and gesture. For the scene priority rule, the system counts the user correction frequency when different interaction intentions are triggered at the same time, adjusts the priority weight, and also judges the user's attention level based on the eye focus duration to optimize the intent judgment criteria.