Virtual reality design system and method based on multi-modal data gesture action recognition

Abstract

The invention discloses a virtual reality design system and method based on multi-modal data gesture action recognition, and relates to the technical field of virtual reality, and the method comprises a gesture recognition unit which is used for collecting multi-modal data generated by hand actions of a user, and recognizing gesture actions of the user based on the multi-modal data; the control unit is connected with the gesture recognition unit and is used for mapping the gesture action into an operation instruction and controlling modeling software to generate an updated virtual reality display interface according to the operation instruction; the VR display unit is connected with the control unit and is used for receiving and displaying the updated virtual reality display interface; the multi-modal data can be fused with sensing information of different sources, the accuracy and robustness of gesture recognition are improved, and continuous and complex hand actions of a user can be accurately recognized; in combination with a gesture-function mapping configuration method, gesture actions can be mapped into corresponding modeling operation instructions, so that a user can control software operation through natural gestures in a VR environment.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a virtual reality design system and method based on multimodal data gesture recognition. Background Technology

[0002] Virtual Reality (VR) is a technology that uses computers to generate three-dimensional interactive environments, allowing users to have an immersive experience similar to the real world. Its core lies in combining multi-sensory feedback, including visual, auditory, and tactile feedback, to simulate real-world scenarios and enable interaction. With the rapid development of VR technology and breakthroughs in 5G, artificial intelligence, and sensor technologies, VR is increasingly being used in entertainment, education, and industrial design. However, despite continuous equipment upgrades, VR interaction methods still have many limitations, hindering its further popularization and application in complex tasks.

[0003] Most mainstream VR systems currently use controllers for interaction. While they can map button operations to control commands, they cannot achieve natural hand-free interaction. Furthermore, gesture recognition solutions based on single-modal data are limited by the limitations of sensor information, making it difficult to accurately recognize continuous and complex hand movements of users. They also cannot reliably map these gestures to corresponding modeled operation commands. As a result, users lack the means to interact with VR through natural gestures, which severely limits the convenience of operation. Summary of the Invention

[0004] The purpose of this application is to provide a virtual reality design system and method based on multimodal data gesture recognition, which can solve the problem mentioned above that users lack means to interact through natural gestures in VR environments.

[0005] To achieve the above objectives, this application provides the following solution: In a first aspect, this application provides a virtual reality design system based on multimodal data gesture recognition, comprising: A gesture recognition unit is used to collect multimodal data generated by the user's hand movements and to recognize the user's gesture movements based on the multimodal data; A control unit, connected to the gesture recognition unit, is used to map the gesture action into an operation command, and control the modeling software to generate an updated virtual reality display interface according to the operation command; The VR display unit is connected to the control unit and is used to receive and display the updated virtual reality display interface.

[0006] In one embodiment, the multimodal data includes electromyographic signals and IMU signals.

[0007] In one embodiment, the gesture recognition unit includes: An electromyography (EMG) sensor is used to acquire the electrical signals from the muscles. An IMU sensor is used to acquire the IMU signal.

[0008] The core processing module, connected to the electromyography sensor, the IMU sensor, and the control unit, is used to recognize the user's gesture based on the electromyography signal, the IMU signal, and a preset gesture recognition model, and to transmit the gesture to the control unit.

[0009] In one embodiment, the gesture recognition unit is a myoelectric armband worn on the user's forearm.

[0010] In one embodiment, the control unit has a built-in preset gesture library to map the gesture actions into corresponding operation commands.

[0011] In one embodiment, the VR display unit further includes an interactive prompt module, used to provide feedback on the gesture, operation command, and function activation result in the form of a visual layer in the VR visual interface.

[0012] Secondly, this application also provides a virtual reality design method based on multimodal data gesture recognition, including: Collect multimodal data generated by the user's hand movements, and recognize the user's hand gestures based on the multimodal data; The gestures are mapped to operation commands, and the modeling software is controlled to generate an updated virtual reality display interface based on the operation commands. The updated virtual reality display interface is received and displayed through the VR display unit.

[0013] Thirdly, this application also provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described virtual reality design method based on multimodal data gesture recognition.

[0014] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described virtual reality design method based on multimodal data gesture recognition.

[0015] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described virtual reality design method based on multimodal data gesture recognition.

[0016] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a virtual reality design system based on multimodal data gesture recognition. A gesture recognition unit collects multimodal data generated by the user's hand movements and recognizes the user's gestures based on this data. A control unit, connected to the gesture recognition unit, maps gestures to operation commands and controls the modeling software to generate an updated virtual reality display interface based on these commands. A VR display unit, connected to the control unit, receives and displays the updated virtual reality display interface. In this application, multimodal data can fuse sensor information from different sources, overcoming the limitations of single-modality recognition and improving the accuracy and robustness of gesture recognition. This enables accurate recognition of continuous and complex hand movements. Combined with a gesture-operation mapping configuration method, gestures can be mapped to corresponding modeling operation commands, allowing users to control the software in the VR environment through natural gestures. This effectively solves the problems of unnatural interaction with traditional controllers and the low accuracy and poor continuity of single-modality gesture recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a structural block diagram of a virtual reality design system based on multimodal data gesture recognition according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating the gestures of a virtual reality design system based on multimodal data gesture recognition, according to an embodiment of this application. Figure 3 This is a schematic diagram of a head-mounted display device for a virtual reality design system based on multimodal data gesture recognition, according to an embodiment of this application. Figure 4 This is a flowchart illustrating a virtual reality design method based on multimodal data gesture recognition according to an embodiment of this application. Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] See Figure 1 This application provides a virtual reality design system based on multimodal data gesture recognition, including a gesture recognition unit, a control unit, and a VR display unit.

[0022] In this embodiment of the application, the gesture recognition unit is used to collect multimodal data generated by the user's hand movements and to recognize the user's gesture movements based on the multimodal data.

[0023] Multimodal data, by fusing signals from different sources, achieves complementary advantages, thereby overcoming the limitations of single-modality data and significantly improving the accuracy, robustness, and naturalness of gesture recognition. Specifically, multimodal data includes electromyography (EMG) signals and IMU signals.

[0024] This application achieves high-precision and large-range coordinated control by fusing electromyography (EMG) signals and IMU signals. EMG signals can capture subtle muscle activity, suitable for detecting fine manipulations; while IMU signals sense forearm posture and movement trajectory, supporting the detection of large-range movements. The two modalities complement each other, overcoming the limitations of single technologies.

[0025] In this embodiment, the gesture recognition unit includes an electromyography (EMG) sensor, an induction generator (IMU) sensor, and a core processing module. The EMG sensor is used to collect electromyographic signals, and the IMU sensor is used to collect IMU signals. The core processing module is connected to the EMG sensor, the IMU sensor, and the control unit, and is used to recognize the user's gesture based on the EMG signals, the IMU signals, and a preset gesture recognition model, and then transmit the gesture to the control unit.

[0026] For example, the electromyography (EMG) sensor uses an 8-channel EMG sensor to acquire more comprehensive muscle activity data, while the IMU sensor uses a 9-axis IMU sensor to capture forearm posture and motion information from all angles.

[0027] The 8-channel electromyography (EMG) sensor simultaneously acquires multiple sets of muscle electrical signals through eight independent acquisition points distributed across the user's forearm, enabling precise capture of activation patterns in different muscle groups. The multi-channel configuration provides more comprehensive spatial distribution information of muscle activity, significantly enhancing the ability to distinguish complex hand movements and providing rich physiological signal features for accurate recognition of various gestures. The 9-axis IMU sensor integrates a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, comprehensively tracking the forearm's motion in three-dimensional space. It accurately reconstructs the forearm's absolute posture, rotational trajectory, and spatial displacement, providing the system with continuous and complete arm kinematic data, effectively capturing the user's wide range of spatial movements and fine posture adjustments.

[0028] A gesture recognition model is a machine learning model configured to output corresponding gesture classification results based on input electromyography (EMG) signals and IMU signals. Specifically, the gesture recognition model achieves gesture classification by fusing EMG and IMU signals. First, the acquired raw signals are preprocessed, and then input into a trained machine learning classifier (such as the Random Forest algorithm or the SVM algorithm). The model analyzes EMG activity patterns and forearm movement characteristics to identify specific gestures performed by the user in real time, such as clenching a fist, opening it, or clicking, and converts the recognition results into corresponding gesture classification outputs.

[0029] The gesture recognition model achieves accurate classification of gestures based on multimodal data fusion and machine learning algorithms. It requires preprocessing the raw surface electromyography (EMG) signals acquired by the EMG sensor to extract temporal and frequency domain features; simultaneously, it performs attitude calculation and motion trajectory reconstruction on accelerometer and gyroscope data acquired by the IMU sensor. Then, feature-level fusion combines the features of the two modalities into a unified feature vector, which is input into a pre-trained random forest or support vector machine classifier. By analyzing the correlation between muscle activation patterns and forearm motion dynamics, a mapping relationship between multi-dimensional features and specific gestures is established.

[0030] In this embodiment, the gesture recognition unit is a myoelectric armband worn on the user's forearm. It can accurately capture the muscle electrical signals generated by the related muscle activities of the hand, without restricting hand movements. This ensures the effectiveness of signal acquisition to support the accuracy of gesture recognition, while also conforming to the user's natural operating posture, thus improving the convenience and comfort of the interaction process.

[0031] In this embodiment, the control unit is connected to the gesture recognition unit and is used to map gesture actions into operation commands, and control the modeling software to generate an updated virtual reality display interface according to the operation commands.

[0032] This application also includes a communication module, which wirelessly transmits the recognition results to the control unit. The control unit maps each gesture to a corresponding mouse operation command, facilitating natural control of modeling commands such as selection, dragging, scaling, or rotation in the modeling software, replacing traditional mouse input. Preferably, the modeling software in this application is Blender software.

[0033] Specifically, the control unit has a built-in preset gesture library to map gesture actions to operation commands. The preset gesture library refers to a predefined set of standard gesture-operation mapping relationships. See Appendix. Figure 2 For example, the "Relax" gesture typically maps to the default mouse operation state or release action, indicating no active input. The "Click" gesture maps to a left mouse button click, used to select objects in the modeling software interface. The "Long Press" gesture maps to a long left mouse button press, enabling dragging or object movement. The "Up," "Down," "Left," and "Right" gestures map to the vertical and horizontal movement directions of the mouse cursor, respectively, controlling cursor navigation in the VR environment. The "Wheel 1" and "Wheel 2" gestures can be mapped to mouse wheel functionality, used for model zooming, view rotation, or tool switching. Those skilled in the art can also set other mapping relationships according to actual needs.

[0034] In this embodiment, the VR display unit is connected to the control unit and is used to receive and display the updated virtual reality display interface. See also Figure 3 The VR display unit uses a VR head-mounted display device, which is the main display terminal in the system and can provide users with an immersive visual experience.

[0035] The procedure for using a VR headset is as follows: Press and hold the power button for 3 seconds to turn on the headset, then put it on and adjust it to a clear and comfortable position. Lift the back headrest and bring the headset close to your eyes. Adjust the headset to ensure your field of vision is unobstructed and the image is clear. Once comfortable, fasten the back headrest and tighten the knob at the back of the head. Then, fine-tune the top strap to reduce pressure on your forehead. Adjust the interpupillary distance (IPD) in the system settings interface by clicking the "+" or "-" buttons until the image is clear.

[0036] The PC serves as the execution platform for the control unit, and its integrated interface mapping and interaction module acts as a crucial bridge between the PC operating system and the VR display environment. This module maps the PC's software interface to the VR display unit, allowing users to directly view and interact with the Blender interface on the PC within the VR display unit.

[0037] The interface mapping and interaction module can utilize the OVR Toolkit, a desktop window mapping tool built on SteamVR. Its core function is to map and display a 2D PC interface in real time within a 3D VR environment. This interface mapping and interaction module mainly consists of four core parts: a window capture module, a SteamVR integration module, a rendering and spatial positioning engine, and an interaction processing module.

[0038] Specifically, the window capture module acquires the virtual reality display interface generated on the PC; the integration module transmits this image stream to SteamVR; the rendering engine uses the interface provided by SteamVR to draw the image as a positionable "panel" in the VR field of view; the interaction module ensures that the user's operations in VR can be accurately transmitted back to the corresponding PC window, thus forming a complete closed loop of "capture, transmission, rendering, and interaction", and finally mapping and integrating the PC interface into the VR experience.

[0039] In this application, to achieve high-speed and stable content transmission from the PC to the VR display unit, a streaming channel is constructed using a game streaming assistant and the SteamVR framework to seamlessly project the real-time running interface of the modeling software onto the VR display unit. The game streaming assistant ensures low-latency, high-frame-rate image transmission to the VR display unit. This ensures low latency and high frame rate operation of the system, reducing issues such as stuttering and image tearing, and providing users with a smooth and natural interactive experience.

[0040] In this embodiment, the VR display unit further includes an interactive prompt module, used to provide feedback on gesture actions, operation commands, and function activation results in the form of a visual layer within the VR visual interface. Thus, through a real-time displayed visual layer in the VR environment, the current gesture state, its corresponding function, and its activation result can be provided to the user, helping the user intuitively understand the system's recognition and response to gestures, effectively avoiding misoperations and recognition errors.

[0041] In other embodiments, this application may further incorporate signals of more dimensions, such as voice control and computer vision gesture tracking. Voice control allows users to complete non-gesture operations via voice commands, while computer vision-based gesture tracking assists in capturing gestures in a wider range of scenarios. These modalities, working in conjunction with existing signals, can cover different operational scenarios and needs within the system, avoiding the functional limitations of single or limited modal interactions, ultimately forming an interactive system that is closer to natural human interaction habits and has more comprehensive functions. Those skilled in the art can choose the type of multimodal data according to the actual situation, without specific limitations.

[0042] This application integrates Blender modeling software, SteamVR streaming technology, OVR Toolkit interface mapping, and electromyography armband gesture control to construct an immersive, highly flexible VR design system. This system can be applied not only to 3D modeling but also to VR equipment training, virtual assembly, and other scenarios, demonstrating excellent cross-domain transfer capabilities and providing users with a more natural and efficient interactive experience.

[0043] Based on the same inventive concept, this application also provides a virtual reality design method based on multimodal data gesture recognition. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the virtual reality design method based on multimodal data gesture recognition provided below can be found in the limitations of the virtual reality design system based on multimodal data gesture recognition described above, and will not be repeated here.

[0044] See appendix Figure 4 This application also provides a virtual reality design system based on multimodal data gesture recognition, the steps of which are as follows: S100: Collects multimodal data generated by the user's hand movements and recognizes the user's hand gestures based on the multimodal data; S200: Maps hand gestures to operation commands and controls the modeling software to generate an updated virtual reality display interface based on the operation commands; S300: Receives and displays the updated virtual reality display interface through the VR display unit.

[0045] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 5 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection.

[0046] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0047] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0048] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0049] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0050] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0051] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0052] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0053] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0054] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A virtual reality design system based on multimodal data gesture recognition, characterized in that, include: A gesture recognition unit is used to collect multimodal data generated by the user's hand movements and to recognize the user's gesture movements based on the multimodal data; A control unit, connected to the gesture recognition unit, is used to map the gesture action into an operation command, and control the modeling software to generate an updated virtual reality display interface according to the operation command; The VR display unit is connected to the control unit and is used to receive and display the updated virtual reality display interface.

2. The virtual reality design system based on multimodal data gesture recognition according to claim 1, characterized in that, The multimodal data includes electromyography (EMG) signals and IMU signals.

3. The virtual reality design system based on multimodal data gesture recognition according to claim 2, characterized in that, The gesture recognition unit includes: An electromyography (EMG) sensor is used to acquire the electrical signals from the muscles. An IMU sensor is used to acquire the IMU signal. The core processing module, connected to the electromyography sensor, the IMU sensor, and the control unit, is used to recognize the user's gesture based on the electromyography signal, the IMU signal, and a preset gesture recognition model, and to transmit the gesture to the control unit.

4. The virtual reality design system based on multimodal data gesture recognition according to claim 3, characterized in that, The gesture recognition unit is a myoelectric armband, worn on the user's forearm.

5. The virtual reality design system based on multimodal data gesture recognition according to claim 1, characterized in that, The control unit has a built-in preset gesture library to map the gesture actions into corresponding operation commands.

6. The virtual reality design system based on multimodal data gesture recognition according to claim 1, characterized in that, The VR display unit also includes an interactive prompt module, which is used to provide feedback on the gestures, operation commands, and function activation results in the form of visual layers in the VR visual interface.

7. A virtual reality design method based on multimodal data gesture recognition, characterized in that, include: Collect multimodal data generated by the user's hand movements, and recognize the user's hand gestures based on the multimodal data; The gestures are mapped to operation commands, and the modeling software is controlled to generate an updated virtual reality display interface based on the operation commands. The updated virtual reality display interface is received and displayed through the VR display unit.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the virtual reality design method based on multimodal data gesture recognition as described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the virtual reality design method based on multimodal data gesture recognition as described in claim 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the virtual reality design method based on multimodal data gesture recognition as described in claim 7.

Images