Multi-gesture interaction method and system based on human-computer interaction technology

Abstract

According to the multi-gesture interaction method and system based on the human-computer interaction technology, a user wears a head-mounted display, gesture actions of the user are captured through a camera or a depth sensor of the head-mounted display, and the captured gesture actions are analyzed into interaction instructions; and the head-mounted display sends the interaction instruction to the equipment, and the equipment identifies the interaction instruction and executes corresponding operation on the model according to the interaction instruction. Through multi-gesture recognition and interaction control, all-around functions of model grabbing, rotating, zooming, disassembling, resetting and menu management are achieved. The method is smooth in operation and easy to use, and supports seamless switching of various scenes. Through the compatibility with the head-mounted display device and various input devices, the user can freely control the virtual content in a single-person or multi-person cooperation mode, and the interaction efficiency and immersion are improved.

Description

Technical Field

[0001] The present invention relates to human-computer interaction technology in the field of virtual reality and augmented reality (VR / AR), and in particular to a multi-gesture interaction method and system based on human-computer interaction technology, which is suitable for the interaction and operation of models in virtual reality (VR) devices and augmented reality (AR) devices. Background Art

[0002] With the development of virtual reality and augmented reality technologies, users need more natural and efficient ways to interact with virtual content. Traditional control methods such as handles and voice commands have certain limitations, such as unintuitive control or insufficient precision. Interaction by recognizing user gestures can simulate physical actions in reality and provide a more intuitive operation method.

[0003] Current issues: Problem 1. Single function, difficult to meet complex needs: Most current gesture recognition systems only support basic grabbing or rotating operations, with relatively simple functions and unable to cope with complex operation scenarios. For example, in design, simulation, or education and training, users may need to complete multiple operations at the same time, such as grabbing, rotating, and then disassembling or scaling. Smooth switching between these operations has not been well supported in existing systems.

[0004] Problem 2: The gesture definition is confusing and accidental touches are likely to occur: Some gesture recognition systems lack standardization in gesture definitions, which may cause users to accidentally touch or execute wrong commands due to similar gestures during operation. This problem not only affects the stability of interaction, but also reduces the user experience quality and increases the complexity of operation. Summary of the invention

[0005] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a multi-gesture interaction method and system based on human-computer interaction technology, which supports users to grab, rotate, scale, disassemble and reset models through multiple gestures, and can seamlessly switch between different operation scenarios. This method and system can reduce the occurrence of false touches by standardizing gesture definition and recognition logic, and enhance the flexibility and accuracy of operations, thereby improving the interaction efficiency and immersive experience of users in virtual environments.

[0006] The technical solution adopted by the present invention to solve its technical problem is: A multi-gesture interaction method based on human-computer interaction technology, wherein a user wears a head-mounted display (HMD), captures the user's gestures through a camera or depth sensor of the head-mounted display (HMD), and interprets the captured gestures as interaction instructions; the head-mounted display (HMD) sends the interaction instructions to a device, and the device recognizes the interaction instructions and performs corresponding operations on the model according to the interaction instructions (such as grabbing, scaling, disassembling, resetting, and calling out a menu model).

[0007] A multi-gesture interaction system based on human-computer interaction technology, the multi-gesture interaction system includes a gesture recognition module, a model loading and management module, an interaction control module, a user interface and a menu system, and the modules work together to ensure smooth loading and operation of virtual models; wherein: Model loading and management module: supports model loading in multiple formats (JPG images, PNG images, MP4 videos and 3D models). Users can select and load different models through the menu and manage the display status of the model (show, hide, delete); Gesture recognition module: uses a camera or depth sensor to capture user gestures (such as grabbing, zooming, and disassembling); interprets the captured gestures into interactive commands; Interaction control module: performs corresponding operations (such as grabbing, scaling, disassembling, resetting, and calling up menu models) according to the interaction instructions. The system ensures the real-time operation and synchronizes the state changes of the virtual model. User interface and menu system: Users use specific gestures to call out menus and control menus to manage models and set up the system; the menu is displayed at the user's fingertips and can realize multi-function switching (such as loading new models, entering multiplayer collaboration mode, or resetting the scene).

[0008] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a multi-gesture interaction method based on human-computer interaction technology is implemented.

[0009] A storage medium stores a computer program, which, when executed by a processor, implements a multi-gesture interaction method based on human-computer interaction technology.

[0010] The present invention also has the following additional technical features: As a further specific optimization of the technical solution of the present invention: a gesture recognition module is pre-installed in a head mounted display (HMD); a model loading and management module, an interactive control module, and a user interface and menu system are pre-installed in the device.

[0011] As a further specific optimization of the technical solution of the present invention: gesture actions include but are not limited to grab gestures, zoom gestures, disassembly gestures, reset gestures and menu calling gestures.

[0012] As a further specific optimization of the technical solution of the present invention: the grabbing gesture is that the user completes the grabbing action by pinching the thumb and index finger; the zooming gesture is that the user uses both hands to grab at the same time and changes the distance between the two hands; the disassembly gesture is that the user makes a pistol gesture with both hands and pulls them outward or moves them inward; the reset gesture is that the user stretches out both palms, palms facing outwards, and palms tilted relative to each other; the menu calling gesture is that the user stands up the left palm, palm facing inwards, and pinches the thumb and index finger.

[0013] As a further specific optimization of the technical solution of the present invention: the process of parsing the captured gesture actions into interaction instructions is: the camera or depth sensor of the head-mounted display (HMD) captures the user's gesture actions to take photos or videos to obtain gesture action images, and pre-processes the gesture action images to obtain pre-processed images; extracts feature information from the pre-processed images, identifies the feature information and converts it into instructions to obtain interaction instructions.

[0014] As a further specific optimization of the technical solution of the present invention: when the gesture action is an interaction instruction of a grabbing gesture, the interaction control module can move the model; when the gesture action is an interaction instruction of a zooming gesture, the interaction control module can change and adjust the size of the model; when the gesture action is an interaction instruction of a disassembly gesture, the interaction control module can disassemble the model or merge the model; when the gesture action is an interaction instruction of a reset gesture, the interaction control module can reset the initial state of the model; when the gesture action is an interaction instruction of a menu calling gesture, the interaction control module can call out the menu, and the menu is displayed at the fingertip position of the menu calling gesture.

[0015] The present invention solves the problems of single gesture function, confusing gesture definition, and easy accidental touches, and realizes flexible interactive operations between users and virtual content. The system of the present invention can accurately identify multiple gestures and standardize gesture definitions to reduce accidental touches. At the same time, by optimizing the gesture switching logic, seamless interaction in complex operation scenarios is achieved, improving the user experience. Specifically, compared with the prior art, the present invention has the following advantages: Advantage 1: Efficient and intuitive interactive experience: The present invention uses multi-gesture recognition technology to achieve natural interaction between users and virtual models. Users can control virtual content like operating objects in real life through gestures such as grabbing, scaling, rotating, and disassembling, providing a highly intuitive and immersive operating experience.

[0016] Advantage 2: Rich and diverse gesture operations: The system supports multiple gesture interactions such as grabbing, rotating, scaling, disassembling, merging, resetting and menu calling, achieving comprehensive coverage from simple operations to complex scenes. Users can freely switch gestures according to different needs to achieve efficient virtual model operation.

[0017] Advantage 3: No additional control device is required: Users can complete all interactions through natural gestures, which greatly simplifies device requirements and operation processes.

[0018] Advantage 4: Reduce false touches and improve recognition accuracy: By standardizing gesture definitions and optimizing recognition logic, the system can accurately distinguish different gestures, reduce false touches, and ensure the accuracy and stability of interaction. This feature greatly improves the user experience and makes the operation smoother and more reliable.

[0019] Advantage 5, good compatibility: The multi-gesture interaction system of the present invention can run on mainstream virtual reality (VR) and augmented reality (AR) devices, such as Quest, PICO, VisonPro, Rokid, HTC Vive, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the grabbing model gesture interaction of the present invention; Figure 2 This is a schematic diagram of gesture interaction of a reduced model of the present invention; Figure 3 A schematic diagram of the gesture interaction of the enlarged model of the present invention; Figure 4 This is a schematic diagram of the gesture interaction of the disassembly model of the present invention; Figure 5 This is a schematic diagram of the gesture interaction of the merged model of the present invention; Figure 6 A schematic diagram of the gesture interaction of the reset model of the present invention; Figure 7 A schematic diagram of the gesture interaction for calling out a menu of the present invention; Figure 8 A schematic diagram of the gesture interaction for grabbing an object according to the present invention; Fig. 9 A schematic diagram of the gesture interaction for grabbing an object according to the present invention; Fig.10 This is a schematic diagram of the gesture interaction for zooming in and out of an object according to the present invention; Fig.11 This is a schematic diagram of the gesture interaction for zooming in and out of an object according to the present invention; Fig.12 A schematic diagram of the gesture interaction for calling out a menu of the present invention; Fig.13 A schematic diagram of the gesture interaction for calling out a menu of the present invention; Fig.14 It is a schematic diagram of the system relationship structure of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0022] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0023] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0025] A multi-gesture interaction method based on human-computer interaction technology comprises the following steps: The user wears a head-mounted display (HMD), and the user's gestures are captured by a camera or a depth sensor of the head-mounted display (HMD), and the captured gestures are parsed into interaction instructions.

[0026] The head-mounted display (HMD) sends interaction commands to the device, which recognizes the interaction commands and performs corresponding operations on the model according to the interaction commands (such as grabbing, scaling, disassembling, resetting, and calling up the menu model).

[0027] In some of the embodiments, a gesture recognition module is pre-installed in a head mounted display (HMD).

[0028] In some of the embodiments, a model loading and management module, an interactive control module, and a user interface and menu system are pre-installed in the device.

[0029] Gesture actions include, but are not limited to, grab gestures, zoom gestures, dismantle gestures, reset gestures, and menu-calling gestures.

[0030] The grab gesture is for the user to pinch the thumb and index finger to complete the grabbing action; the zoom gesture is for the user to grab with both hands at the same time and change the distance between the hands; the disassembly gesture is for the user to make a pistol gesture with both hands and pull them outward or bring them closer together; the reset gesture is for the user to stretch out both palms, palms facing outwards, and palms tilted relative to each other; the menu calling gesture is for the user to stand up the left palm, palm facing inwards, and pinch the thumb and index finger together.

[0031] The process of parsing the captured gesture actions into interaction instructions is as follows: the camera or depth sensor of the head mounted display (HMD) captures the user's gesture actions to take photos or videos to obtain gesture action images, and the gesture action images are preprocessed to obtain preprocessed images; feature information is extracted from the preprocessed images, the feature information is identified and converted into instructions to obtain interaction instructions.

[0032] When the gesture action is an interaction instruction of a grab gesture, the interaction control module can move the model; when the gesture action is an interaction instruction of a zoom gesture, the interaction control module can change and adjust the size of the model; when the gesture action is an interaction instruction of a disassembly gesture, the interaction control module can disassemble the model or merge the model; when the gesture action is an interaction instruction of a reset gesture, the interaction control module can reset the initial state of the model; when the gesture action is an interaction instruction of a menu call-out gesture, the interaction control module can call out the menu, and the menu is displayed at the fingertip position of the menu call-out gesture.

[0033] A multi-gesture interaction system based on human-computer interaction technology includes a gesture recognition module, a model loading and management module, an interaction control module, a user interface and a menu system. The modules work together to ensure smooth loading and operation of the virtual model.

[0034] Model loading and management module: supports loading models in multiple formats (JPG images, PNG images, MP4 videos and 3D models). Users can select and load different models through the menu and manage the display status of the model (show, hide, delete).

[0035] Gesture recognition module: uses a camera or depth sensor to capture user gestures (such as grabbing, zooming, and disassembling); and interprets the captured gestures into interactive commands.

[0036] Interaction control module: Execute corresponding operations (such as grabbing, scaling, disassembling, resetting, and calling up menu models) according to the interaction instructions. The system ensures the real-time operation and synchronizes the state changes of the virtual model.

[0037] User interface and menu system: Users use specific gestures to call out menus and control menus to manage models and set up the system; the menu is displayed at the user's fingertips and can realize multi-function switching (such as loading new models, entering multiplayer collaboration mode, or resetting the scene).

[0038] The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a multi-gesture interaction method based on human-computer interaction technology is implemented.

[0039] In one embodiment, the electronic device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-gesture interaction method based on human-computer interaction technology is implemented.

[0040] A storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-gesture interaction method based on human-computer interaction technology. A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods.

[0041] In one embodiment, any reference to memory, storage, database or other medium used may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0042] Figure 8This is a schematic diagram of the system relationship structure of the present invention. It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] In one embodiment, the crawling model (e.g. Figure 1 shown): The user pinches the tips of his thumb and index finger together, and the system recognizes the gesture as a "grab" command.

[0044] When the user gestures close to the model and pinches, the system binds the model to the user gesture, allowing the user to adjust the model position by moving the gesture.

[0045] In one embodiment, the scaling model (e.g. Figure 2 and Figure 3 shown): The user grabs the model with both hands at the same time and adjusts the size of the model by changing the distance between the two hands.

[0046] When the hands are pulled apart, the model is enlarged; when the hands are brought closer, the model is reduced. The system synchronizes the changes in the distance between the hands in real time to ensure smooth operation.

[0047] In one embodiment, the split and merge models (e.g. Figure 4 and Figure 5 shown): The user makes a "pistol" gesture with both hands and pulls them apart. The system recognizes this gesture as a "disassemble" command and disassembles the model into sub-components.

[0048] When the hands are brought together and into the "pistol" position again, the system executes the "merge" command to reassemble the sub-components of the model into a whole.

[0049] In one embodiment, resetting the model (e.g. Figure 6 shown): The user triggers the initial state reset of the model by extending both palms, with the palms facing outwards, or with the palms tilted towards each other.

[0050] The system clears all changes to the model and restores it to the default state when it was loaded.

[0051] In one embodiment, a menu is called up (eg Figure 7 shown): The user stands up his left hand with the palm facing inwards and pinches the tips of his thumb and index finger together. The system calls out the menu, which is displayed at the fingertips.

[0052] Users can perform operations such as model loading, mode switching or system settings by clicking on options in the menu with their index finger.

[0053] In one embodiment, to initiate a connection with a device: The user wears a head-mounted display (HMD) device and captures gestures through a camera or depth sensor.

[0054] The system enters monitoring mode and captures the user's gestures and movements in real time.

[0055] In one embodiment, gesture mapping and operation execution: When the system recognizes the user's gesture, it maps it to a corresponding command, such as "grab" or "zoom".

[0056] When the user performs a grab gesture, the model can be dragged and repositioned; when the user performs a zoom gesture, the model can be dynamically resized.

[0057] In one embodiment, the multi-function menu switches: The user calls out the menu through gestures and selects different models or operation modes in the menu.

[0058] The system loads a new model or switches to multiplayer collaboration mode based on the user's selection.

[0059] In one embodiment, multiple people collaborate and synchronize operations: Users can enter multiplayer mode through the menu, and the system will synchronize the user's gestures and model operations to other participants' devices.

[0060] In a multi-person scenario, any user's operations on the model will be synchronized in real time to ensure consistency of collaboration.

[0061] In one embodiment, grabbing an object: To "grasp" an object, stretch out your palm and approach the virtual object, then clench your palm to grasp the object.

[0062] In one embodiment, to scale an object: "Grasp" to scale the object. Grasp the object with both hands at the same time. Control the scaling of the object by detecting the distance between the two hands. Bringing the hands closer will shrink the object, and pulling the hands apart will enlarge the object.

[0063] In one embodiment, the menu is called up: Make a fist, then open it, palm facing inward, to bring up the menu.

[0064] In summary, the present invention realizes the full range of functions of model grabbing, rotating, scaling, disassembling, resetting and menu management through multi-gesture recognition and interactive control. The system of the present invention operates smoothly, is easy to use, and supports seamless switching of multiple scenes. Through the compatibility with head-mounted display (HMD) devices and multiple input devices, the present invention allows users to freely manipulate virtual content in single-person or multi-person collaborative mode, thereby improving interaction efficiency and immersion.

[0065] Those skilled in the art should understand that the technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A multi-gesture interaction method based on human-computer interaction technology, characterized in that: The user wears a head-mounted display, captures the user's gestures through a camera or a depth sensor of the head-mounted display, and interprets the captured gestures into interaction instructions; The head-mounted display sends the interaction instructions to the device, and the device recognizes the interaction instructions and performs corresponding operations on the model according to the interaction instructions.

2. The multi-gesture interaction method based on human-computer interaction technology according to claim 1, characterized in that: The head mounted display is pre-installed with a gesture recognition module; the device is pre-installed with a model loading and management module, an interactive control module, and a user interface and menu system.

3. The multi-gesture interaction method based on human-computer interaction technology according to claim 1, characterized in that: The gesture actions include but are not limited to grabbing gestures, zooming gestures, disassembling gestures, resetting gestures and calling out menu gestures.

4. The multi-gesture interaction method based on human-computer interaction technology according to claim 2, characterized in that: The grabbing gesture is that the user completes the grabbing action by pinching the thumb and index finger tips together; the zooming gesture is that the user grabs with both hands at the same time and changes the distance between the two hands; the disassembly gesture is that the user makes a pistol gesture with both hands and pulls them outward or moves them inward; the reset gesture is that the user stretches out both palms, palms facing outwards, and palms tilted relative to each other; the menu calling gesture is that the user stands up the left palm, palm facing inwards, and pinches the thumb and index finger together.

5. The multi-gesture interaction method based on human-computer interaction technology according to claim 1, characterized in that: The process of parsing the captured gesture action into the interaction instruction is as follows: the camera or depth sensor of the head mounted display captures the user's gesture action to take a photo or video to obtain a gesture action image, and pre-processes the gesture action image to obtain a pre-processed image; Feature information is extracted from the preprocessed image, the feature information is identified and converted into instructions, and the interactive instructions are obtained.

6. The multi-gesture interaction method based on human-computer interaction technology according to claim 2, characterized in that: When the gesture action is an interaction instruction of a grab gesture, the interaction control module can move the model; when the gesture action is an interaction instruction of a zoom gesture, the interaction control module can change the size of the model; when the gesture action is an interaction instruction of a disassembly gesture, the interaction control module can disassemble the model or merge the model; when the gesture action is an interaction instruction of a reset gesture, the interaction control module can reset the initial state of the model; When the gesture action is an interactive instruction for invoking a menu gesture, the interactive control module may invoke a menu, and the menu is displayed at the fingertip position of the menu invoking gesture.

7. A multi-gesture interaction system based on human-computer interaction technology, characterized in that: The multi-gesture interaction system includes a gesture recognition module, a model loading and management module, an interaction control module, a user interface and a menu system. The modules work together to ensure smooth loading and operation of the virtual model. The model loading and management module supports loading models in multiple formats. Users can select and load different models through the menu and manage the display status of the models. The gesture recognition module: uses a camera or a depth sensor to capture the user's gestures; and interprets the captured gestures into interaction instructions; The interactive control module: performs corresponding operations according to the interactive instructions. The system ensures the real-time operation and synchronizes the state changes of the virtual model. The user interface and menu system: the user calls out menus and controls menus through specific gestures to perform model management and system settings; the menu is displayed at the user's fingertips, and the menu can realize multi-function switching.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the multi-gesture interaction method based on human-computer interaction technology according to any one of claims 1 to 6 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-gesture interaction method based on human-computer interaction technology as described in any one of claims 1 to 6 is implemented.

Images