Human-computer interaction method and human-computer interaction system

Abstract

The present invention discloses a human-computer interaction method, namely a human-computer interaction system, the interaction method comprising: obtaining the positioning information and gesture information of the user's hand; determining the content to be operated by the virtual operator in the display content of the virtual reality display device according to the obtained positioning information; determining the corresponding interactive operation mode according to the obtained gesture information; executing and responding to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode. The technical solution of the present invention is applied to solve the problem that the existing interactive handle buttons are insufficient and cannot realize more interactive modes required by users.

Description

Technical Field

[0001] The present invention relates to a human-computer interaction technology, and in particular to a human-computer interaction method and a human-computer interaction system for a virtual reality display device. Background Art

[0002] With the development of three-dimensional display technology and virtual reality technology, more and more technical teams are developing corresponding human-computer interaction technologies for related new display technologies. The human-computer interaction method currently used in the market for virtual reality display devices, after determining the interactive content to be operated, usually uses the movement of the interactive handle (or interactive pen) and its buttons to grab, put down, move and rotate the interactive items in the displayed content to achieve human-computer interaction. This method is implemented by binding the buttons on the handle to the corresponding trigger events on the content platform. However, the buttons on the general handle are limited. If you want to realize more functions such as zooming in, zooming out, entering, exiting, turning pages, full screen, etc., it seems stretched. Seriously affecting the user experience.

[0003] It can be seen that the existing interaction method has the defect of not being able to realize more interaction methods required by users due to insufficient buttons on the interaction handle, and needs to be improved. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a human-computer interaction method and a human-computer interaction system to solve the problem that the existing interactive handle has insufficient buttons and cannot realize more interactive modes required by users.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0006] The present invention provides a human-computer interaction method for realizing interaction between a user and a virtual reality display device, comprising:

[0007] Obtain the user's hand positioning information and gesture information;

[0008] Determining, according to the acquired positioning information, content to be operated by the virtual operator in content displayed by the virtual reality display device;

[0009] Determining a corresponding interactive operation mode according to the acquired gesture information;

[0010] Execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0011] Furthermore, the positioning information includes: position information and rotation information; the step of obtaining the positioning information of the user's hand specifically includes:

[0012] capturing an image of a rigid body bound to the user's hand through an optical motion capture system to obtain position information and rotation information of the user's hand; or,

[0013] The image of the marking point bound to the user's hand is captured by an optical motion capture system to obtain the position information of the user's hand; and the rotation information of the user's hand is obtained by an inertial system.

[0014] Furthermore, the step of obtaining the user's hand gesture information specifically includes:

[0015] Using the optical motion capture system to capture an image of a rigid body bound to the user's hand to obtain gesture information of the user's hand; or;

[0016] The image of the user's hand is captured by a gesture recognition system to obtain gesture information of the user's hand.

[0017] Furthermore, the step of determining the content to be operated by the virtual operator in the content displayed by the virtual reality display device according to the acquired positioning information specifically includes:

[0018] Determining the position to be operated of the virtual operator in the displayed content according to the acquired positioning information of the user's hand and a preset position mapping relationship;

[0019] Determine the content to be operated corresponding to the position to be operated.

[0020] Furthermore, the step of determining the corresponding interactive operation mode according to the acquired gesture information specifically includes:

[0021] According to the acquired gesture information, determining the interactive operation mode corresponding to the gesture information according to a preset mapping relationship between gesture information and operation mode; or,

[0022] According to the acquired gesture information, the specific key on the keyboard corresponding to the gesture information is determined according to the preset mapping relationship between gesture information and keyboard keys, and the interactive operation mode corresponding to the specific key is determined according to the preset mapping relationship between keyboard keys and operation modes.

[0023] Furthermore, the virtual reality display device includes: a 3D large-screen virtual reality display device, a 3D desktop display device or a 3D head-mounted display device.

[0024] Furthermore, when the virtual reality display device includes a 3D large-screen virtual reality display device, the method further includes:

[0025] Acquiring positioning information of the 3D interactive glasses worn by the user through the optical motion capture system;

[0026] According to the positioning information of the 3D interactive glasses, the display viewing angle of the content displayed by the 3D large-screen virtual reality display device is adjusted in real time.

[0027] The present invention also provides a human-computer interaction system, comprising:

[0028] Virtual reality display device, used to display image content;

[0029] Optical motion capture system, used to obtain the position information and gesture information of the user's hand;

[0030] An inertial system, used to obtain rotation information of the user's hand;

[0031] The data processing device is used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device according to the position information obtained by the optical motion capture system and the rotation information of the user's hand obtained by the inertial system; and determine the interactive operation mode corresponding to the gesture information according to the gesture information obtained by the optical motion capture system; and execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0032] Accordingly, the present invention also provides a human-computer interaction system, comprising:

[0033] Virtual reality display device, used to display image content;

[0034] Optical motion capture system, used to obtain the position information of the user's hands;

[0035] An inertial system, used to obtain rotation information of the user's hand;

[0036] A gesture recognition system, used to obtain the gesture information of the user's hand;

[0037] The data processing device is used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device according to the position information obtained by the optical motion capture system and the rotation information of the user's hand obtained by the inertial system; and determine the interactive operation mode corresponding to the gesture information according to the gesture information obtained by the gesture recognition system; and execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0038] Accordingly, the present invention also provides a human-computer interaction system, comprising:

[0039] Virtual reality display device, used to display image content;

[0040] Optical motion capture system, used to obtain the user's hand positioning information and gesture information;

[0041] A data processing device is used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device according to the positioning information obtained by the optical motion capture system; and to determine the interactive operation mode corresponding to the gesture information according to the gesture information obtained by the optical motion capture system; and to execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0042] The present invention provides a human-computer interaction method and a human-computer interaction system. When a user interacts with a virtual reality display device, the user no longer relies on a traditional interactive handle or an interactive pen. Therefore, the interactive operation mode is no longer limited by the number of physical buttons on the interactive handle. Moreover, since the user can make a variety of different gestures to control the movement of the virtual operator in real time, more interactive operations that meet the user's needs can be performed. Therefore, the human-computer interaction method of the present invention can get rid of the constraints of the interactive handle and the interactive pen, and can realize more human-computer interaction functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the first embodiment of the human-computer interaction method provided by the present invention.

[0044] Figure 2 This is a flow chart of the second embodiment of the human-computer interaction method provided by the present invention.

[0045] Figure 3a-3f A schematic diagram of gesture information provided by an embodiment of the present invention.

[0046] Figure 4 This is a schematic flow chart of a third embodiment of the human-computer interaction method provided by the present invention.

[0047] Figure 5 This is a structural block diagram of the first embodiment of the human-computer interaction system provided by the present invention.

[0048] Figure 6 This is a structural block diagram of the second embodiment of the human-computer interaction system provided by the present invention.

[0049] Figure 7 This is a structural block diagram of the third embodiment of the human-computer interaction system provided by the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0051] Based on the problem that the current user-virtual reality interactive device interaction method is limited by the defects of the interactive handle, resulting in the interaction mode being insufficient to meet the user's more interactive needs, the present invention provides a human-computer interaction method for realizing the interaction between the user and the virtual reality display device, and the virtual reality display device may include: a 3D large-screen virtual reality display device, a 3D desktop display device, or a 3D head-mounted display device. In other words, the human-computer interaction method of the present invention can be applied to 3D large-screen human-computer interaction, 3D desktop human-computer interaction (such as zspace desktop screen interaction), or head-mounted human-computer interaction (such as AR head-mounted display device or VR head-mounted display device).

[0052] like Figure 1 As shown, the core of the human-computer interaction method proposed by the present invention includes the following steps:

[0053] Step S100, pre-set the mapping relationship between gesture information and interactive operation mode. The interactive operation mode may include: grab, drop, move, rotate, zoom in, zoom out, enter, exit, page turn, full screen, etc.

[0054] There are two ways to pre-set the mapping relationship between gesture information and interactive operation modes, which will be described in detail in subsequent embodiments.

[0055] Step S101, obtaining the positioning information and gesture information of the user's hand.

[0056] During the interaction, the gesture information and positioning information of the user's hand can be obtained through a motion capture system (such as an optical motion capture system or an inertial motion capture system), wherein the positioning information may include position information and rotation information.

[0057] Step S102: determining the content to be operated by the virtual operator in the content displayed by the virtual reality display device according to the acquired positioning information.

[0058] After obtaining the positioning information of the user's hand, the six degrees of freedom information of the user's hand can be calculated through a fusion algorithm such as Kalman filtering, and the six degrees of freedom information can be input into a content platform such as the Unity platform in real time, and finally presented to the content displayed by the virtual reality display device. In the content displayed by the virtual reality device, there is a virtual operator, which corresponds to the user's hand. When the six degrees of freedom information of the user's hand changes, the six degrees of freedom of the virtual operator also change, that is, the user can control the movement of the virtual operator in the content displayed by the virtual reality display device in real time by moving his own hand.

[0059] Step S103: determining the interactive operation mode corresponding to the gesture information according to the acquired gesture information.

[0060] Step S104, executing and responding to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0061] The benefit of such interaction in the embodiment of the present invention is that the user no longer relies on the traditional interactive handle or interactive pen when performing interactive operations with the virtual reality display device, so the interactive operation mode is no longer limited by the number of physical buttons on the interactive handle; and, because the user can make a variety of different gestures to control the movement of the virtual operating hand in real time, more interactive operations that meet the user's needs can be performed. Therefore, the human-computer interaction method of the present invention can get rid of the constraints of the interactive handle and the interactive pen, and can realize more human-computer interaction functions.

[0062] There can be different ways to obtain the positioning information and gesture information of the user's hand. Among them, the positioning information includes: position information and rotation information. For example, when obtaining the positioning information of the user's hand, the position information and rotation information of the user's hand can be obtained simultaneously through the optical motion capture system; or, the position information of the user's hand can be obtained through the optical motion capture system, and the rotation information of the user's hand can be obtained through the inertial system. For another example, when obtaining the gesture information of the user's hand, it can be directly obtained through the aforementioned optical motion capture system, or a gesture recognition system can be added. The added gesture recognition system is specifically used to obtain the gesture information of the user's hand. Below, it will be described in detail according to different situations.

[0063] like Figure 2 FIG. 2 is a flow chart of a second embodiment of the human-computer interaction method provided by the present invention. In this embodiment, an image of a rigid body bound to a user's hand is captured by an optical motion capture system to obtain position information and rotation information of the user's hand without adding an inertial system. This embodiment will be described in detail below.

[0064] Figure 2 The human-computer interaction method shown specifically includes:

[0065] Step S200, pre-set the mapping relationship between gesture information and interactive operation mode. The interactive operation mode may include: grab, drop, move, rotate, zoom in, zoom out, enter, exit, page turn, full screen, etc.

[0066] In this step, there are two ways to implement it.

[0067] The first method is to preset the mapping relationship between gesture information and interactive operation mode and store it in a mapping table. For example, when the gesture is Figure 3a , the corresponding operation mode can be a zoom operation. Figure 3c , the corresponding operation mode can be a zoom-out operation. Figure 3e , the corresponding operation mode may be a grab operation. In a similar manner, different interactive operation modes may be set for different gestures. Another way is to not directly preset the correspondence between gesture information and interactive operation modes, but to preset the mapping relationship between gesture information and keyboard keys, and then preset the mapping relationship between keyboard keys and interactive operation modes, and store the corresponding mapping relationship in a mapping table. For example, when the gesture is Figure 3a When the gesture is , the corresponding keyboard key is A, and the operation mode corresponding to A can be a zoom operation; when the gesture is Figure 3c When the gesture is , the corresponding keyboard key is S, and the corresponding operation mode can be a zoom-out operation. Figure 3e When , the corresponding keyboard key is D, and the corresponding operation mode may be a grab operation. In a similar manner, different keyboard keys corresponding to different gestures and different interactive operation modes corresponding to different keys may be set.

[0068] Step S201: capture an image of a rigid body bound to a user's hand through an optical motion capture system to obtain position information and rotation information of the user's hand.

[0069] That is, what is described in this step is a method of obtaining the position information and rotation information of the user's hand only through the optical motion capture system. The specific implementation method is as follows:

[0070] The optical motion capture system includes: two or more optical motion capture cameras and multiple markers, some of which can form a rigid body. The rigid body composed of the markers is pasted on the back of the user's hand, and the remaining markers are pasted on the user's fingers.

[0071] At the beginning of capture, multiple optical motion capture cameras capture images of the user's hands and transmit the captured images to a data processing device. By processing the images, the positions of the marker points on the user's hands are identified, and then the position information (x, y, z) and rotation information (yaw, roll, pitch) of the user's hands can be identified.

[0072] Step S202, obtaining the user's hand gesture information.

[0073] In this step, when obtaining the gesture information of the user's hand, there are two operation methods. One is: directly use the optical motion capture system in step S201 to obtain the image of the rigid body bound to the user's hand to obtain the gesture information of the user's hand. That is to say, when the optical motion capture system obtains the user's positioning information, it also obtains the gesture information of the user's hand. The advantage of this method is that there is no need to add an additional gesture recognition system. Of course, there are also shortcomings when performing gesture recognition in this way: for example, the brightness adjustment of the optical motion capture camera is restricted. When the optical motion capture camera is adjusted to a brightness that is better for rigid body capture (this brightness is relatively dark, used to remove background environmental interference, and try to retain the grayscale of the light ball and remove the background grayscale), it is too dark for gesture recognition, so it can only recognize gestures at a relatively close distance, such as Figure 3a-3f However, when the brightness of the motion capture camera is adjusted to a better range for gesture recognition (i.e., when the brightness is adjusted to a slightly higher level to capture gestures farther away), a lot of unnecessary background light will be introduced into the rigid body recognition, interfering with the rigid body or marker capture. Therefore, when using this method to obtain positioning information and gesture information, it is necessary to balance the brightness of the optical motion capture camera well.

[0074] Another way is to obtain the gesture information of the user's hand through another gesture recognition system. In order to ensure the normal use of the optical motion capture system and the gesture recognition system in the same application scenario, the working wavelengths of the optical motion capture camera in the optical motion capture system and the camera in the gesture recognition system are required to be different. For example, when the working wavelength of the optical motion capture camera in the optical motion capture system is 940nm, the working wavelength of the camera in the gesture recognition system is 850nm. Among them, the specific presentation form of the gesture recognition system can be a gesture recognition device, such as a single ordinary camera, a single binocular camera, a single TOF camera, etc. All cameras must be calibrated before use (including optical motion capture cameras and cameras for user gesture recognition) so that the coordinate systems of all cameras are unified, and the corresponding pixels of different cameras shooting the same two-dimensional coordinate point are paired.

[0075] Step S203: determining the position to be operated of the virtual operator in the displayed content according to the acquired positioning information of the user's hand and a preset position mapping relationship.

[0076] Step S204, determining the content to be operated corresponding to the position to be operated.

[0077] Since the position information and rotation information of the user's hand have been obtained in step S202, in step S203, the six-degree-of-freedom information of the user's hand can be calculated by a fusion algorithm such as Kalman filtering, and the six-degree-of-freedom information can be input into a content platform such as Unity platform in real time, and finally presented to the display content of the virtual reality display device. Then, according to the preset position information relationship, the position to be operated and the content to be operated of the virtual operator in the display content can be determined.

[0078] Step S205: determining the corresponding interactive operation mode according to the gesture information obtained in step S202.

[0079] After the specific gesture information is identified in step S202, the preset mapping table of gesture information and operation mode can be searched to obtain the specific operation mode corresponding to the current gesture information. Alternatively, after the specific gesture information is identified in step S202, the mapping table of gesture information and keyboard keys and the mapping table of keyboard keys and operation modes can be searched in sequence to obtain the specific operation mode corresponding to the current gesture information.

[0080] Step S206, executing and responding to the triggering operation of the virtual manipulator treating the operation content in an interactive operation manner.

[0081] In this step, when the gesture is successfully recognized, the corresponding event (or function) can be triggered by the gesture through the preset mapping relationship. For example, after the gesture is successfully recognized, the corresponding key model will be automatically and continuously sent, and the corresponding event (or function) can be triggered by the gesture by matching the key with the trigger event of the virtual reality display device content platform.

[0082] In this embodiment, since there are dozens of commonly used recognizable gestures, this solution can realize multifunctional human-computer interaction, that is, many different functions can be accurately triggered through gesture recognition.

[0083] like Figure 4 , which is a flow chart of the third embodiment of the human-computer interaction method provided by the present invention. In this embodiment, an image of a rigid body bound to a user's hand is captured by an optical motion capture system to obtain position information of the user's hand, and rotation information of the user's hand is obtained by an inertial system. This embodiment will be described in detail below.

[0084] Figure 4 The human-computer interaction method shown comprises:

[0085] Step S400, pre-set the mapping relationship between gesture information and interactive operation mode. The interactive operation mode may include: grab, drop, move, rotate, zoom in, zoom out, enter, exit, page turn, full screen, etc.

[0086] In this step, there are two ways to implement it.

[0087] The first method is to preset the mapping relationship between gesture information and interactive operation mode and store it in a mapping table. For example, when the gesture is Figure 3a , the corresponding operation mode can be a zoom operation. Figure 3c , the corresponding operation mode can be a zoom-out operation. Figure 3e , the corresponding operation mode may be a grab operation. In a similar manner, different interactive operation modes may be set for different gestures. Another way is to not directly preset the correspondence between gesture information and interactive operation modes, but to preset the mapping relationship between gesture information and keyboard keys, and then preset the mapping relationship between keyboard keys and interactive operation modes, and store the corresponding mapping relationship in a mapping table. For example, when the gesture is Figure 3a When the gesture is , the corresponding keyboard key is A, and the operation mode corresponding to A can be a zoom operation; when the gesture is Figure 3c When the gesture is , the corresponding keyboard key is S, and the corresponding operation mode can be a zoom-out operation. Figure 3e When , the corresponding keyboard key is D, and the corresponding operation mode may be a grab operation. In a similar manner, different keyboard keys corresponding to different gestures and different interactive operation modes corresponding to different keys may be set.

[0088] Step S401: Use an optical motion capture system to capture images of marker points bound to the user's hand to obtain position information of the user's hand.

[0089] That is, what is described in this step is that only the position information of the user's hand is obtained through the optical motion capture system. The specific implementation method is as follows:

[0090] The optical motion capture system includes: two or more optical motion capture cameras and multiple markers, which are pasted on the back of the user's hand.

[0091] At the beginning of capture, multiple optical motion capture cameras capture images of the marker points bound to the user's hands and transmit the captured images to a data processing device. By processing the images, the positions of the marker points on the user's hands are identified, and then the position information (x, y, z) of the user's hands can be identified.

[0092] Step S402: Obtain the rotation information of the user's hand through the inertial system.

[0093] The inertial system can be, for example, an IMU module. The specific operation of this step can be to bind a wireless IMU module to the user's wrist or back of the hand, and obtain the rotation information (yaw, roll, pitch) of the user's hand through the wireless IMU module. Then, the obtained rotation information is transmitted to the data processing device.

[0094] Step S403, obtaining the user's hand gesture information.

[0095] In this step, when obtaining the user's gesture information, there are two operation methods. One is: directly use the optical motion capture system in step S401 to obtain the image of the rigid body bound to the user's hand to obtain the gesture information of the user's hand. That is to say, when the optical motion capture system obtains the user's positioning information, it also obtains the user's hand gesture information. The advantage of this method is that there is no need to add an additional gesture recognition system. Of course, this method also has shortcomings when performing gesture recognition: for example, the brightness adjustment of the optical motion capture camera is restricted. When the optical motion capture camera is adjusted to a brightness that is better for rigid body capture (this brightness is relatively dark, used to remove background environmental interference, and try to retain the light ball grayscale and remove the background grayscale), it is too dark for gesture recognition, so it can only recognize gestures at a relatively close distance, such as Figure 3a-3f However, when the brightness of the motion capture camera is adjusted to a better range for gesture recognition (i.e., when the brightness is adjusted to a slightly higher level to capture gestures farther away), a lot of unnecessary background light will be introduced into the rigid body recognition, interfering with the rigid body or marker capture. Therefore, when using this method to obtain positioning information and gesture information, it is necessary to balance the brightness of the optical motion capture camera well.

[0096] Another way is to obtain the gesture information of the user's hand through another gesture recognition system. In order to ensure the normal use of the optical motion capture system and the gesture recognition system in the same application scenario, the working wavelengths of the optical motion capture camera in the optical motion capture system and the camera in the gesture recognition system are required to be different. For example, when the working wavelength of the optical motion capture camera in the optical motion capture system is 940nm, the working wavelength of the camera in the gesture recognition system is 850nm. Among them, the specific presentation form of the gesture recognition system can be a gesture recognition device, such as a single ordinary camera, a single binocular camera, a single TOF camera, etc. All cameras must be calibrated before use (including optical motion capture cameras and cameras for user gesture recognition) so that the coordinate systems of all cameras are unified, and the corresponding pixels of different cameras shooting the same two-dimensional coordinate point are paired.

[0097] Step S404: determining the position to be operated of the virtual operating hand in the displayed content according to the acquired positioning information of the user's hand and a preset position mapping relationship.

[0098] Step S405, determining the content to be operated corresponding to the position to be operated.

[0099] Since the position information and rotation information of the user's hand have been obtained in steps S402 and S403, in step S404, the six-degree-of-freedom information of the user's hand is obtained and bound to the content platform (such as the Unity platform) in real time through the combination of sensor fusion algorithms such as Kalman filtering, and finally presented in the display content of the virtual reality display device, so that the user's hand can control the movement of the virtual operator in real time. Then, according to the pre-set position information relationship, the position to be operated and the content to be operated of the virtual operator in the display content can be determined.

[0100] Step S406: determine the corresponding operation mode according to the gesture information obtained in step S403.

[0101] After the specific gesture information is identified in step S403, the preset mapping table of gesture information and operation mode can be searched to obtain the specific operation mode corresponding to the current gesture information. Alternatively, after the specific gesture information is identified in step S403, the mapping table of gesture information and keyboard keys and the mapping table of keyboard keys and operation modes can be searched in sequence to obtain the specific operation mode corresponding to the current gesture information.

[0102] Step S407, executing and responding to the triggering operation of the virtual manipulator treating the operation content in an interactive operation manner.

[0103] In this step, when the gesture is successfully recognized, the corresponding event (or function) can be triggered by the gesture through the preset mapping relationship. For example, after the gesture is successfully recognized, the corresponding key model will be automatically and continuously sent, and the corresponding event (or function) can be triggered by the gesture by matching the key with the trigger event of the virtual reality display device content platform.

[0104] In this embodiment, since there are dozens of commonly used recognizable gestures, this solution can realize multifunctional human-computer interaction, that is, many different functions can be accurately triggered through gesture recognition.

[0105] According to the above, this human-computer interaction scenario can be applied to the interaction of 3D large-screen virtual reality display devices. Therefore, when applied to 3D large-screen interaction, users also need to wear 3D interactive glasses and paste markers on the 3D interactive glasses. Figure 1 , Figure 2 as well as Figure 4 When the embodiment shown is applied to 3D large screen interaction, the human-computer interaction method further includes:

[0106] The positioning information of the 3D interactive glasses worn by the user is obtained through the optical motion capture system; and the display viewing angle of the content displayed by the 3D large-screen virtual reality display device is adjusted in real time according to the positioning information of the 3D interactive glasses. In specific operations, the position information and rotation information of the marking points on the 3D interactive glasses worn by the user are obtained through the optical motion capture system, thereby obtaining the six-degree-of-freedom information of the 3D interactive glasses, and the six-degree-of-freedom information is connected to the content platform, such as the Unity platform, in real time, so as to realize the binding and output of the position and inclination information of the 3D interactive glasses to the camera of the display device displaying the content, thereby realizing the real-time adjustment of the display viewing angle of the content displayed by the 3D large-screen virtual reality display device, and achieving the purpose of the display device displaying the content moving with the movement of the human head (interactive glasses).

[0107] The above is a description of the human-computer interaction method of the embodiment of the present invention. Figure 5-7 , a human-computer interaction system for executing the human-computer interaction method of an embodiment of the present invention is described in detail. When describing the human-computer interaction system, the contents described above will not be repeated.

[0108] like Figure 5 , which is a structural block diagram of a first embodiment of a human-computer interaction system provided by the present invention. The human-computer interaction system 500 includes:

[0109] A virtual reality display device 51 is used to display image content and may include: a 3D large-screen virtual reality display device, a 3D desktop display device or a 3D head-mounted display device.

[0110] The optical motion capture system 52 is used to obtain the position information and gesture information of the user's hand. The optical motion capture system 52 may include a marker and an optical motion capture camera.

[0111] The inertial system 53 is used to obtain the rotation information of the user's hand. The inertial system 53 may be a wireless IMU module.

[0112] The data processing device 54 is used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device according to the position information obtained by the optical motion capture system 52 and the rotation information of the user's hand obtained by the inertial system 53; and to determine the interactive operation mode corresponding to the gesture information according to the gesture information obtained by the optical motion capture system 52; and to execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0113] In the human-computer interaction system of the embodiment of the present invention, when the user interacts with the virtual reality display device, the user no longer relies on the traditional interactive handle or interactive pen, so the interactive operation mode is no longer limited by the number of physical buttons on the interactive handle; and because the user can make a variety of different gestures to control the movement of the virtual operating hand in real time, more interactive operations that meet the user's needs can be performed. Therefore, the human-computer interaction method of the present invention can get rid of the constraints of the interactive handle and the interactive pen, and can realize more human-computer interaction functions.

[0114] like Figure 6 , which is a structural block diagram of a second embodiment of a human-computer interaction system provided by the present invention. The human-computer interaction system 600 includes:

[0115] Virtual reality display device 61, the virtual reality display device 51 is used to display image content, and may include: a 3D large-screen virtual reality display device, a 3D desktop display device or a 3D head-mounted display device.

[0116] The optical motion capture system 62 is used to obtain the position information of the user's hand; the optical motion capture system 62 may include a marker point and an optical motion capture camera.

[0117] The inertial system 63 is used to obtain the rotation information of the user's hand; wherein the inertial system 63 can be a wireless IMU module.

[0118] The gesture recognition system 64 is used to obtain the gesture information of the user's hand; wherein, the gesture recognition system 64 can be: a gesture recognition device, such as a single ordinary camera, a single binocular camera, a single TOF camera, etc.

[0119] The data processing device 65 is used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device 61 according to the position information obtained by the optical motion capture system 62 and the rotation information of the user's hand obtained by the inertial system 63; and to determine the interactive operation mode corresponding to the gesture information according to the gesture information obtained by the gesture recognition system 64; and to execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0120] In the human-computer interaction system of the embodiment of the present invention, when the user interacts with the virtual reality display device, the user no longer relies on the traditional interactive handle or interactive pen, so the interactive operation mode is no longer limited by the number of physical buttons on the interactive handle; and because the user can make a variety of different gestures to control the movement of the virtual operating hand in real time, more interactive operations that meet the user's needs can be performed. Therefore, the human-computer interaction method of the present invention can get rid of the constraints of the interactive handle and the interactive pen, and can realize more human-computer interaction functions.

[0121] like Figure 7 , which is a structural block diagram of a third embodiment of a human-computer interaction system provided by the present invention. The human-computer interaction system 700 includes:

[0122] A virtual reality display device 71 is used to display image content and may include: a 3D large-screen virtual reality display device, a 3D desktop display device or a 3D head-mounted display device.

[0123] The optical motion capture system 72 is used to obtain the positioning information and gesture information of the user's hand, wherein the positioning information includes position information and rotation information. The optical motion capture system 72 may include a marker point and an optical motion capture camera.

[0124] The data processing device 73 is used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device 71 according to the positioning information obtained by the optical motion capture system 72; and to determine the interactive operation mode corresponding to the gesture information according to the gesture information obtained by the optical motion capture system 72; and to execute and respond to the triggering operation of the virtual operator on the content to be operated according to the interactive operation mode.

[0125] In the human-computer interaction system of the embodiment of the present invention, when the user interacts with the virtual reality display device, the user no longer relies on the traditional interactive handle or interactive pen, so the interactive operation mode is no longer limited by the number of physical buttons on the interactive handle; and because the user can make a variety of different gestures to control the movement of the virtual operating hand in real time, more interactive operations that meet the user's needs can be performed. Therefore, the human-computer interaction method of the present invention can get rid of the constraints of the interactive handle and the interactive pen, and can realize more human-computer interaction functions.

[0126] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A human-computer interaction method for realizing interaction between a user and a virtual reality display device, characterized in that: include: Obtain the user's hand positioning information and gesture information; Determining, according to the acquired positioning information, content to be operated by the virtual operator in content displayed by the virtual reality display device; Determining a corresponding interactive operation mode according to the acquired gesture information; Executing and responding to a triggering operation of the virtual operator on the content to be operated according to the interactive operation mode; The positioning information includes: position information and rotation information; the step of obtaining the positioning information of the user's hand specifically includes: capturing an image of a rigid body bound to the user's hand through an optical motion capture system to obtain position information and rotation information of the user's hand; or, The image of the marker point bound to the user's hand is captured by an optical motion capture system to obtain the position information of the user's hand; and the rotation information of the user's hand is obtained by an inertial system; The step of obtaining the user's hand gesture information specifically includes: Using the optical motion capture system to capture an image of a rigid body bound to the user's hand to obtain gesture information of the user's hand; or; Taking an image of the user's hand by a gesture recognition system to obtain gesture information of the user's hand; The step of determining the content to be operated by the virtual operator in the display content of the virtual reality display device according to the acquired positioning information specifically includes: determining the position to be operated of the virtual operator in the display content according to a preset position mapping relationship according to the acquired positioning information of the user's hand; and determining the content to be operated corresponding to the position to be operated; Among them, the six-degree-of-freedom information of the user's hand is calculated through a fusion algorithm such as Kalman filtering, and the six-degree-of-freedom information is input into the content platform in real time, and finally presented to the display content of the virtual reality display device; then according to the pre-set position information relationship, the position to be operated and the content to be operated of the virtual operator in the display content can be determined.

2. The human-computer interaction method according to claim 1, characterized in that: The step of determining the corresponding interactive operation mode according to the acquired gesture information specifically includes: According to the acquired gesture information, determining the interactive operation mode corresponding to the gesture information according to a preset mapping relationship between gesture information and operation mode; or, According to the acquired gesture information, the specific key on the keyboard corresponding to the gesture information is determined according to the preset mapping relationship between gesture information and keyboard keys, and the interactive operation mode corresponding to the specific key is determined according to the preset mapping relationship between keyboard keys and operation modes.

3. The human-computer interaction method according to claim 1, characterized in that: The virtual reality display device includes: a 3D large-screen virtual reality display device, a 3D desktop display device or a 3D head-mounted display device.

4. The human-computer interaction method according to claim 2, characterized in that: When the virtual reality display device includes a 3D large-screen virtual reality display device, the method further includes: Acquiring positioning information of the 3D interactive glasses worn by the user through the optical motion capture system; According to the positioning information of the 3D interactive glasses, the display viewing angle of the content displayed by the 3D large-screen virtual reality display device is adjusted in real time.

5. A human-computer interaction system, characterized in that: include: Virtual reality display device, used to display image content; Optical motion capture system, used to obtain the position information and gesture information of the user's hand; An inertial system, used to obtain rotation information of the user's hand; A data processing device, configured to determine the content to be operated by the virtual operator in the content displayed by the virtual reality display device according to the position information acquired by the optical motion capture system and the rotation information of the user's hand acquired by the inertial system; and determining, according to the gesture information acquired by the optical motion capture system, an interactive operation mode corresponding to the gesture information; and executing and responding to a triggering operation of the virtual operator on the content to be operated according to the interactive operation mode; The image of the marker point bound to the user's hand is captured by an optical motion capture system to obtain the position information of the user's hand; and the rotation information of the user's hand is obtained by an inertial system; Capturing an image of a rigid body bound to the hand of the user by the optical motion capture system to obtain gesture information of the hand of the user; The data processing device is used to: determine the position to be operated of the virtual manipulator in the displayed content according to the acquired position information and rotation information of the user's hand and a preset position mapping relationship; and determine the content to be operated corresponding to the position to be operated; Among them, the six-degree-of-freedom information of the user's hand is calculated through a fusion algorithm such as Kalman filtering, and the six-degree-of-freedom information is input into the content platform in real time, and finally presented to the display content of the virtual reality display device; Then, according to the preset position information relationship, the position to be operated and the content to be operated of the virtual operator in the displayed content can be determined.

6. A human-computer interaction system, characterized in that: include: Virtual reality display device, used to display image content; An optical motion capture system to obtain the position and rotation information of the user's hands; A gesture recognition system, used to obtain the gesture information of the user's hand; A data processing device, used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device according to the position information and rotation information acquired by the optical motion capture system; and determining, according to the gesture information acquired by the gesture recognition system, an interactive operation mode corresponding to the gesture information; and executing and responding to a triggering operation of the virtual operator on the content to be operated according to the interactive operation mode; capturing an image of a rigid body bound to the user's hand through an optical motion capture system to obtain position information and rotation information of the user's hand; Taking an image of the user's hand by a gesture recognition system to obtain gesture information of the user's hand; The data processing device is used to: determine the position to be operated of the virtual manipulator in the displayed content according to the acquired position information and rotation information of the user's hand and a preset position mapping relationship; and determine the content to be operated corresponding to the position to be operated; Among them, the six-degree-of-freedom information of the user's hand is calculated through a fusion algorithm such as Kalman filtering, and the six-degree-of-freedom information is input into the content platform in real time, and finally presented to the display content of the virtual reality display device; Then, according to the preset position information relationship, the position to be operated and the content to be operated of the virtual operator in the displayed content can be determined.

7. A human-computer interaction system, characterized in that: include: Virtual reality display device, used to display image content; An optical motion capture system for obtaining the position, rotation and gesture information of the user's hand; A data processing device, used to determine the content to be operated by the virtual operator in the display content of the virtual reality display device according to the position information and rotation information acquired by the optical motion capture system; and determining, according to the gesture information acquired by the optical motion capture system, an interactive operation mode corresponding to the gesture information; and executing and responding to a triggering operation of the virtual operator on the content to be operated according to the interactive operation mode; capturing an image of a rigid body bound to the user's hand through an optical motion capture system to obtain position information and rotation information of the user's hand; Capturing an image of a rigid body bound to the hand of the user by the optical motion capture system to obtain gesture information of the hand of the user; The data processing device is used to: determine the position to be operated of the virtual manipulator in the displayed content according to the acquired position information and rotation information of the user's hand and a preset position mapping relationship; and determine the content to be operated corresponding to the position to be operated; Among them, the six-degree-of-freedom information of the user's hand is calculated through a fusion algorithm such as Kalman filtering, and the six-degree-of-freedom information is input into the content platform in real time, and finally presented to the display content of the virtual reality display device; Then, according to the preset position information relationship, the position to be operated and the content to be operated of the virtual operator in the displayed content can be determined.

Images