VR controller-based gesture recognition method and system for VR psychological catharsis room

By recording and identifying the gestures of the VR controller, the problem of the gap between the complexity of the handle interaction method and the actual operation is solved, and more precise and natural gesture interaction is achieved, which improves the immersion and therapeutic effect of the virtual reality psychological catharsis room.

CN119088224APending Publication Date: 2024-12-06SOUTHWEST JIAOTONG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411272519.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing VR controller-based handle interaction mode is complex and has a gap with the actual operation, which affects the immersion and therapeutic effect of the virtual reality psychological catharsis room.

Method used

By recording and identifying gestures by users operating VR controllers, obtaining and storing the position, rotation and direction information of gestures, gesture recognition is realized, and loading recognition data through asynchronous methods, providing more natural and accurate gesture interaction.

Benefits of technology

It realizes more accurate and natural gesture recognition, reduces hardware costs and performance consumption, and improves the user experience and the effectiveness of psychotherapy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088224A_ABST
    Figure CN119088224A_ABST
Patent Text Reader

Abstract

The invention discloses a gesture recognition method and system based on a VR controller for a VR psychological catharsis room, relates to the computer technology, and solves the technical problems that an existing handle interaction mode based on the VR controller is complex in operation and large in difference with real operation, and the immersion of the psychological catharsis room based on virtual reality is affected. The method comprises the following steps: A, recording a specific gesture made when a user operates a VR controller, and storing the position, rotation and direction information of the gesture as gesture recognition data; b, acquiring a file storage path, and storing the gesture recognition data into a file; c, obtaining a file path needing to be loaded, and loading gesture recognition data through an asynchronous method; according to the invention, based on the VR controller, gesture recognition is realized, more types, more accurate and more natural gesture recognition methods can be provided, and the performance consumption is lower.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a gesture recognition method and system based on a VR controller for a VR psychological venting room. Background Art

[0002] With the rapid development of virtual reality technology, users' demand for immersive experience is also increasing. Existing VR interaction methods include motion capture, eye tracking, gesture tracking, and handle interaction based on VR controllers. Different methods have different advantages and disadvantages. Motion capture technology achieves interaction by capturing the user's body movements. Although this method can provide a high degree of immersion and a natural operating experience, the required motion capture system usually requires complex and expensive hardware equipment, and has high requirements for the environment, which limits its application in ordinary homes and small venues.

[0003] Eye tracking technology achieves interaction by monitoring the user's eye movements. Although this method can achieve fast and intuitive operation, due to the complexity of eye movements and individual differences, eye tracking technology still faces challenges in accuracy and stability. In addition, some users will feel uncomfortable during use, affecting the user experience.

[0004] Gesture tracking technology has emerged in the past two years. Although this method can improve the user's sense of immersion to a certain extent by capturing the user's hand movements to achieve interaction, due to the hardware settings of the device, the accuracy of the captured user's hand movements is not high. At the same time, due to the limitations of the device processor, the capture frequency of gesture tracking cannot meet expectations.

[0005] Compared with the previous technologies, the handle interaction method based on VR controller has been widely used due to its accuracy and reliability. Users can operate by pressing the buttons on the handle. This method has low hardware cost and a relatively flat learning curve. However, the traditional handle interaction method still has certain limitations. For example, due to the limited number of buttons, users may need to memorize multiple button combinations for complex operations, which makes the operation more complicated. In addition, there is a certain gap between the traditional handle operation method and natural hand movements, which will also affect the user's immersion.

[0006] In the field of mental health education, the use of virtual reality technology to achieve psychological venting rooms, psychological sand tables, etc., all need to provide users with a deeper sense of immersion to simulate the realism of venting rooms, sand tables, etc. in reality; especially in venting rooms, there are more gestures involved than in psychological sand tables and psychological counseling rooms. Users may need to have gestures such as grabbing a brush to doodle, putting on boxing gloves to hit a sandbag, etc. However, traditional VR devices do not have many buttons for operation, and they are different from hand movements in reality, which to a certain extent reduces the user's sense of immersion and affects the effectiveness of psychological counseling and treatment. Summary of the invention

[0007] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a gesture recognition method and system based on a VR controller for a VR psychological venting room, which solves the technical problem that the existing handle interaction method based on the VR controller is complex to operate, has a large gap with the actual operation, and affects the immersive feeling of the psychological venting room based on virtual reality.

[0008] The gesture recognition method based on VR controller for VR psychological venting room includes the following steps: Step A: Recording specific gestures made by the user when operating the VR controller, and storing the position, rotation, and direction information of the gestures as gesture recognition data; Step B: Get the file saving path and save the gesture recognition data to the file; Step C: Get the file path to be loaded and load the gesture recognition data through an asynchronous method.

[0009] Further, step A comprises: A1. Obtain the position and rotation information of the user's head mounted device HMD: first obtain the game object of the HMD, and then obtain the position and rotation information of the HMD; A2, monitor whether the controller object is activated, if activated, enter A3; A3. Record the gestures made by the user operating the VR controller; A4. Determine the gesture and update the gesture data.

[0010] Further, step A2 comprises: Detect whether there is an activated controller, and check whether the left and right hand controllers of the user are triggered respectively. If a controller is triggered, the triggered controller will be used as the active controller. If neither controller is pressed, no operation will be performed; when it is detected that the controller is pressed, start recording gestures, and set the value used to judge the gesture recognition status to true. The trigger value controller pressing force exceeds the threshold.

[0011] Further, step A3 includes: If the gesture has started and the controller's trigger is still pressed, update the HMD's position and rotation information, get the position and rotation information of the active controller and update the gesture trajectory; if the user releases the trigger to indicate the gesture is over, set the active controller to null; Further, step A4 includes: If the user releases the trigger, the gesture recording ends and the relevant information of the gesture is obtained, including the position, rotation and direction information as gesture recognition data; it is determined whether a new gesture sample is currently being recorded. If so, the name of the currently recorded gesture and the total number of recorded gesture samples are output; if not, it indicates that no new gesture is currently being recorded, and the gesture is compared with the recorded gesture recognition data. If the comparison fails, an error message is output, and if the comparison is successful, the name of the gesture is output.

[0012] Furthermore, comparing the gesture with the recorded gesture recognition data includes: comparing the direction of each action of the gesture with the direction of all samples of the corresponding gesture in the gesture library; if the directions are opposite or the deviation is greater than a threshold, it indicates that they are not the same gesture; if the directions are consistent or the deviation is less than the threshold, continuing to calculate the variance of the gesture and all samples, respectively calculating the variance between the direction of each action of the gesture and all samples, and then performing addition processing; comparing the obtained result with the threshold set by the user; if the result is less than the threshold, it indicates that the gesture is the same gesture and the recognition is successful; if the result is greater than the threshold, it indicates that the gesture is not the same gesture and the recognition fails.

[0013] Further, step B comprises: Get the complete save path of the file, and asynchronously save the gesture recognition data to the file path. If the return value is 0, it means the save operation is successful, update the relevant status information and return true; if the return value is not 0, it means the operation failed, update the console text to display the error information and return false.

[0014] Further, step C comprises: Get the file path of the user's gesture data, load the gesture file through an asynchronous method and get the return value. If the return value is not 0, it means that the asynchronous loading has failed. Read the file content. If the file content is empty or the length is 0, it means that the file does not exist. Continue to update the console information to show that the file cannot be found and the range is false, indicating that the loading operation is recognized; if the return value is 0, it means that the loading is successful, update the console information and return true.

[0015] The VR controller-based gesture recognition system for VR psychological venting room includes: User head device or HMD: VR helmet worn by the user; VR controller: A handle used to interact with a VR headset; Gesture recording and recognition module: used to determine whether it is in a gesture state; update and record the position, rotation and direction information of the HMD and controller in real time; compare and identify the position, rotation and direction information of the gesture with the recorded gesture recognition data; Gesture storage module: used to obtain the file saving path; save gesture recognition data to the file; Gesture loading module: used to obtain the file path to be loaded; load gesture recognition data through asynchronous methods.

[0016] The beneficial effects of the present invention include: 1. The present invention realizes gesture recognition based on a VR controller, and can provide gesture recognition methods with more types, more accuracy, and more naturalness, and has lower performance consumption.

[0017] 2. The present invention opens interfaces for gesture recording, storage, and loading. Users can record custom gestures according to specific needs and change the default gesture storage and loading paths, making it more convenient for users to use. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A gesture recognition structure diagram based on a VR controller provided in an embodiment of the present invention.

[0019] Figure 2 A schematic diagram of buttons of a VR controller provided by an embodiment of the present invention.

[0020] Figure 3 A Z-shaped gesture path diagram is provided in an embodiment of the present invention.

[0021] Figure 4 A schematic diagram of a Z-gesture path space coordinate system provided in an embodiment of the present invention.

[0022] Figure 5 A gesture recognition flow chart provided for an example of the present invention.

[0023] Figure 6 A gesture storage flow chart provided for an example of the present invention.

[0024] Figure 7 A gesture loading flow chart provided for an example of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0026] User head device (HMD): a VR helmet worn by a user; in this embodiment, the helmet exists in the form of an object.

[0027] VR controllers: Two handles of VR devices, such as Figure 2 As shown; in this embodiment, both handles exist in the form of objects.

[0028] Gestures: The gestures mentioned in the present invention are all gestures made by a user when operating a VR controller based on a VR controller (for example, a user presses a trigger key on the controller and waves the controller to make a waving motion).

[0029] Gesture recording: Records specific gestures made by users when operating VR controllers, and stores the position, rotation, and direction information of the gestures.

[0030] Gesture storage: Save the gesture information stored during gesture recording in the form of a file to a specific local file path.

[0031] Gesture loading: Load a locally saved file containing gesture data into the running program.

[0032] The gesture recognition method based on VR controller for VR psychological venting room includes the following steps: Step A: Gesture recording and recognition, such as Figure 5 As shown, it includes: determining whether it is in a gesture state; updating and recording the position, rotation and direction information of the HMD and the controller in real time; comparing and identifying the position, rotation and direction information of the gesture with the recorded gesture recognition data; A1. Obtain the location and rotation information of the user's head-mounted device: Get the user's head device, that is, the HMD game object (referring to an object in the scene), read the transform.position value of the HMD object, and assign it to the Vector3 type variable hmd_p (representing the position information of the HMD). Similarly, read the transform.rotation value of the HMD object and assign it to the Quaternion type variable hmd_q (representing the rotation information of the HMD).

[0033] A2. Determine whether the controller object is activated: Secondly, determine whether there is an activated controller. If not, it means that the gesture has not started yet; continue to monitor whether the left and right controllers are triggered. If any controller is triggered (that is, the pressure exceeds the threshold of 0.85), the existing function OVRInput.Get() is called to start the gesture and set the triggered controller as the activated controller; if no controller is triggered, no operation is performed.

[0034] In this example, it is determined that the index finger trigger key of the right controller is triggered, that is, the right controller is set as the activated controller.

[0035] A3. Gesture recording: If the user continues to hold the controller and is dragging it, the position of the gesture continues to be updated, and the position and rotation information of the HMD and the activated controller are obtained and updated in real time through transform.position and transform.rotation. When the user releases the controller, the gesture ends.

[0036] In this example, the user holds down the index finger trigger button of the right controller and moves in a Z-shaped path. When the path ends, the user releases the index finger trigger button of the right controller to end the gesture.

[0037] A4. Gesture recognition and update: When the gesture ends, determine whether the user is recording a new gesture sample at this time (by setting a Boolean variable and setting it to be visible in the Inspector. If the user needs to record a custom gesture sample, set it to true. When the value is detected to be true, it means that the user is customizing the gesture sample.). If so, display the currently recorded gesture sample information, including the gesture name and the number of samples recorded, and the gesture recording step ends here; if not, start the gesture recognition step.

[0038] Gesture recognition: The comparison of the gesture with the recorded gesture recognition data includes: comparing the direction of each action of the gesture with the direction of all samples of the corresponding gesture in the gesture library. If the directions are opposite or the deviation is greater than the threshold, it indicates that they are not the same gesture. If the directions are consistent or the deviation is less than the threshold, continue to calculate the variance of the gesture and all samples, calculate the variance between the direction of each action of the gesture and all samples respectively, and then add them together. Compare the result with the threshold set by the user. If the result is less than the threshold, it indicates that the gesture is the same gesture and the recognition is successful. If the result is greater than the threshold, it indicates that the gesture is not the same gesture and the recognition fails.

[0039] Compare the position and rotation information of the gesture with the recorded gesture recognition data. The specific comparison involves calculating the average value of the position and rotation information, and calculating the variance. When the average value or variance exceeds the critical value, false is returned, indicating that the recognition failed and the gesture does not exist in the gesture library. If the average value and variance are within the critical value, true is returned and the name of the gesture that matches it is displayed.

[0040] In this example, in order to simplify the calculation and make the calculation results more intuitive, it is assumed that the Z path is completely in the XOY coordinate plane (that is, the Z coordinate value in the spatial coordinate system is 0), such as Figure 3-4 The position data of the Z gesture path is shown in the figure, where X1, X2, and X3 are all space vectors.

[0041] First, the directions of X1, X2, and X3 are compared with the directions of all samples of gesture Z in the gesture library. If the directions are opposite or the deviation is greater than the threshold, it indicates that they are not the same gesture. If the directions are consistent or the deviation is less than the threshold, the variance of the gesture and all samples is calculated. In this example, the variances between X1, X2, and X3 and all samples are calculated separately, and then added together. The result is compared with the threshold set by the user. If the result is less than the threshold, it indicates that the gesture is the same gesture and the recognition is successful. If the result is greater than the threshold, it indicates that the gesture is not the same gesture and the recognition fails.

[0042] Step B: Gesture storage, such as Figure 6 As shown, it includes obtaining the file saving path and saving the gesture recognition data to the file; First, when the user passes in a file name or file path, the Path.IsPathRooted() method is called to determine whether the passed in path is an absolute path. If it is an absolute path, the path is returned directly. If it is not an absolute path, a relative path is generated, and Application.persistentDataPath is combined with the file name to generate a complete relative path and return it.

[0043] After obtaining the complete path, call the saveFileAsync() method to asynchronously save the gesture file and obtain the return value. If the return value is 0, it means that the save is successful, and the console message is updated. If the return value is not 0, it means that the save fails, and the console message is updated to display the information of the save failure, including the error message.

[0044] Step C: Gesture loading, such as Figure 7 As shown, it includes: obtaining the file path to be loaded, and loading the gesture recognition data through an asynchronous method.

[0045] First, when the user passes in a file name or file path, the Path.IsPathRooted() method is called to determine whether the passed in path is an absolute path. If it is an absolute path, the path is returned directly. If it is not an absolute path, a relative path is generated, and Application.persistentDataPath is combined with the file name to generate a complete relative path and return it.

[0046] After obtaining the complete path, call loadFromFileAsync() to asynchronously load the gesture file and get the return value. If the return value is not 0, it means that the asynchronous loading failed. Call the ReadAllBytes() method to read the content of the file. If the file content is empty or the length is 0, it means that the file does not exist. Update the console message to display the information that the file cannot be found, and return false, indicating that the loading operation failed. Call the loadFromBuffer() method to load the gesture file from the buffer and get the return value. If the return value is not 0, it means that the loading from the buffer failed: update the console message to display the loading failure information, including the error message, and return false, indicating that the loading operation failed. Otherwise, it indicates that the loading is successful. Update the console message to display the loading success and return true.

[0047] In another embodiment, a gesture recognition system based on a VR controller for a VR psychological catharsis room is provided, such as Figure 1 As shown, including: User head device, i.e. HMD: a VR helmet worn by a user; in this embodiment, the helmet exists in the form of an object; VR controller: a handle used to interact with a VR helmet; in this embodiment, both handles exist in the form of objects; Gesture recording and recognition module: used to determine whether it is in a gesture state; update and record the position, rotation and direction information of the HMD and controller in real time; compare and identify the position, rotation and direction information of the gesture with the recorded gesture recognition data; Gesture storage module: used to obtain the file saving path; save gesture recognition data to the file; Gesture loading module: used to obtain the file path to be loaded; load gesture recognition data through asynchronous methods.

[0048] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A gesture recognition method based on VR controller for VR psychological venting room, It is characterized in that The following steps are involved: Step A: Recording specific gestures made by the user when operating the VR controller, and storing the position, rotation, and direction information of the gestures as gesture recognition data; Step B: Get the file saving path and save the gesture recognition data to the file; Step C: Get the file path to be loaded and load the gesture recognition data through an asynchronous method.

2. The gesture recognition method based on a VR controller for a VR psychological venting room according to claim 1, It is characterized in that Step A includes: A1. Obtain the position and rotation information of the user's head mounted device HMD: first obtain the game object of the HMD, and then obtain the position and rotation information of the HMD; A2, monitor whether the controller object is activated, if activated, enter A3; A3. Record the gestures made by the user operating the VR controller; A4. Determine the gesture and update the gesture data.

3. The gesture recognition method based on a VR controller for a VR psychological venting room according to claim 1, It is characterized in that Step A2 includes: Detect whether there is an activated controller, and check whether the left and right hand controllers of the user are triggered respectively. If any controller is triggered, the triggered controller will be used as the active controller. If neither controller is triggered, no operation will be performed. When it is detected that the controller is triggered, start recording gestures, and set the value used to judge the gesture recognition status to true. The trigger refers to the controller pressing force exceeding the threshold.

4. The gesture recognition method based on a VR controller for a VR psychological venting room according to claim 1, It is characterized in that Step A3 includes: If the gesture has started and the controller's trigger is still pressed, update the HMD's position and rotation information, get the position and rotation information of the activated controller and update the gesture trajectory; if the controller's trigger is not pressed, indicating the gesture is over, set the activated controller to null.

5. The gesture recognition method based on a VR controller for a VR psychological venting room according to claim 1, It is characterized in that Step A4 includes: If the user releases the trigger, the gesture recording ends and the relevant information of the gesture is obtained, including the position, rotation and direction information as gesture recognition data. It is determined whether a new gesture sample is currently being recorded. If so, the name of the currently recorded gesture and the total number of recorded gesture samples are output; if not, it indicates that no new gesture is currently being recorded. The gesture is compared with the recorded gesture recognition data. If the comparison fails, an error message is output. If the comparison is successful, the name of the corresponding gesture is output.

6. The gesture recognition method based on a VR controller for a VR psychological venting room according to claim 5, It is characterized in that The comparing the gesture with the recorded gesture recognition data includes: the comparing the gesture with the recorded gesture recognition data includes: comparing the direction of each action of the gesture with the direction of all samples of the corresponding gesture in the gesture library, if the directions are opposite or the deviation is greater than the threshold, it indicates that they are not the same gesture, if the directions are consistent or the deviation is less than the threshold, then continue to calculate the variance of the gesture and all samples, calculate the variance between the direction of each action of the gesture and all samples respectively, and then perform addition processing, compare the obtained result with the threshold set by the user, if the result is less than the threshold, it indicates that the gesture is the same gesture and the recognition is successful, if the result is greater than the threshold, it indicates that the gesture is not the same gesture and the recognition fails.

7. The gesture recognition method based on a VR controller for a VR psychological venting room according to claim 1, It is characterized in that Step B includes: Get the complete save path of the file, and asynchronously save the gesture recognition data to the file path. If the return value is 0, it means the save operation is successful, update the relevant status information and return true; if the return value is not 0, it means the operation failed, update the console text to display the error information and return false.

8. The gesture recognition method based on VR controller for VR psychological venting room according to claim 1, It is characterized in that Step C includes: Get the file path of the user's gesture data, load the gesture file through an asynchronous method and get the return value. If the return value is not 0, it means that the asynchronous loading has failed. Read the file content. If the file content is empty or the length is 0, it means that the file does not exist. Continue to update the console information to show that the file cannot be found and the range is false, indicating that the loading operation is recognized; if the return value is 0, it means that the loading is successful, update the console information and return true.

9. A gesture recognition system based on VR controller for VR psychological venting room, It is characterized in that include: User head device or HMD: VR helmet worn by the user; VR controller: A handle used to interact with a VR headset; Gesture recording and recognition module: used to determine whether it is in a gesture state; update and record the position, rotation and direction information of the HMD and controller in real time; compare and identify the position, rotation and direction information of the gesture with the recorded gesture recognition data; Gesture storage module: used to obtain the file saving path; Save gesture recognition data to a file; Gesture loading module: used to obtain the file path to be loaded; load gesture recognition data through asynchronous methods.